JP5156692B2 - Pseudo data generation device, pseudo data generation method, and computer program - Google Patents

Description

  The present invention relates to a pseudo data generation device, a pseudo data generation method, and a computer program.

  In recent years, “information utilization” has attracted attention in many organizations regardless of whether it is a company or a public institution. It can be said that this is a manifestation of increasing awareness that the vast amount of information (data) accumulated through the IT (Information Technology) of business is useful for business improvement and decision making.

  Along with this, “BI (Business Intelligence)” has attracted attention. BI is a mechanism that systematically and systematically analyzes internal and external data to generate knowledge and insight useful for business, or a system and technology that realizes such a mechanism. In order to visualize management and make effective decisions, we have introduced BI.

  However, in many cases, the introduction effect of BI cannot be fully exhibited. The reason for this is thought to be that, due to the lack of human resources with analytical skills, it was impossible to master the BI, and as a result, knowledge and insights that led to decision making could not be extracted.

  Therefore, there is an urgent need to develop human resources to acquire skills related to information utilization that can make full use of BI. Such human resource development efforts are carried out by, for example, a data utilization consulting company. For this purpose, it is effective to conduct “data analysis training” based on past consulting projects. This practice is a training to learn the data analysis framework and how to interpret the analysis results by actually analyzing the data, and you can learn the know-how of analysis consulting.

  On the other hand, Non-Patent Document 1 analyzes the meaning of the character string of the original data by the automatic analysis function, and resembles the information such as the name, address, and telephone number of the table determined to be personal information by the pseudo data conversion function. An application for automatically and safely creating pseudo data for test / verification from production data by converting to a character string is disclosed. As a result, in database development, the generated pseudo data can be used to obtain the same results as testing / verification using production data, and the production of pseudo data can be greatly reduced. This makes it possible to create and use pseudo data without business knowledge.

Sea Net Networks Japan Co., Ltd. releases DB Secure Utility, a pseudo data automatic creation and report issuing tool for DB development, [online], System EXE, Inc. [Search April 27, 2009], Internet <http : //japan.zdnet.com/news/devsys/story/0,2000056182,20368504,00.htm>

  When data is analyzed in the “data analysis practice” as described above, it is ideal to use actually collected data (hereinafter referred to as “actual data”). However, actual data collected by companies and public institutions is highly confidential data from the viewpoint of personal information protection and communication confidentiality. Therefore, in companies that conduct consulting for data utilization, it is almost impossible to store actual data stored for consulting by customers or use it for purposes other than consulting. Therefore, in a consulting company, it is necessary to create pseudo data and perform data analysis training using the pseudo data. Even if the actual data is available, the actual data is often large-scale, so only a part is extracted from the viewpoint of cost and small-scale pseudo data is created. Must be destroyed. In order to acquire skills and accumulate know-how, the created pseudo data needs to be similar in nature to the actual data, that is, the distribution tendency of the setting values of each parameter must be similar.

  Also, when developing a system such as a large-scale DWH (Data WareHouse) / data mart, the development process requires the load and counting speed to be as accurate as possible, and verification with data closer to the actual data is required. Yes. Similarly, in the development of other analysis systems, if there is a time constraint on the analysis time including the construction of an analysis model, it is required to verify the system with more accurate data. Also in this case, it is necessary to use pseudo data in which the distribution tendency of the set values of the actual data and each parameter is similar.

  However, in the technique of Non-Patent Document 1, the meaning of the character string is interpreted, and the character string conversion function is realized only for the data related to the personal information, but the actual data itself is retained, It is necessary to copy the actual data setting value itself, and when generating actual data with a smaller capacity than the actual data, generate pseudo data that retains the distribution tendency of each parameter setting value in the actual data is not.

  The present invention has been made in view of the above circumstances, and its purpose is to maintain the distribution capacity of each parameter setting value in the actual data without storing the actual data itself. It is an object of the present invention to provide a pseudo data generation device, a pseudo data generation method, and a computer program that can generate pseudo data.

  In order to solve the above problems, the present invention provides an information reading unit that reads actual data, and distribution of set values that can be taken by analysis model parameters that are used in an analysis model that is a calculation formula for occurrence probability of a predetermined action or event. A parameter creating unit that calculates a value indicating a parameter setting value in the actual data corresponding to the analysis model parameter, a setting value that can be taken by the analysis model parameter is determined at random, and the analysis model is used. The probability of occurrence of the predetermined action or event is calculated from the set value of the analysis model parameter determined at random and the value indicating the distribution of the set value of the analysis model parameter created by the parameter creation unit. , The set value of the analysis model parameter determined at random for the generation of pseudo data based on the calculated probability If it is determined whether or not to use for the generation of pseudo data, the process of determining the setting values of the parameters constituting the pseudo data based on the setting values of the analysis model parameters determined at random is repeated, A pseudo data generation device comprising: a pseudo data generation unit that generates data.

  In addition, the present invention is the above-described pseudo data generation device, wherein the pseudo data generation unit generates a set value of a parameter constituting the pseudo data for a predetermined number of records.

  Further, the present invention provides the above-described pseudo data generation device, wherein a value indicating a distribution of set values that can be taken by the analysis model parameter used in the analysis model is stored in the pseudo data corresponding to the analysis model parameter. The setting value distribution is calculated from the value indicating the distribution of the calculated setting value and the value indicating the distribution of the setting value generated based on the setting value of the parameter in the actual data. A verification unit that determines whether or not the deviation is greater than a predetermined value, and if the verification unit determines that there is a deviation, deletes part or all of the pseudo data and creates pseudo data in the pseudo data generation unit And a pseudo data regeneration unit for instructing.

Further, the present invention provides a pseudo-data generating method for use in a pseudo-data generating device, information reading unit, and an information reading unit step of reading actual data, parameter creation unit, the probability of occurrence of a predetermined behavior or events A parameter creation step of calculating a value indicating a distribution of setting values that can be taken by the analysis model parameter used in the analysis model that is a calculation formula of the parameter from a setting value of the parameter in the actual data corresponding to the analysis model parameter; The pseudo data generation unit randomly determines setting values that can be taken by the analysis model parameter, and the analysis model is used to randomly determine the setting value of the analysis model parameter and the parameter generation step. And the occurrence of the predetermined action or event from the value indicating the distribution of the set value of the analysis model parameter. If the probability is calculated, whether to use the set value of the analysis model parameter determined at random for the generation of pseudo data based on the calculated probability, and if it is determined to use for the generation of pseudo data, A pseudo data generating step of generating pseudo data by repeating the process of determining the set values of parameters constituting the pseudo data based on the set values of the analysis model parameters determined at random; It is a generation method.

  In addition, the present invention provides a computer used as a pseudo data generation device with a value indicating a distribution of setting values that can be taken by an analysis model parameter used in an analysis model that is a calculation formula for occurrence probability of a predetermined action or event. A parameter creation unit that calculates the setting value of the parameter in the actual data corresponding to the analysis model parameter, the setting value that the analysis model parameter can take is determined at random, and the analysis model is used to determine the setting value that is randomly determined The occurrence probability of the predetermined action or event is calculated from the setting value of the analysis model parameter and the value indicating the distribution of the setting value of the analysis model parameter created by the parameter creation unit, and the calculated probability The set value of the analysis model parameter determined at random is used to generate pseudo data based on If it is determined whether to use for the generation of pseudo data, it repeats the process of determining the setting values of the parameters constituting the pseudo data based on the setting values of the analysis model parameters determined at random, the pseudo data It is a computer program characterized by functioning as a pseudo data generation part which generates.

  According to the present invention, the pseudo data having the necessary data size that retains the distribution tendency of the setting values of each parameter in the actual data without accumulating the actual data itself or copying and using the setting values. Can be generated. Therefore, it is possible to generate pseudo data while reducing costs, and it is possible to accumulate data analysis consulting know-how that is close to reality by using the generated pseudo data for data analysis exercises and the like. . In addition, this pseudo data can be used for verification of the analysis system to obtain a highly accurate verification result.

It is a block diagram of the pseudo data generation apparatus by one embodiment of this invention. It is a figure which shows the data structural example of the actual data by the embodiment. It is a figure which shows the data structural example of the analysis information table by the embodiment. It is a figure which shows the example of the analysis scenario by the embodiment. It is a figure which shows the data structural example of the simulation parameter table by the embodiment. It is a figure which shows the processing flow of the data storage process by the embodiment. It is a figure which shows the processing flow of the pseudo data generation process by the embodiment. It is a figure which shows the data structural example of the intermediate | middle table by the embodiment. It is a figure which shows the data structural example of the pseudo data by the embodiment.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a functional block diagram of a pseudo data generation apparatus 1 according to an embodiment of the present invention. In the figure, the pseudo data generation device 1 can be realized by a computer device such as a server or a personal computer, for example, and includes a control unit 10, an input unit 20, an information reading unit 30, a display unit 40, a storage unit 50, and processing. The unit 60 is provided.

  The control unit 10 includes a central processing unit (CPU) and various memories, and controls each unit, temporarily stores data, transfers data, and the like. The input unit 20 is a keyboard, mouse, button, or the like, and receives information input by a user's operation. The display unit 40 is a display such as an LCD (Liquid Crystal Display) or a CRT (cathode ray tube). The information reading unit 30 reads information from a portable recording medium such as a magnetic disk or a CD-ROM. Alternatively, the information reading unit 30 may receive information from another computer device connected via a network.

  The storage unit 50 includes an analysis information storage unit 51, an analysis model storage unit 52, an actual data storage unit 53, an intermediate data storage unit 54, and an extracted data storage unit 55. The processing unit 60 includes an analysis information writing unit 61, An analysis model writing unit 62, a parameter creation unit 63, an analysis information search unit 64, an extracted data output unit 65, a pseudo data generation unit 66, a verification unit 67, and a pseudo data regeneration unit 68 are provided.

  The analysis information writing unit 61 writes in the analysis information storage unit 51 the analysis content that has been performed using the actual data and the analysis information that indicates the result of the analysis based on the analysis content. The actual data is information related to service usage collected in a business system that executes various service business processes. Such services include, for example, sales of products and provision of web pages. Furthermore, when the actual data may be accumulated, the analysis information writing unit 61 writes the actual data read by the information reading unit 30 to the actual data storage unit 53 and also stores the analysis information and the actual data in the analysis information storage unit 51. Write information indicating the association with data.

  The analysis model writing unit 62 writes the analysis model selected by the user in the analysis model storage unit 52. An analysis model is a calculation formula for calculating a probability that a user will use a service based on a set value of a predetermined parameter. Note that the parameters used in the analysis model are associated with the parameters constituting the actual data.

  The parameter creation unit 63 generates simulated parameter information indicating the distribution of the setting values of each parameter used in the analysis model from the actual data read by the information reading unit 30 and writes it into the analysis model storage unit 52. Further, the parameter creation unit 63 writes the association between the analysis information corresponding to the actual data and the simulated parameter information generated based on the actual data in the analysis information storage unit 51.

The analysis information search unit 64 searches the analysis information in the analysis information storage unit 51 according to the conditions input by the user, and displays the search result on the display unit 40.
When the actual data is associated with the analysis information selected by the user, the extracted data output unit 65 writes the actual data to the extracted data storage unit 55, and the simulated parameter information is added to the analysis information selected by the user. Are associated with each other, the pseudo data generated using the simulation parameter information is written into the extracted data storage unit 55.

  The pseudo data generation unit 66 generates pseudo data based on the simulation parameter information associated with the analysis information selected by the user, and writes the pseudo data in the intermediate data storage unit 54. The verification unit 67 generates simulation parameter information for verification using the analysis model from the pseudo data generated by the pseudo data generation unit 66, and the simulation parameter information used when generating the pseudo data, that is, actual data Verify the deviation from the original simulation parameter information generated from. If the verification by the verification unit 67 is unsuccessful, the pseudo data regeneration unit 68 instructs the pseudo data generation unit 66 to regenerate the pseudo data.

Next, an example of each data will be described.
FIG. 2 is a diagram illustrating a data configuration example of actual data input to the pseudo data generation device 1.
In the figure, the actual data is composed of POS (Point Of Sale) data and customer attribute data. The POS data is composed of a plurality of records including information on the date, day of the week and time of the purchase by the customer, the store code of the purchase store, the product code for specifying the purchased product, and the price of the product. An ID for identifying the record is set in the record. The customer attribute data consists of a plurality of records including the sex, age, etc. of the customer, and an ID for identifying the record is set in each record. Hereinafter, the types of parameters constituting the actual data, for example, the date, day of the week, time zone, store code, product code, amount of money, the gender and age of the customer attribute data, etc. will be referred to as “actual data parameters”.

As described above, the actual data is information related to the use of the service collected in a business system that executes processing in various service businesses. That is, the actual data that is the target in the present embodiment is information collected about the information on the person who performed some action such as using a service, and the situation where the action was performed. Alternatively, it may be information collected regarding a situation in which the event has occurred, such as a log of system operation status and a failure that has occurred.
For example, in the case of actual data for web usage, instead of the POS data, information for identifying a record, the date, day of the week, and time zone when the customer accessed the web, and the access destination URL (Universal Resource Locator) The access log data consisting of a plurality of records including can be used.

FIG. 3 is a diagram illustrating a data configuration example of an analysis information table that is input to the pseudo data generation device 1 and stored in the analysis information storage unit 51.
In this figure, the analysis information table is based on the analysis number that identifies the record, the analysis title, the industry / field keyword indicating what industry or field the analysis is for, the purpose of the analysis, the analysis scenario used, and the analysis scenario. It consists of records including analysis information consisting of analysis content, analysis content details, and analysis result details, presence / absence of a simulation parameter table (see FIG. 5 described later), and analysis data storage location. As the analysis data storage location, the storage location of the actual data is set when the simulation parameter table is “none”, and the storage location of the simulation parameter table is set when the simulation parameter table is “present”. In addition, the storage location of information describing them is set in the analysis content and the analysis content details.

  FIG. 4 is a diagram illustrating an example of an analysis scenario. The analysis scenario type shown in FIG. 4 is set in the analysis scenario of the analysis information table shown in FIG. An analysis scenario is a process in which a parameter setting value obtained from actual data or a value determined in accordance with the setting value is substituted into a predetermined formula for analysis to calculate an index value representing some tendency or characteristic. Are performed in a predetermined order. By analyzing the index value thus obtained, it is possible to determine the tendency and characteristics of the target content in the analysis scenario.

FIG. 5 is a diagram illustrating a data configuration example of a simulation parameter table stored in the analysis model storage unit 52.
The simulation parameter table includes simulation parameter information of each parameter used in the analysis model. Hereinafter, parameters used for the analysis model are referred to as “analysis model parameters”. In the figure, the simulation parameter table includes simulation parameter information corresponding to each analysis model parameter such as gender, age, product type, store, time zone, and month, and the simulation parameter information is taken by the analysis model parameter. It is shown that the obtained setting value is associated with a distribution value indicating the distribution of the setting value in actual data.

Next, the operation of the pseudo data generation device 1 according to the present embodiment will be described.
FIG. 6 is a diagram showing a processing flow of data accumulation processing according to the present embodiment.
In the figure, the user inputs an instruction to store actual data or generate simulated parameter information and a storage location of the actual data through the input unit 20 of the simulated data generation apparatus 1 (step S105). . Further, the user inputs analysis information including an analysis title, an industry / field keyword, an analysis purpose, an analysis scenario, an analysis content based on the analysis scenario, an analysis content detail, and an analysis result detail via the input unit 20 (step S110). .

  In step S105, when an instruction to store actual data is input (step S115: accumulation), the information reading unit 30 is based on the storage location of the input actual data via a portable recording medium or a network. The real data is read out from the other computer devices connected to the real data storage unit 53 (step S120). Further, the analysis information writing unit 61 records the analysis information input in step S110, the information specifying the storage area in which the actual data is written in step S120, and the simulation parameter table “none”. The information is added to the analysis information table in the analysis information storage unit 51 (step S125).

  On the other hand, when an instruction to generate simulation parameter information is input in step S105 (step S115: simulation), the user inputs an analysis model to be used through the input unit 20 (step S130). The analysis model writing unit 62 writes the input analysis model in the analysis model storage unit 52.

  The analysis model may be selected from a list stored in advance in the analysis model storage unit 52. The analysis model may or may not be included as part of the analysis scenario. This is because, for example, when using a plurality of analysis models in an analysis scenario, it may be better to apply a simple model than to create pseudo data using all the same analysis model, The analysis model needs to include at least parameters used in analysis scenarios used for data analysis practice and system verification. Here, it is assumed that an analysis model including the following (Expression 1) and (Expression 2) is input.

When the set value x _nk (k is any one of _{1 to} m) is set to the analysis model parameter pn that can be set to m set values x _n1 , x _n2 ,..., X _nm , (pn) _j = 1 (j = k), (pn) _j = 0 (j ≠ k). For example, when the analysis model parameter p1 is “gender”, the setting value x ₁₁ = “male” and the setting value x ₁₂ = “female” can be two values, the setting value “male” is set in “gender”. (P1) ₁ = 1 and (p1) ₂ = 0. Β _{nj represents} a distribution value corresponding to the set value x _nj of the analysis model parameter pn. Β ₇ is a constant term, and t is a numerical value of “time expressed in a predetermined unit”. Note that γ _1i and γ _2i are values determined in accordance with a set value of a certain analysis model parameter pn.

  As described above, the analysis model uses the setting values of each analysis model parameter to calculate the probability of occurrence of the same action or event as when actual data is collected, such as use of a service. Here, the setting values of each analysis model parameter are determined independently, and there is no correlation between the setting values of a plurality of analysis model parameters.

  Subsequently, the user inputs the association between the analysis model parameter and the actual data parameter, and the association between the setting value of the analysis model parameter and the setting value of the actual data parameter through the input unit 20.

For example, the analysis model parameter “gender” corresponds to the actual data parameter “sex” of the customer attribute data, and the setting values “male” and “female” of the analysis model parameter “sex” are the actual data parameters “ Corresponding to the set values “M” and “F” of “sex” are input.
The analysis model parameter “product type” corresponds to the “product code” of the POS data, and the setting values “A”, “B”,... Of the analysis model parameter “product type” are the actual data parameters “ Corresponding to the set values “0001”, “0002”,.

The analysis model parameter “age” corresponds to the actual data parameter “age” of the customer attribute data, and the set values “˜19”, “20-29”,..., “70” ”of the analysis model parameter“ age ”. Are input corresponding to the set values “0” to “19”, “20” to “29”,..., “70” or more of the actual data parameter “age” of the customer attribute data.
Here, the sex, age, product type, store, time zone, and month of the analysis model parameters correspond to the sex of customer attribute data, age, product code of POS data, store code, time zone, and date, respectively. To do.

  Subsequently, the information reading unit 30 reads the actual data from a portable recording medium or another computer device connected via the network based on the storage area of the input actual data, and the parameter creating unit 63. Generates a simulation parameter table based on the actual data (step S135). Here, it is assumed that the actual data shown in FIG. 2 has been read, and the simulation parameter table generation process will be described in detail below.

  When the parameter creation unit 63 sequentially selects the analysis model parameters used in the analysis model selected or input in step S130, the parameter creation unit 63 performs, for each selected analysis model parameter, an actual value corresponding to each set value of the analysis model parameter. The number of records in which the set value of the data parameter is set to the actual data, that is, the set number is counted.

  For example, in the case of the above example, the parameter creation unit 63 sequentially selects the sex, age, product type, store, time zone, and month of the analysis model parameter. When the currently selected analysis model parameter is “gender”, the corresponding actual data parameter, that is, the actual data parameter “sex” of the customer attribute data is specified. The parameter creation unit 63 counts the number of records in which “M” is set in the identified actual data parameter “sex” and the number of records in which “F” is set, and sets the analysis model parameter “sex” respectively. The number of settings for the value “male” and the number of settings for the setting value “female”.

  When the currently selected analysis model parameter is “age”, the corresponding actual data parameter, that is, the actual data parameter “age” of the customer attribute data is specified. The parameter creation unit 63 sets the number of records in which any set value from “0” to “19” is set to the specified actual data parameter “age”, and any set value from “20” to “29” The number of records set,..., The number of records for which any setting value of “70” or more is set is set to the set values “˜19”, “20-29”,. The set number is “70”.

Subsequently, the parameter creation unit 63 standardizes the set number of each set value for each analysis model parameter by the following (Equation 3) to obtain a distribution value. Note that pn is a type of analysis model parameter and takes m set values x _nk (k = 1 to m).

Analysis model parameters pn settings _{x nk} distribution value = {(the set number of the set value _{x nk)} - (set value _x n1 average set number of _{~x nm)}} / (the set number of the set value _x n1 _{~x nm} Standard deviation) (Equation 3)

For example, the analysis model parameter A takes α, β, and γ set values (or set value ranges), and the actual data records in which the set values α, β, and γ are set are 100, 200, and 300, respectively. Suppose there was. In this case, the average set number is (100 + 200 + 300) / 3 = 200, and variance = (standard deviation σ) ² = {(100−200) ² + (200−200) ² + (300−200) ² } / 3 It is. Therefore, the distribution value of the set value α of the analysis model parameter A is (100−200) / σ.
Similarly, when the distribution value when the analysis model parameter A is β and γ is calculated, simulated parameter information of the analysis model parameter A is generated from the calculation result.

  The distribution value may be generated as follows. For example, when all the member information of a store and the POS data in which the attribute information of the purchased member or the code for identifying the member is included in the record are stored as actual data, among the members, the purchased member, the purchase Since it can be acquired as information of members who have not been purchased, the purchase probability for the customer can be calculated. Therefore, it is possible to set the purchase probability in the analysis model and obtain the parameter distribution value by the maximum likelihood estimation method. Specifically, for example, the purchase probability obtained from the actual data is estimated using each of (Equation 2) and (Equation 1). However, when estimating, it is necessary to take measures such as making a model from which one of the parameters is removed in order to prevent a rank drop.

  When the parameter creation unit 63 generates simulation parameter information in which the set value and the distribution value are associated with each analysis model parameter, the parameter creation unit 63 stores a simulation parameter table including the simulation parameter information in the analysis model storage unit 52. In addition to writing (step S140), the information specifying the storage area in which the simulation parameter table is written and the analysis model input in step S130 are written in association with each other (step S145).

  Further, the analysis information writing unit 61 analyzes the record in which the analysis information input in step S110, the information specifying the storage area in which the simulation parameter table is written, and the simulation parameter table “present” are set. It adds to the analysis information table memorize | stored in the information storage part 51 (step S150).

FIG. 7 is a diagram showing a process flow of the pseudo data generation process according to the present embodiment.
In the figure, the user searches the search conditions of data used for data analysis training and system verification, for example, analysis title, industry / field keyword, analysis purpose, analysis scenario, and the like by the input unit 20 of the pseudo data generation device 1. A target and a search term are input (step S205). When the analysis information search unit 64 specifies a record in which the search term indicated by the search condition is set in the search target indicated by the input search condition from the analysis information table in the analysis information storage unit 51, the analysis information search unit 64 specifies The analysis information set in the recorded record is displayed on the display unit 40 (step S210). The user confirms the output analysis information, and when the appropriate analysis information is not displayed (step S215: NO), the process from step S205 is performed again, and the search condition input and the search result display process are performed. repeat.

  When the user selects appropriate analysis information from the displayed analysis information, information specifying the selected analysis information is input by the input unit 20 (step S215: YES). The extracted data output unit 65 identifies the record of the analysis information table in which the analysis information identified by the information input in step S215 is set, and determines the presence / absence of the simulation parameter table and the analysis data storage location from the record. read out. When the simulation parameter table “none” is set (step S220: NO), the extracted data output unit 65 reads the actual data stored in the analysis data storage position from the actual data storage unit 53 and extracts the extracted data. The data is written in the storage unit 55 (step S225).

  On the other hand, when the simulation parameter table “present” is set (step S220: YES), the extracted data output unit 65 outputs the analysis data storage location to the pseudo data generation unit 66 and instructs generation of pseudo data. To do. When the pseudo data generation unit 66 displays a message instructing to input the number of transactions on the display unit 40, the user inputs the number of transactions, that is, the number of records constituting the pseudo data through the input unit 20 (step S230). .

  When the pseudo data generation unit 66 reads the simulation parameter table stored in the analysis data storage position in the analysis model storage unit 52 and the analysis model associated with the simulation parameter table, the following is performed. Pseudo data is generated (step S235).

  First, the pseudo data generation unit 66 generates, for each analysis model parameter included in the read analysis model, an intermediate table composed of records having columns corresponding to setting values that can be taken by the analysis model parameter. To the intermediate data storage unit 54. Furthermore, the pseudo data generation unit 66 writes pseudo data including actual data parameters corresponding to the analysis model parameters in the intermediate data storage unit 54.

  FIG. 8 is a diagram illustrating an example of the intermediate table. For example, when an analysis model composed of (Expression 1) and (Expression 2) is read, the analysis model parameters “sex”, “age”, “product type”, “store”, “time” included in the analysis model are read. For the “band” and “month”, columns corresponding to the setting values that can be taken by each of these analysis model parameters, for example, for the analysis model parameter “gender”, the columns of the setting values “male” and “female”, the analysis model parameter “ Columns of setting values “˜19”, “20-29”,..., “70˜” for “age”, columns of setting values “A”, “B”,. For the model parameter “time zone”, an intermediate table having setting values “0”, “1”,..., “23” columns is generated.

  The pseudo data generation unit 66 adds a new record to the intermediate table. This added record is assumed to be the i-th record in the intermediate table. Therefore, i is the initial value 1 immediately after the intermediate table is generated. For each analysis model parameter of the added record, the pseudo data generation unit 66 randomly selects one setting value that can be taken by the analysis model parameter, sets “1” in the column corresponding to the selected setting value, Set “0” in the column corresponding to the set value.

  For example, in the case of the first record in the intermediate table shown in FIG. 8, “1” is selected in the column corresponding to the setting value “male” randomly selected from the setting values that can be taken by the analysis model parameter “gender”. “0” is set in the column corresponding to the setting value “female” that did not exist. In addition, “1” is displayed in a column corresponding to a setting value “˜19” randomly selected from setting values that can be taken by the analysis model parameter “age”, and other setting values “20 to 29” that are not selected. , “0” is set in the column corresponding to “70˜”. Similarly, for each of the analysis model parameters “product type”, “store”, “time zone”, “store”, “month”, “1” is set only for the column corresponding to the randomly selected setting value. Set “0” in the column corresponding to the value.

Subsequently, the pseudo data generation unit 66 calculates the service use probability using the set value of the intermediate table and the read analysis model. When using an analysis model composed of (Expression 1) and (Expression 2), first, the pseudo data generation unit 66 calculates a preference. β ₁₁ and β ₁₂ correspond to the distribution value “0.5” corresponding to the set value “male” and the set value “female” of the analysis model parameter p1 “sex” in the simulation parameter table shown in FIG. The distribution value is “0.4”. (P1) ₁ is “1” if the set value is “male”, “0” if it is not “male”, and (p1) ₂ is “1” if the set value is “female”, “ Since it is “0” when not “female”, the values set in the columns corresponding to the “gender” setting values “male” and “female” in the intermediate table are “(p1) ₁ ” and “(p1) ₂ ”, respectively. Can be used as the value of "." Therefore, in the case of the first record of the intermediate table shown in FIG. 8, Σ of the first term is “−0.01 × 1 + 0.04 × 0”.

Similarly, β ₂₁ ..., Β ₂₆ , β ₂₇ are distributed values “1.5”,..., Set values “60˜” corresponding to the set values “˜19” of the analysis model parameter p2 “age” shown in FIG. The distribution value “0.2” corresponding to “69” and the distribution value “−0.3” corresponding to the set value “70˜”. Similarly to the above, the values set in the columns corresponding to the set values “˜19”,..., “60-69”, “70˜” of “age” in the intermediate table are “(p2) ₁ ,..., “(P2) ₆ ”, “(p2) ₇ ”. Therefore, in the case of the first record in the intermediate table shown in FIG. 8, the Σ of the second term is “1.5 × 1 +... + 0.2 × 0 + (− 0.3) × 0”.
Note that a set value corresponding to the “time zone” column in which 1 is set is substituted for t. Further, γ ₁₁ and γ ₂₁ are set to predetermined values corresponding to the set values of predetermined analysis model parameters.

As described above, when the pseudo data generation unit 66 calculates the preference V _it by (Equation 1) using the set value of the intermediate table and the distribution value of the simulation parameter table, the calculated preference V with _it, to calculate the purchase probability preference _{p it} by (equation 2). Pseudo data generation unit 66, when generating a uniform random number (0, 1), if uniform random number (0, 1) is less than the purchase probability _{p it} is determined that the purchase is the purchase probability _{p it} more If not, it is determined not to purchase. Then, only if it is determined that the purchase, the generated record to add the pseudo data based on the i-th record of the intermediate table used to calculate the purchase probability p _it.
The uniform random number (0, 1) is a random number from 0 to 1 obtained by generating a natural random number sequence having a predetermined maximum value and dividing it by the maximum value.

FIG. 9 is a diagram illustrating an example of pseudo data. The pseudo data includes a record including setting values of actual data parameters corresponding to each analysis model parameter in the intermediate table. When the pseudo data generation unit 66 determines to purchase based on the purchase probability _pit , when a new record is added to the pseudo data, the column in which “1” is set is identified from each analysis model parameter of the intermediate table. Then, the setting value of the actual data parameter corresponding to the setting value of the column is set in the record added to the pseudo data. However, when a set value of a plurality of actual data parameters corresponds to a set value of one analysis model parameter, one of the corresponding set values of the actual data parameter is randomly selected.
For example, in the case of the setting value “˜19” of the analysis model parameter “age”, the setting value is randomly selected from the setting values “0” to “19” of the corresponding actual data parameter “age”. In addition, in the case of the setting value “January” of the analysis model parameter “month”, a setting randomly selected from the setting values “January 1” to “January 31” of the corresponding actual data parameter “date” Value.

  As described above, when the set values of the actual data parameters “sex”, “age”, “product code”, “time zone”, “date”, “store code” of the pseudo data are set in the added record, The setting values of other actual data parameters determined by the setting values of these actual data parameters are written into the record. For example, the set value for the day corresponding to the set value for the date and the amount corresponding to the set value for the product code are written. Further, the ID is written in the added record.

  For example, when the setting value distribution of the day of the week is desired to be the same as the actual data, the simulation parameter information of the analysis model parameter “day of the week” is generated. Then, when generating the pseudo data record from the record of the intermediate table, the date is selected at random from the dates of the predetermined period corresponding to the day of the week. The predetermined period may be input by the user when generating the pseudo data, and when generating the simulation parameter information from the actual data, the period is acquired from the date included in the actual data and the simulation parameter is acquired. It may be stored in a table.

  As described above, when a record is added to the pseudo data, the pseudo data generation unit 66 determines whether the number of records of the pseudo data has reached the number of transactions. If not, update the value of i to 1 and add the i-th record to the intermediate table, randomly select the setting value of each analysis model parameter, and use the analysis model to If the probability of use is calculated, the use of the service is determined based on the probability, and the service is determined to be used, the record of the setting value based on the i-th record of the intermediate table is stored as pseudo data Repeat the above process to add.

  When the number of records in the pseudo data reaches the number of transactions, the verification unit 67 generates simulation parameter information from the pseudo data using the generated pseudo data and the analysis model by the same process as in step S135 of FIG. (Step S240). That is, when the verification unit 67 sequentially selects the analysis model parameters used in the analysis model, for each selected analysis model parameter, setting of the actual data parameter in the pseudo data corresponding to each setting value of the analysis model parameter Count the number of records that have a value set. Then, a distribution value is created for each setting value of each analysis model parameter, and a verification simulation parameter table including simulation parameter information for each analysis model parameter is generated from the calculation result.

The verification unit 67 uses each of the simulation parameter information in the simulation parameter table used for generating the pseudo data in step S235 by using an existing method for statistically checking whether there is a divergence between the two data groups. The verification parameter information in the verification simulation parameter table generated in S240 is verified.
For this verification, for example, a t-test (test of difference between average values) for statistically checking whether there is a difference between the average values of the two groups can be used. Specifically, the null hypothesis in the t-test is “no difference between the average values of the two groups”, and the calculated t-value is larger than the value of the t-distribution table (for example, 95% confidence interval) in the degree of freedom. If it is small, the null hypothesis is not rejected, and the result is that there is no difference in the mean, and the verification is successful. For the verification, a verification method according to the distribution assumed for the set value of each analysis model parameter is used.

If the verification is unsuccessful (step S245: NO), the processing from step S235 for generating pseudo data is performed again.
When generating the pseudo data again, the pseudo data regenerating unit 68 deletes the record from the pseudo data currently stored in the intermediate data storage unit 54 as follows.

(1) As a result of the verification in step S245, an analysis model parameter for which verification was unsuccessful is specified. When the set value of the analysis model parameter takes a continuous value, it is determined whether the average is above or below. If the average is below, delete the record of the actual data in which the setting value of the actual data parameter corresponding to the setting value smaller than the average value is set, and if the average is above, the average value Records of actual data in which actual data parameter setting values corresponding to larger setting values are set are deleted by a predetermined ratio.
(2) Delete a predetermined number or a predetermined ratio of records at random. For example, it is used when a normal distribution is not taken, such as when the set value is binary.
(3) Delete all records.

  After deleting the record of the pseudo data by any of the above, the pseudo data regenerating unit 68 instructs the pseudo data generating unit 66 to create the pseudo data. As a result, the pseudo data generation unit 66 adds the i-th record to the intermediate table, randomly selects the setting value of each analysis model parameter, calculates the probability of using the service using the analysis model, and If the service is used based on the probability and it is determined that the service is used, the above process of adding a set value record based on the i-th record of the intermediate table to the pseudo data Repeat until the number of data records reaches the number of transactions.

On the other hand, if the verification of the pseudo data is successful in step S245 (step S245: YES), the extracted data writing unit 65 uses the pseudo data generated in step S235 and written in the intermediate data storage unit 54 as the extracted data. Writing to the storage unit 55 (step S250).
The user performs data analysis training and a system test using the pseudo data output to the extracted data storage unit 55.

According to the above embodiment, in order to effectively perform a data analysis practice for the purpose of acquiring accumulated data analysis skills that can make full use of BI, past analysis cases and actual data were generated. The simulation parameter information indicating the set value distribution of the actual data is accumulated in association, and the information selected from the accumulated data as close to the purpose of the analysis is obtained from the simulation parameter information corresponding to the analysis example. Pseudo data can be generated. Therefore, pseudo data can be quickly prepared from past analysis cases, and practical data analysis training can be performed.
In addition, since the pseudo data of the required number of transactions can be generated from the pseudo parameter information, the data storage area can be reduced, and the cost can be reduced.
Further, by verifying the pseudo data once generated, pseudo data closer to the actual data can be generated, and a more realistic data analysis exercise can be performed.
In addition, since multiple analysis cases, which are know-how such as analysis procedures, can be stored in the same format, it is possible to search for and use different types of analysis in a unified manner, making effective analysis possible. It is possible to practice.
Further, by using the pseudo data generated as described above for system verification, a highly accurate verification result can be obtained.

  The pseudo data generation apparatus 1 described above has a computer system therein. The analysis information writing unit 61, the analysis model writing unit 62, the parameter creation unit 63, the analysis information search unit 64, the extracted data output unit 65, the pseudo data generation unit 66, the verification unit 67, and the pseudo data generation device 1 The process of the operation of the pseudo data regenerating unit 68 is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The present invention is not limited to the embodiment described above, and various modifications can be made without departing from the spirit of the present invention, and it goes without saying that these are also included in the scope of the present invention. Yes.

DESCRIPTION OF SYMBOLS 1 ... Pseudo-data production | generation apparatus 10 ... Control part 20 ... Input part 30 ... Information reading part 40 ... Display part 50 ... Storage part 51 ... Analysis information storage part 52 ... Analysis model storage part 53 ... Actual data storage part 54 ... Intermediate data storage 55: Extracted data storage unit 60 ... Processing unit 61 ... Analysis information writing unit 62 ... Analysis model writing unit 63 ... Parameter creation unit 64 ... Analysis information search unit 65 ... Extraction data writing unit 66 ... Pseudo data generation unit 67 ... Verification unit 68 ... Pseudo data regeneration unit

Claims (5)

An information reading section for reading actual data;
A value indicating the distribution of setting values that can be taken by the analysis model parameter used in the analysis model, which is a formula for calculating the probability of occurrence of a predetermined action or event, and setting the parameter in the actual data corresponding to the analysis model parameter A parameter creation unit to calculate from the value;
A setting value that can be taken by the analysis model parameter is randomly determined, and the setting value of the analysis model parameter determined at random using the analysis model and the setting of the analysis model parameter created by the parameter creation unit Whether to calculate the probability of occurrence of the predetermined action or event from the value indicating the distribution of values, and use the set value of the analysis model parameter determined at random for generating pseudo data based on the calculated probability If it is determined to be used for generating pseudo data, the process of determining the parameter setting values constituting the pseudo data based on the randomly determined setting values of the analysis model parameters is repeated, and the pseudo data is A pseudo data generation unit to generate;
A pseudo data generation device comprising:   The pseudo data generation device according to claim 1, wherein the pseudo data generation unit generates set values of parameters constituting the pseudo data for a predetermined number of records. A value indicating a distribution of setting values that can be taken by the analysis model parameter used in the analysis model is calculated from the setting values of the parameter in the pseudo data corresponding to the analysis model parameter, and the distribution of the calculated setting value is calculated. A verification unit that determines whether or not the distribution of the set value deviates from a predetermined value from the value indicating the distribution of the set value generated based on the set value of the parameter in the actual data;
When it is determined that the verification unit has deviated, a pseudo data regeneration unit that deletes part or all of the pseudo data and instructs the pseudo data generation unit to create pseudo data;
The pseudo data generation device according to claim 1, further comprising: A pseudo data generation method used in a pseudo data generation device,
An information reading unit, an information reading unit step for reading actual data;
The fruit parameter creation unit, a value showing the distribution of set values that can be taken for analysis model parameters used in the analysis model is a formula for calculating the probability of a given action or events, corresponding to the analysis model parameters A parameter creation step to calculate from the set value of the parameter in the data;
The pseudo data generation unit randomly determines a setting value that can be taken by the analysis model parameter, and using the analysis model, the setting value of the analysis model parameter determined at random, and created in the parameter creation step, The occurrence probability of the predetermined action or event is calculated from the value indicating the distribution of the setting value of the analysis model parameter, and the analysis model parameter determined at random for generating pseudo data based on the calculated probability A process for determining whether or not to use a set value and determining a set value of a parameter constituting the pseudo data based on the set value of the analysis model parameter determined at random when it is determined that the set value is used for generating pseudo data To generate pseudo data, and to generate pseudo data,
The pseudo data generation method characterized by having. A computer used as a pseudo data generation device,
A value indicating the distribution of setting values that can be taken by an analysis model parameter used in an analysis model, which is a formula for calculating the probability of occurrence of a predetermined action or event, and a parameter setting value in actual data corresponding to the analysis model parameter Parameter creation unit to calculate from
A setting value that can be taken by the analysis model parameter is randomly determined, and the setting value of the analysis model parameter determined at random using the analysis model and the setting of the analysis model parameter created by the parameter creation unit Whether to calculate the probability of occurrence of the predetermined action or event from the value indicating the distribution of values, and use the set value of the analysis model parameter determined at random for generating pseudo data based on the calculated probability If it is determined to be used for generating pseudo data, the process of determining the parameter setting values constituting the pseudo data based on the randomly determined setting values of the analysis model parameters is repeated, and the pseudo data is A pseudo data generation unit to generate,
A computer program that functions as a computer program.

Images