CN115543428A - Simulated data generation method and device based on strategy template - Google Patents

Simulated data generation method and device based on strategy template Download PDF

Info

Publication number
CN115543428A
CN115543428A CN202211345059.5A CN202211345059A CN115543428A CN 115543428 A CN115543428 A CN 115543428A CN 202211345059 A CN202211345059 A CN 202211345059A CN 115543428 A CN115543428 A CN 115543428A
Authority
CN
China
Prior art keywords
data
configuration
template
strategy template
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211345059.5A
Other languages
Chinese (zh)
Inventor
陈俊
韩阳
刘圆
李杨
肖俐平
张茜
张辉
李烨
张冬蕾
杨汉林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chinese People's Liberation Army 31007
Original Assignee
Chinese People's Liberation Army 31007
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chinese People's Liberation Army 31007 filed Critical Chinese People's Liberation Army 31007
Priority to CN202211345059.5A priority Critical patent/CN115543428A/en
Publication of CN115543428A publication Critical patent/CN115543428A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/70—Software maintenance or management
    • G06F8/71—Version control; Configuration management

Abstract

The embodiment of the application belongs to the field of data processing, and relates to a method and a device for generating simulation data based on a strategy template. Wherein the method comprises: initializing the data source and the basic strategy template by acquiring the data source and the basic strategy template, wherein the initialization is used for representing the updating configuration of the basic strategy template, performing visual configuration on the initialized basic strategy template to obtain a target strategy template, and generating target data according to the data source and the target strategy template. According to the technical scheme, the basic strategy model can be automatically initialized based on the data source, the man-made configuration workload is greatly reduced, and meanwhile, the complex requirements of the application layer scene can be configured aiming at the characteristics of the data source by combining visual configuration, so that the generated target data can support the development and the test of the application data service, and the universality is high.

Description

Simulation data generation method and device based on strategy template
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for generating simulation data based on a policy template.
Background
The enterprise collects and aggregates global data through the construction of a data middle station, a data warehouse and the like, and performs data analysis and processing according to a service scene to form an index library, a label library and a layered theme asset. Compared with the traditional database, the data warehouse emphasizes the combination of service requirement driving, shields the bottom complex service, integrates the domain-divided subject data according to each service domain, provides a data service interface based on the data of the application layer, and realizes open sharing.
During the development process of the application software, simulation data is a ring which cannot be lost by a test system. Particularly, in some fields, sensitive and private real data cannot be provided externally, and a simulation data generation tool is generally adopted to generate simulation data similar to the real data, so that the simulation data generated by simulation can test the capabilities of normal operation, calculation processing and the like of the application system. However, some of the related simulation data generation tools focus on solving simulation requests for generating simulation data in a large batch, some focus on solving private simulation data generated when a data simulation request is initiated to the front end, corresponding simulation data generation rules need to be set independently for different simulation requests, simulation data generation efficiency is low, and the related simulation data generation tools have simple default rules and cannot be suitable for simulation data requirements of different scenes.
Disclosure of Invention
The embodiment of the application aims to provide a simulation data generation method and device based on a strategy template so as to solve the problems of low simulation data generation efficiency and poor universality.
In order to solve the above technical problem, an embodiment of the present application provides a method for generating simulation data based on a policy template, which adopts the following technical solutions:
acquiring a data source and a basic strategy template;
initializing a data source and a basic strategy template, wherein the initialization is used for representing the updating configuration of the basic strategy template;
carrying out visual configuration on the initialized basic strategy template to obtain a target strategy template;
and generating target data according to the data source and the target strategy template.
In some embodiments, initializing the data source and the base policy template includes:
analyzing a data source to obtain a data object, wherein the data object comprises data elements and configuration strategies corresponding to the data elements, and the configuration strategies are used for describing mathematical models of the data elements;
taking the data elements and the configuration strategies corresponding to the data elements as configuration parameters;
and updating the basic strategy template according to the configuration parameters to obtain the initialized basic strategy template.
In some embodiments, the data elements include reference relationships describing data elements, and the parsing the data source to obtain the data object includes:
reading a plurality of data elements from a data source;
determining a generation order of a plurality of data elements; in some embodiments, visually configuring the basic policy template to obtain the target policy template includes:
sending the initialized basic strategy template to a terminal, and performing visual display;
responding to a configuration request of a terminal, adding configuration conditions to configuration strategies in the initialized basic strategy template to generate a target strategy template comprises:
when the configuration condition comprises dictionary constraint of the data element, adding the dictionary constraint into the initialized basic strategy template;
when the configuration conditions comprise association constraints among the data elements, adding the association constraints into the initialized basic strategy template;
when the configuration condition includes a distribution function of the data elements, the distribution function is added to the initialized base policy template.
In some embodiments, generating target data from the data source and target policy templates comprises:
determining the generation sequence of each data element from the target strategy template;
connecting original data corresponding to a data source;
and generating target data according to the target strategy template and the original data and the generation sequence.
In order to solve the above technical problem, an embodiment of the present application further provides a simulation data generating apparatus based on a policy template, including:
the acquisition module is used for acquiring a data source and a basic strategy template;
the system comprises an initialization configuration module, a data source and a basic strategy template, wherein the initialization configuration module is used for initializing the data source and the basic strategy template, and the initialization processing is used for representing the updating configuration of the basic strategy template;
the visual configuration module is used for carrying out visual configuration on the initialized basic strategy template to obtain a target strategy template;
and the data generation module is used for generating target data according to the data source and the target strategy template.
In some embodiments, initializing the configuration module comprises:
the data analysis unit is used for analyzing the data source to obtain a data object, wherein the data object comprises data elements and configuration strategies corresponding to the data elements, and the configuration strategies are used for describing mathematical models of the data elements;
the parameter configuration unit is used for taking the data elements and the configuration strategies corresponding to the data elements as configuration parameters;
and the template updating unit is used for updating the basic strategy template according to the configuration parameters so as to obtain the initialized basic strategy template.
In some embodiments, the data parsing unit includes:
a reading subunit for reading a plurality of data elements from a data source;
a determining subunit configured to determine a generation order of the plurality of data elements;
and the reference subunit is used for taking the generation sequence as the reference relation between the data elements.
In some embodiments, the visualization configuration module comprises:
the display unit is used for sending the initialized basic strategy template to the terminal and carrying out visual display;
and the adding unit is used for responding to the configuration request of the terminal and adding configuration conditions to the configuration strategy in the initialized basic strategy template so as to generate the target strategy template.
In some embodiments, the adding unit comprises:
the dictionary constraint subunit is used for adding the dictionary constraint to the initialized basic strategy template when the configuration condition comprises the dictionary constraint of the data element;
the association constraint subunit is used for adding the association constraint to the initialized basic strategy template when the configuration condition comprises the association constraint among the data elements;
and the joint distribution subunit is used for adding the distribution function to the initialized basic strategy template when the configuration condition comprises the distribution function of the data element.
In some embodiments, the data generation module comprises:
the order determining unit is used for determining the generation order of each data element from the target strategy template;
the data connection unit is used for connecting original data corresponding to the data source;
and the data generation unit is used for generating target data according to the target strategy template and the original data and the generation sequence.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which includes a memory and a processor, where the memory stores computer readable instructions, and the processor executes the computer readable instructions to implement the steps of the above simulation data generation method.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the computer-readable instructions implement the steps of the above simulation data generation method.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
the method comprises the steps of initializing a data source and a basic strategy template by acquiring the data source and the basic strategy template, wherein the initialization is used for representing the updating configuration of the basic strategy template, performing visual configuration on the initialized basic strategy template to obtain a target strategy template, and generating target data according to the data source and the target strategy template, so that the basic strategy template can be automatically initialized based on the data source, the workload of manual configuration parameters is greatly reduced, and meanwhile, the configuration can be performed aiming at the characteristics of the data source and aiming at the complex requirements of an application layer scene by combining the visual configuration, so that the generated target data can support the development and the test of application data services, and the universality is enhanced.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is an exemplary system architecture diagram to which the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of a data generation method according to the present application;
FIG. 3 is a flowchart of one embodiment of step S202 in FIG. 2;
FIG. 4 is a schematic block diagram of one embodiment of a data generation apparatus according to the present application;
FIG. 5 is a schematic block diagram of one embodiment of a computer device according to the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof in the description and claims of this application and the description of the figures above, are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the foregoing drawings are used for distinguishing between different objects and not for describing a particular sequential order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein may be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. Various communication client applications, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, motion Picture Experts compression standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, motion Picture Experts compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that the data generation method provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, the data generation apparatus is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
In the related art, the generation approaches of the analog data include random generation methods, fuzzy substitution, offset dithering, and the like. The random generation method lacks consideration on topological relation, logic and the like among data, and does not meet the distribution requirement of using simulation data. Fuzzy replacement and offset dithering are based on real data to carry out fuzzy and offset, the topological structure and the logical relationship of the real data are kept, but the fuzzy replacement and the offset dithering are based on the real data, in some practical scenes, the real data are difficult to obtain or are physically isolated from a test environment due to the requirement of confidentiality, and if the logical relationship and the topological structure of the real data need to be shielded, a shielding strategy needs to be considered.
Therefore, the method for generating the simulation data based on the strategy template can be configured facing to the complex requirements of the application layer scene, so that the generated target data can support the development and the test of the application data service, and the universality is improved.
With continued reference to FIG. 2, a flow diagram of one embodiment of a simulation data generation method according to the present application is shown. The data generation method comprises the following steps:
s201: and acquiring a data source and a basic strategy template.
In this embodiment, the electronic device (for example, the server/terminal device shown in fig. 1) on which the data generation method operates may obtain data connected to the data source in a wired connection manner or a wireless connection manner, and generate target data capable of supporting development and testing of the application data service in combination with the data generation manner of the present application. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G/5G connection, a WiFi connection, a bluetooth connection, a wimax data generation connection, a Zigbee connection, a UWB (ultra wideband) connection, and other wireless connection means now known or developed in the future.
The data source may be a database instance or a metadata description of the database. The data source stores all information for establishing database connection, that is, data on a corresponding database or database server can be called through the name of the data source, the name of the data source can be written into the basic policy template, or the data source and the basic policy template are directly bound and connected, and when the basic policy template is used, data on the database or database server corresponding to the data source can be directly called. The basic strategy template adopts a formal description method for specification, shields differences of different database field types and the like, unifies data distribution description and the like.
The basic strategy template is used for configuring basic generation conditions such as a general rule for data generation, a distribution function adopted by data generation, some category parameters and the like according to a default template of data generated by a preset library-level strategy. The library-level policy is described by a general policy rule generated by all analog data in the library-level range, a subordinate data object which is not specified by a rating in the following steps inherits the rule attribute, and the general policy rule mainly comprises configuration rules such as consistency with the existing data, sampling rate, distribution function and the like.
The basic strategy template mainly comprises configuration parameters such as consistency with the existing data, sampling rate, distribution functions and the like, wherein the distribution types comprise distribution functions such as uniform distribution, gaussian distribution, mixed Gaussian distribution and the like, and a simulation generation general rule describing all data is adopted, so that the specific strategy configuration workload is simplified. For example, the general basic strategy generally defaults to a simple logical relationship, the default data distribution consistency is 100%, the sampling rate is 100%, single-interval uniform distribution is adopted, and when mixed gaussian distribution is adopted, the category parameter K is input.
In the embodiment of the application, a user can set or modify the library-level policy in the visual interface and configure the data source.
S202: and performing initialization processing on the data source and the basic strategy template, wherein the initialization processing is used for representing the updating configuration of the basic strategy template.
The basic strategy template generation refers to the data object description to be simulated for the data source and the library-level strategy instantiation, the definition, declaration and default value setting of the data object related to the data source are carried out, namely, the related service metadata and the technology metadata are defined, and the initialized data object description example is added into the basic strategy template, so that the target strategy template can be further set based on the basic strategy template in the follow-up process and is closer to the requirement of a service scene.
In some embodiments, with continued reference to fig. 3, fig. 3 is a flowchart of a specific embodiment of step S202 in fig. 2, that is, the initialization process for the data source and the basic policy template includes the following steps:
s2021: and analyzing the data source to obtain a data object, wherein the data object comprises data elements and configuration strategies corresponding to the data elements, and the configuration strategies are used for describing mathematical models of the data elements.
Analyzing the data source comprises connecting to a database or a database server according to the connection information of the data source, reading service metadata or technical metadata on the database or the database server, and obtaining a data object by analyzing the structure of the metadata. The metadata is structured data extracted from an information resource and used for explaining the characteristics and the content of the information resource.
When the scene of analyzing the data source is data for which the user does not set the metadata definition in the basic policy template, the data source to be simulated is added, and the data object is obtained by analyzing the data source. Step S2021 may not be performed if the user sets metadata-defined data in the base policy template.
The data object is a set of data elements and configuration policies corresponding to the data elements, and meanwhile, the data object includes data object-level policies, that is, the data object-level policies can automatically generate policy instances of the data object, such as the number of records, according to preset library-level policies. The data elements comprise basic definitions, reference relations and policy instances generated by combining existing data in the database with basic policy rules. The basic definition of the data element includes the data element name, type, whether the primary key is available, permission to be empty, comment, and other field contents. The reference relationship comprises a primary table name, a primary table data element name, a foreign key relationship limit, whether null or not and the like, which the data element depends on. The foreign key relationship definition is, for example, a one-to-one, one-to-many table.
The configuration strategy corresponding to the data elements comprises the distribution type, the distribution parameters and the vacancy rate of the data, the numerical models of the vacancy rate, the distribution parameters and the like, and the values of the distribution parameters are determined according to the library-level strategy and the real data in the data source. For example, the distribution parameter may be an upper and lower limit of an interval, a mean, a variance, and the like, and is not limited herein.
S2022: and taking the data elements and the configuration strategies corresponding to the data elements as configuration parameters.
In the embodiment of the present application, data elements are added to the basic policy template as configuration parameters, data elements that are not set in the basic policy template are generally added, for example, data fields of a specific scene, and a mathematical model related to the configuration policy is used as the configuration parameters.
S2023: and updating the basic strategy template according to the configuration parameters to obtain the initialized basic strategy template.
The data source is analyzed or the metadata is customized, so that the data elements in the data object and the configuration parameters such as the configuration strategy corresponding to the data elements are obtained, and the configuration parameters are updated and calculated according to the consistency with the default setting in the basic strategy target. That is, when a data object is obtained from the basic policy template, the data object may be displayed at the terminal, and a default parameter value corresponding to a configuration parameter is called according to the data element of the data object and the configuration parameter such as the configuration policy corresponding to the data element. The basic strategy template is initialized, so that the configuration of the basic strategy template not only meets the data generation requirement of a scene, but also automatically updates the configuration parameters through default parameter values, and the initialized configuration operation is greatly reduced.
In the embodiment of the present application, the formal description of the initialized basic policy template may be implemented in XML. For example, a set can be used to define an initialized base policy template, where the initialized base policy template = (all data objects), data object = (policy rule of data object, data element), and data element = (data element base definition, reference relationship, configuration policy corresponding to data element).
In some embodiments, the data element includes parsing the data source to obtain the data object, including:
reading a plurality of data elements from a data source;
determining a generation order of the plurality of data elements;
the order of generation is taken as the reference relationship between the data elements.
Specifically, considering that the names of the database tables are usually formulated according to a uniform naming rule, if the data layers and the theme classes where the database tables are located have different prefixes, the order of the database tables is determined as much as possible according to the names of the database tables and the reference dependency order in the generation process so as to form a template with stronger comprehensibility for users. Under the condition that a service metadata description database does not exist, reading according to the name of the database table in a sequencing mode, checking whether foreign key reference exists or not, if no foreign key reference exists, instantiating and generating strategy parameters according to a general strategy, adding the strategy parameters into a data object description template, if foreign key reference exists, storing a directed graph by taking a single table as a node, adding an object description list according to the reference sequence of a final directed graph, and under the condition that a cyclic graph appears, sequentially adding the strategy parameters into a data object description part in a basic strategy template according to the ascending sequence of reference count.
S203: and carrying out visual configuration on the initialized basic strategy template to obtain a target strategy template.
And importing and analyzing the basic strategy template generated in the steps, providing a basic strategy for visual interface adjustment automatic generation, and embodying other constraint conditions, wherein the basic strategy is mainly used for adjusting and refining the configuration strategy of the data elements, and adding the configuration conditions such as dictionary constraint of the data elements, dependency constraint between the data elements, refined distribution function and the like according to requirements.
Illustratively, when the field strategy instance is viewed and set, the relationship bloodlines and the generation strategies of other fields of the selected field are displayed, the set strategy conflict is prompted, and the relationship bloodlines are updated after setting, so that the generation strategy of each field is not set in a multi-head mode. The bottom level of field relationships caches conditional dependencies and associations of fields in a tree structure. The nodes of the tree structure comprise composite nodes and simple nodes, the simple nodes are uniquely determined by one field of the table, the composite nodes are composed of a plurality of simple nodes and represent the joint distribution of a plurality of fields, and the parent nodes and the subtrees of the tree represent the conditional dependency relationship.
According to the technical scheme, a database-level strategy, a table-object-level strategy and a field-element-level multi-level strategy are adopted, a basic strategy template for generating simulation data for use is generated by automatically initializing the basic strategy template through standardized strategy template description, artificial configuration workload is greatly reduced, and meanwhile, specific table and field configuration can be refined aiming at complex data requirements of an application layer scene according to the characteristics of a data source by combining visual detailed configuration, a target strategy template is perfected, so that the generated target data can support application data service development and test, the standardized strategy template description can decouple the requirements of the simulation database, the real database and the application scene, the difference of heterogeneous databases is shielded, and the universality is high.
In some embodiments, visually configuring the initialized basic policy template to obtain the target policy template includes:
sending the initialized basic strategy template to a terminal, and performing visual display;
and responding to a configuration request of the terminal, and adding configuration conditions to the configuration strategy in the initialized basic strategy template to generate a target strategy template.
Specifically, a user inputs or imports configuration conditions of a configuration policy into a basic policy template in a visual interface, the configuration conditions are used as configuration requests and sent to a server, and the server adds the configuration conditions to the configuration policy in the initialized basic policy template according to the configuration conditions. The configuration condition may be a constraint condition, mainly adjustment and refinement of the configuration policy, for example, the added configuration condition may be a dictionary constraint, a dependency constraint between data elements, a refined distribution function, and the like.
Illustratively, the content of the visualization may include expanding the data objects and their fields in the form of a data source tree, each element of which is unique. Checking the strategy examples corresponding to the fields, selecting detailed configuration conditions such as a plurality of field modification settings and the like, generating a template strategy template, wherein adding the configuration conditions can comprise modifying independent distribution of the fields in batch, and modifying joint distribution and condition joint distribution of the fields in combination, regular representation and the like. Specifically, configuration conditions are added to the data source tree check field, including but not limited to the following configuration options:
when the configuration condition comprises dictionary constraint of the data element field, adding the dictionary constraint into the initialized basic strategy template to provide an input dictionary value;
when the configuration condition selects the independent distribution or the joint distribution of the fields, checking whether dictionary constraint exists in the fields, and if so, providing distribution setting based on dictionary values;
when the configuration condition selects the condition distribution of the selected field, it is checked whether the configuration field and the condition field have dictionary constraints, and if so, the distribution setting based on the dictionary value is provided.
And determining the sequence of the table and the field by combining the data source tree and the relation cache tree to generate a target strategy template.
And sequentially processing the tables of the data source tree, and writing fields in the tables into the target strategy template according to the sequence of independent distribution, joint distribution and conditional distribution.
When the processing fields are independently distributed, adding the generation strategy of the fields into an independent distribution node list of a target strategy template;
when the processing fields are in joint distribution, adding a plurality of joint fields in the distribution into a joint distribution node list of the target strategy template;
when the processing field is the condition distribution, finding the relation cache tree where the field is located from the field-tree index table, traversing the relation cache tree, sequentially processing all fields of the branch tree belonging to the current processing table from leaf nodes to tree nodes where the processing field is located, adding all table fields which are not added into the target strategy template in the branch tree into a condition distribution node list of the target strategy template, wherein the condition distribution node list comprises two nodes of independent condition distribution and combined condition distribution.
By the aid of the library-level strategies, the strategies of the data objects and the configuration strategies of the data elements, the simulation generation mode of the data can be controlled on multi-level granularity, the strategy templates can be automatically generated through a small number of library-level strategies, basic strategy templates can be refined on the basis of the data object-level strategies and the configuration strategies of the data elements, manual configuration work is reduced to the maximum extent, and requirements of scenes are met.
In some embodiments, adding a configuration condition to a configuration policy in the initialized base policy template includes:
when the configuration condition comprises dictionary constraint of the data element, adding the dictionary constraint into the initialized basic strategy template;
when the configuration condition comprises the association constraint among the data elements, adding the association constraint into the initialized basic strategy template;
when the configuration condition includes a distribution function of the data elements, the distribution function is added to the initialized base policy template.
In the embodiment of the application, the data elements in the data objects in the visual interface are selected, the dictionary constraint is set for the configuration conditions, the setting mode can set the dictionary enumeration value set through operations such as importing or inputting, and the dictionary enumeration value set is added to the data elements corresponding to the initialized basic policy template. In the embodiment of the present application, the data element to which the dictionary enumeration value set is added may be used as a dictionary data element. Where dictionary data element = (data element base definition, dictionary enumeration value set). In the embodiment of the application, the relationship is generated by setting the association constraint between the plurality of data elements in the visual interface, namely, mainly constraining the data between the plurality of data elements. For example, a platform table, a device table, and a platform-mounted device table exist in the database. The platform-mounted device table records a plurality of types of devices mounted on the platform of the set category. For example, a class 1 platform may be equipped with multiple classes of devices, and a class 1 device may also be equipped with multiple classes of platforms. When the platform carrying device table is set to generate the target basic strategy template, the platform ID value element in the platform carrying device table and the platform ID value in the platform table are set to be in an n:1 relationship, the device ID value element in the platform carrying device table and the device ID value in the device table are set to be in an n:1 relationship, and the device ID value element and the device ID value in the device table are not allowed to be empty. Therefore, when the platform-mounted device table is configured, the set value range of the platform ID field and the set value range of the device ID field in the platform-mounted device table belong to the ID values in the platform table and the device table, respectively.
The distribution function includes, but is not limited to, a joint distribution function, a conditional distribution function, an independent distribution function, and the like.
In the embodiment of the present application, the existence of the data table may require that some data elements appear simultaneously, that is, by setting a joint distribution function of multiple data elements in the same data object in the visualization interface. For example, when the platform-mounted device table has a platform type and a device type element, the data of the type a platform and the type B device needs to exist to meet the application query requirement, and at this time, the joint distribution function can be set as the data ratio of the platform type to the device type.
It should be noted that the target policy template of the same data element in the same data object can be defined only in one place, so as to avoid inconsistency between the data element after repeated definition and the target policy template.
In the embodiment of the present application, a conditional distribution function may also be configured in the visualization interface, that is, when there is a value of a data element that generates a target according to values or states of other data elements, the values or states of the other data elements need to be determined as the conditional distribution function at this time. For example, when the generation of the frequency is related to the belonging band field, and when the generation strategy of the frequency element is set, the distribution type is selected to be multi-interval uniform distribution through the visual interface, and the band element is selected as a condition in the condition selection, and the zone parameter value under the condition is set.
Further, the independent distribution function setting of the single data elements can be configured in the visual interface. The distribution type and the distribution parameters of the single data elements can be manually adjusted in the visual interface, namely, the distribution type and the distribution parameters comprise the number of intervals, the classification number, the upper and lower limits of the intervals, the mean value, the variance and the like related to the distribution parameters. For example, when the peak data volume in the preset time period is to be obtained, the data volume at a certain time point of each month can be obtained by generating the average value by using a preset random generator according to the data volume corresponding to the peak value of each month in the preset time period, and the variance value is set according to the actual requirement.
The method and the device describe the generation rule of the data through the target strategy template, flexibly support the relationship of data elements and the distribution characteristics such as independence, condition and combination of the data elements, and meet the requirement of simulation data generation under the scene requirement.
S204: and generating target data according to the data source and the target strategy template.
And connecting to the original data on the database or the database server corresponding to the data source according to the connection information of the data source. And generating target data in sequence according to the configuration strategy of the data elements in the target strategy template, wherein the target data is simulation data.
Specifically, the generation order of each data element is determined from the target strategy template, and the simulation data of each data source is sequentially generated according to the generation order. And for the fields depending on the subsequent table fields in the table, pressing the table fields into a stack, and sequentially processing the nodes in the stack until the simulation data of all the nodes are generated after all the nodes in the target strategy template are processed in a traversing manner.
The method comprises the steps of initializing a data source and a basic strategy template by acquiring the data source and the basic strategy template, wherein the initialization is used for representing the updating configuration of the basic strategy template, carrying out visual configuration on the initialized basic strategy template to obtain a target strategy template, and generating target data according to the data source and the target strategy template, so that the basic strategy template can be automatically initialized based on the data source, the workload of manual configuration parameters is greatly reduced, and meanwhile, the configuration can be carried out aiming at the characteristics of the data source and aiming at the complex requirements of an application layer scene by combining the visual configuration, so that the generated target data can support the development and the test of application data services, and the universality is enhanced.
In some embodiments, generating target data from the data source and target policy templates comprises:
determining the generation sequence of each data element from the target strategy template;
connecting original data corresponding to a data source;
and generating target data according to the generation sequence according to the target strategy template and the original data.
Since there is a conditional distribution for some data elements, there may be conditional elements on which they depend that have not yet generated data. Therefore, the generation sequence of each data element is determined from the target strategy template, and the data elements are sequentially placed into the queue of the data elements to be generated according to the generation sequence. And according to the generation sequence, generating the simulation data related to the original data by using the data elements generated preferentially according to the corresponding configuration strategy in the target strategy template. And after the data elements generated preferentially execute the generation of the simulation data, extracting the next data element to be generated from the queue, and continuing to execute the generation of the simulation data according to the configuration strategy corresponding to the next data element to be generated. Namely, the generation of the simulation data is sequentially executed according to the generation sequence until the queue is empty, and the generation of the simulation data is completed, wherein the simulation data is the target data.
According to the method and the device, initialization processing and visual configuration are carried out on the basic strategy template and the data source, so that the generation rule of the simulation data is described, and the data relation and the distribution characteristics such as independence, condition and combination of the data relation are flexibly supported, so that the simulation data generation under the scene requirement is met. That is, by being able to automatically read data structures (i.e., data objects) in the source, simulation data generation is applicable to a variety of relational databases; by setting the database-level strategy, the strategy of the data object and the configuration strategy of the data elements, the simulation generation mode of the data can be controlled on the multi-level granularity, the automatically generated basic strategy template can be set through a small number of database-level strategies, the basic strategy template can be refined on the strategy of the data object and the configuration strategy of the data elements, the manual work is reduced to the maximum extent, the scene requirement is met, and the universality and the convenience of the generation of the simulation data are improved.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with computer readable instructions, which can be stored in a computer readable storage medium, and when executed, the processes of the embodiments of the methods described above can be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
With further reference to fig. 4, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a data generating apparatus, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be applied to various electronic devices.
As shown in fig. 4, the data generating apparatus 400 according to this embodiment includes: an acquisition module 401, an initialization configuration module 402, a visualization configuration module 403, and a data generation module 404. Wherein:
an obtaining module 401, configured to obtain a data source and a basic policy template;
an initialization configuration module 402, configured to perform initialization processing on the data source and the basic policy template, where the initialization processing is used to indicate update configuration of the basic policy template;
a visualization configuration module 403, configured to perform visualization configuration on the initialized basic policy template to obtain a target policy template;
and a data generating module 404, configured to generate target data according to the data source and the target policy template.
In some embodiments, the initialization configuration module 402 includes:
the data analysis unit is used for analyzing the data source to obtain a data object, wherein the data object comprises data elements and configuration strategies corresponding to the data elements, and the configuration strategies are used for describing mathematical models of the data elements;
the parameter configuration unit is used for taking the data elements and the configuration strategies corresponding to the data elements as configuration parameters;
and the template updating unit is used for updating the basic strategy template according to the configuration parameters so as to obtain the initialized basic strategy template.
In some embodiments, the data parsing unit includes:
a reading subunit for reading a plurality of data elements from a data source;
a determining subunit configured to determine a generation order of the plurality of data elements;
and the reference subunit is used for taking the generation sequence as the reference relation between the data elements.
In some embodiments, the visualization configuration module 403 includes:
the display unit is used for sending the initialized basic strategy template to the terminal and carrying out visual display;
and the adding unit is used for responding to the configuration request of the terminal and adding configuration conditions to the configuration strategy in the initialized basic strategy template so as to generate the target strategy template.
In some embodiments, the adding unit comprises:
the dictionary constraint subunit is used for adding the dictionary constraint to the initialized basic strategy template when the configuration condition comprises the dictionary constraint of the data element;
the association constraint subunit is used for adding the association constraint to the initialized basic strategy template when the configuration condition comprises the association constraint among the data elements;
and the joint distribution subunit is used for adding the distribution function to the initialized basic strategy template when the configuration condition comprises the distribution function of the data element.
In some implementations, the data generation module 404 includes:
the order determining unit is used for determining the generation order of each data element from the target strategy template;
the data connection unit is used for connecting original data corresponding to the data source;
and the data generation unit is used for generating target data according to the generation sequence according to the target strategy template and the original data.
With regard to the data generation apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated herein.
The embodiment of the application also provides computer equipment. Referring to fig. 5, fig. 5 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 5 comprises a memory 51, a processor 52, a network interface 53 communicatively connected to each other via a system bus. It is noted that only a computer device 5 having components 51-53 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 51 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or D data generation memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 51 may be an internal storage unit of the computer device 5, such as a hard disk or a memory of the computer device 5. In other embodiments, the memory 51 may also be an external storage device of the computer device 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the computer device 5. Of course, the memory 51 may also comprise both an internal storage unit of the computer device 5 and an external storage device thereof. In this embodiment, the memory 51 is generally used for storing an operating system installed in the computer device 5 and various types of application software, such as computer readable instructions of a data generation method. Further, the memory 51 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 52 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 52 is typically used to control the overall operation of the computer device 5. In this embodiment, the processor 52 is configured to execute computer readable instructions stored in the memory 51 or process data, for example, execute computer readable instructions of the data generation method.
The network interface 53 may comprise a wireless network interface or a wired network interface, and the network interface 53 is generally used for establishing a communication connection between the computer device 5 and other electronic devices.
The present application further provides another embodiment, which is to provide a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the data generation method as described above.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields, and all the equivalent structures are within the protection scope of the present application.

Claims (10)

1. A simulation data generation method based on strategy template is characterized by comprising the following steps:
acquiring a data source and a basic strategy template;
initializing the data source and the basic strategy template, wherein the initialization is used for representing the updating configuration of the basic strategy template;
performing visual configuration on the initialized basic strategy template to obtain a target strategy template;
and generating target data according to the data source and the target strategy template.
2. The method of generating simulation data according to claim 1, wherein the initializing the data source and the base policy template comprises:
analyzing the data source to obtain a data object, wherein the data object comprises data elements and configuration strategies corresponding to the data elements, and the configuration strategies are used for describing mathematical models of the data elements;
taking the data elements and the configuration strategies corresponding to the data elements as configuration parameters;
and updating the basic strategy template according to the configuration parameters to obtain an initialized basic strategy template.
3. The method of claim 2, wherein the data elements include reference relationships describing data elements, and the parsing the data source to obtain the data object includes:
reading a plurality of data elements from the data source;
determining a generation order of a plurality of the data elements;
and taking the generation sequence as a reference relation between the data elements.
4. The method for generating simulation data according to claim 1, wherein the visually configuring the initialized basic policy template to obtain a target policy template comprises:
sending the initialized basic strategy template to a terminal, and performing visual display;
and responding to the configuration request of the terminal, and adding configuration conditions to the configuration strategy in the initialized basic strategy template to generate a target strategy template.
5. The simulation data generation method of claim 4, wherein the adding a configuration condition to the configuration policy in the initialized base policy template comprises:
when the configuration condition comprises a dictionary constraint of the data element, adding the dictionary constraint into the initialized basic strategy template;
when the configuration condition comprises an association constraint between data elements, adding the association constraint into the initialized basic strategy template;
when the configuration condition comprises a distribution function of the data elements, adding the distribution function to the initialized basic strategy template.
6. The simulation data generation method of any one of claims 1 to 5, wherein generating target data from the data source and the target policy template comprises:
determining a generation order of each data element from the target policy template;
connecting the original data corresponding to the data source;
and generating target data according to the generation sequence according to the target strategy template and the original data.
7. A simulation data generation device based on strategy template, characterized by comprising:
the acquisition module is used for acquiring a data source and a basic strategy template;
the initialization configuration module is used for performing initialization processing on the data source and the basic strategy template, wherein the initialization processing is used for representing the updating configuration of the basic strategy template;
the visual configuration module is used for carrying out visual configuration on the initialized basic strategy template to obtain a target strategy template;
and the data generation module is used for generating target data according to the data source and the target strategy template.
8. The data generating apparatus of claim 7, wherein the initialization configuration module comprises:
the data analysis unit is used for analyzing the data source to obtain a data object, wherein the data object comprises data elements and configuration strategies corresponding to the data elements, and the configuration strategies are used for describing mathematical models of the data elements;
the parameter configuration unit is used for taking the data elements and the configuration strategies corresponding to the data elements as configuration parameters;
and the template updating unit is used for updating the basic strategy template according to the configuration parameters so as to obtain the initialized basic strategy template.
9. A computer device comprising a memory having computer readable instructions stored therein and a processor which when executed implements the steps of the simulation data generation method of any of claims 1 to 6.
10. A computer-readable storage medium having computer-readable instructions stored thereon which, when executed by a processor, implement the steps of the simulation data generation method of any of claims 1 to 6.
CN202211345059.5A 2022-10-31 2022-10-31 Simulated data generation method and device based on strategy template Pending CN115543428A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211345059.5A CN115543428A (en) 2022-10-31 2022-10-31 Simulated data generation method and device based on strategy template

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211345059.5A CN115543428A (en) 2022-10-31 2022-10-31 Simulated data generation method and device based on strategy template

Publications (1)

Publication Number Publication Date
CN115543428A true CN115543428A (en) 2022-12-30

Family

ID=84719405

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211345059.5A Pending CN115543428A (en) 2022-10-31 2022-10-31 Simulated data generation method and device based on strategy template

Country Status (1)

Country Link
CN (1) CN115543428A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116501757A (en) * 2023-06-20 2023-07-28 鹏城实验室 ER diagram-based simulation data construction method and device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116501757A (en) * 2023-06-20 2023-07-28 鹏城实验室 ER diagram-based simulation data construction method and device
CN116501757B (en) * 2023-06-20 2023-10-03 鹏城实验室 ER diagram-based simulation data construction method and device

Similar Documents

Publication Publication Date Title
AU2016302371A1 (en) Building and managing data-processing attributes for modeled data sources
US10394805B2 (en) Database management for mobile devices
US20220035847A1 (en) Information retrieval
CN111427971B (en) Business modeling method, device, system and medium for computer system
CN103177329A (en) Rule-based determination and validation in business object processing
CN111858615A (en) Database table generation method, system, computer system and readable storage medium
US11615076B2 (en) Monolith database to distributed database transformation
CN113742338A (en) Structured storage system for project acceptance forms
CN113626223A (en) Interface calling method and device
CN115543428A (en) Simulated data generation method and device based on strategy template
CN110109893A (en) The method and apparatus of data modeling and operation
CN117170655A (en) Metadata processing method and device, data processing equipment and storage medium
CN111782820A (en) Knowledge graph creating method and device, readable storage medium and electronic equipment
CN116483707A (en) Test method, test device, test apparatus, test program, and test program
CN116244387A (en) Entity relationship construction method, device, electronic equipment and storage medium
CN116450723A (en) Data extraction method, device, computer equipment and storage medium
CN113076086B (en) Metadata management system and method for modeling model object using the same
CN108205564B (en) Knowledge system construction method and system
CN109800147A (en) A kind of test cases generation method and terminal device
CN115471582A (en) Map generation method and device, computer equipment and storage medium
US11809398B1 (en) Methods and systems for connecting data with non-standardized schemas in connected graph data exchanges
CN117573199B (en) Model difference comparison analysis method, device, equipment and medium
US20240127379A1 (en) Generating actionable information from documents
US20240119045A1 (en) Systems and Methods for Intelligent Database Report Generation
CN117312307A (en) Service data processing method, device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination