CN114201490A - Data generation system, method and readable storage medium - Google Patents
Data generation system, method and readable storage medium Download PDFInfo
- Publication number
- CN114201490A CN114201490A CN202111475789.2A CN202111475789A CN114201490A CN 114201490 A CN114201490 A CN 114201490A CN 202111475789 A CN202111475789 A CN 202111475789A CN 114201490 A CN114201490 A CN 114201490A
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- batch
- user
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000003860 storage Methods 0.000 title claims abstract description 12
- 238000010276 construction Methods 0.000 claims abstract description 46
- 230000003993 interaction Effects 0.000 claims description 7
- 238000012360 testing method Methods 0.000 abstract description 26
- 238000013515 script Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000002354 daily effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000001520 comb Anatomy 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data generation system, a method and a readable storage medium, wherein the system comprises: the system comprises a parameter resolver, a metadata resolver, a data construction engine, a data route and a data source adapter; the parameter parser is used for collecting metadata and sending the metadata to the metadata parser; the metadata analyzer is used for analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule; the data construction engine is used for simulating and generating different types of data according to the data construction rules; and the data route is used for dividing different types of data into batch data and stream data and outputting the batch data and the stream data to the data source adapter. The technical scheme provided by the application is extremely strong in configuration flexibility, supports reading data from a text file, facilitates users to accurately control field values, can reversely analyze data, analyzes program output, and facilitates comparison of automatic test scripts.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a data generation system, a data generation method and a readable storage medium.
Background
With the application of the mobile internet in the express logistics industry becoming more and more extensive, more and more users can generate a large amount of online data with various rules. The data is the life of the logistics enterprise. Taking the Zhongtong palm-center as an example, the number of daily users exceeds 35 million, the daily average starting times exceeds 900 million, and a huge user group and a high-frequency use scene put higher requirements on the data performance.
During software development testing, test data is often required. These scenarios include: 1. and (3) back-end development: after a new table is built, database test data needs to be constructed, and interface data is generated and provided for a front end to use; 2. testing the performance of the database, simulating bug on a reproduction line: generating a large amount of bottoming test data, testing the performance of a database, and implementing full link pressure test; simulating the mode of generating data by a user as much as possible based on the simulation reference data of the test environment; 3. and (3) stream data testing: for kafka stream data, test data writes need to be generated constantly.
At present, several data writing databases are generally manufactured manually, but the method has the following disadvantages that
1. Wasting working hours: different data needs to be constructed for the fields of different data types of the table.
2. The data volume is small: if a large amount of data needs to be constructed, manual construction of the data is impossible.
3. Not accurate enough: for example, it is necessary to construct a mailbox (to satisfy a certain format), a telephone number (a certain number of digits), an ip address (a fixed format), an age (which cannot be a negative number, and has a size range), and the like. There are certain limits or rules to these test data, and manual construction may not meet the data range or some format requirements, resulting in a back-end procedure error.
4. Multi-table association: manually generated data is small in size, and primary keys used in a plurality of tables are not necessarily related to each other or are not related to each other.
5. Dynamic random writing: for example, for streaming data, kafka needs to be written randomly every few seconds. Or mysql is inserted randomly and dynamically, the manual operation is relatively troublesome, and the number of written data pieces is not good.
No management platform is adopted: the test scene, the execution device and the tested object can not be managed uniformly. The comparison of performance data between different versions is difficult, and the performance trend analysis of the two versions is not easy to be carried out.
6. Test management platformization: and integrating and accumulating the data is not done. And (4) test data, execution items, tested objects and data source test data are subjected to platform.
Disclosure of Invention
To overcome, at least to some extent, the problems in the related art, a data generation system, method and readable storage medium are provided.
According to a first aspect of embodiments of the present application, there is provided a data generation system, the system comprising: the system comprises a parameter resolver, a metadata resolver, a data construction engine, a data route and a data source adapter;
the parameter analyzer is used for collecting metadata and sending the metadata to the metadata analyzer;
the metadata analyzer is used for analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule;
the data construction engine is used for simulating and generating different types of data according to the data construction rules;
and the data route is used for dividing the different types of data into batch data and stream data and outputting the batch data and the stream data to a data source adapter.
Further, the parameter parser is specifically configured to: and analyzing data generated by interaction of the user and the data platform to obtain the metadata.
Further, the metadata parser is specifically configured to:
extracting target metadata from the metadata according to a metadata source specified by a user;
and analyzing the text content of the target metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule.
Further, the data routing is specifically configured to:
dividing the different types of data into batch data and stream data according to different data output types;
and converting the formats of the batch data and the streaming data into a format specified by a user, and outputting the batch data and the streaming data which are converted into the format specified by the user to a data source adapter.
Further, the data source adapter is adapted to a plurality of data sources.
According to a second aspect of embodiments of the present application, there is provided a data generation method, the method including:
collecting metadata;
analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule;
according to the data construction rule, simulating to generate different types of data;
and dividing the different types of data into batch data and stream data, and outputting the batch data and the stream data to each data source.
Further, the collecting metadata includes:
and analyzing data generated by interaction of the user and the data platform to obtain the metadata.
Further, the analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule includes:
extracting target metadata from the metadata according to a metadata source specified by a user;
and analyzing the text content of the target metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule.
Further, the dividing the different types of data into batch data and stream data and outputting the batch data and the stream data to each data source includes:
dividing the different types of data into batch data and stream data according to different data output types;
and converting the formats of the batch data and the streaming data into the formats specified by the user, and outputting the batch data and the streaming data which are converted into the formats specified by the user to each data source.
According to a third aspect of the embodiments of the present application, there is provided a readable storage medium, on which an executable program is stored, the executable program, when executed by a processor, implementing the steps of the data generation method provided in the above technical solution.
By adopting the technical scheme, the invention can achieve the following beneficial effects: the metadata analyzer analyzes the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule, the data construction engine simulates and generates different types of data according to the data construction rule, the data routing divides the different types of data into batch data and streaming data and outputs the batch data and the streaming data to the data source adapter, the configuration flexibility is strong, the data reading from a text file is supported, a user can conveniently and accurately control field values, the data can be reversely analyzed, the output of a program is analyzed, and the comparison of an automatic test script is facilitated.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a block diagram illustrating the structure of a data generation system in accordance with an exemplary embodiment;
FIG. 2 is a diagram illustrating identification of production environment and test environment database table structure inconsistencies, according to an illustrative embodiment;
FIG. 3 is a schematic diagram illustrating table data change monitoring in accordance with an exemplary embodiment;
FIG. 4 is a flow chart illustrating a method of data generation according to an example embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the examples given herein without any inventive step, are within the scope of the present invention.
FIG. 1 is a block diagram illustrating the structure of a data generation system, as shown in FIG. 1, according to an exemplary embodiment, including: the system comprises a parameter resolver, a metadata resolver, a data construction engine, a data route and a data source adapter;
the parameter parser is used for collecting metadata and sending the metadata to the metadata parser;
the metadata analyzer is used for analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule;
the data construction engine is used for simulating and generating different types of data according to the data construction rules;
and the data route is used for dividing different types of data into batch data and stream data and outputting the batch data and the stream data to the data source adapter.
In particular, the different types of data generated in the data construction engine may include, but are not limited to: self-increment id, integer, floating point, date, enumeration, string, name, street, mailbox, telephone number, IP, and short, among others.
According to the data generation system provided by the embodiment of the invention, metadata is collected through the parameter analyzer and is sent to the metadata analyzer, the metadata analyzer analyzes the metadata according to the preset metadata generation rule to obtain table field metadata and a data construction rule, the data construction engine simulates and generates different types of data according to the data construction rule, the data routing divides the different types of data into batch data and streaming data and outputs the batch data and the streaming data to the data source adapter, the configuration flexibility is strong, the data reading from a text file is supported, a user can conveniently and accurately control field values, the data can be reversely analyzed, the output of a program can be analyzed, and the comparison of automatic test scripts is convenient.
Further, the parameter parser is specifically configured to: and analyzing data generated by interaction of the user and the data platform to obtain metadata.
In some embodiments, data generated by user interaction with the data platform, entered from a client command line or page, may be parsed, but is not limited to.
Further, the metadata parser is specifically configured to:
extracting target metadata from the metadata according to a metadata source specified by a user;
and analyzing the text content of the target metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule.
In some embodiments, the metadata source may be, but is not limited to, from a local file, a remote database table, or a custom thread number.
It should be noted that, in the embodiments of the present invention, the "preset metadata generation rule" is not limited, and in some embodiments, the preset metadata generation rule may be set by a person skilled in the art according to actual needs.
Further, the data routing is specifically configured to:
dividing different types of data into batch data and stream data according to different data output types;
and converting the formats of the batch data and the streaming data into the format specified by the user, and outputting the batch data and the streaming data which are converted into the format specified by the user to a data source adapter.
It will be appreciated that an output format converter is included in the data routing for converting the formats of the bulk data and the streaming data to a user specified format. In some embodiments, the user-specified format may include, but is not limited to: text, JSON, XML, CSV, SQL, Excel, or ProtoBuftext, etc.
Specifically, the data routing is further configured to: the generation frequency of stream data is set.
It is understood that the data routing further includes: and the timer is used for setting the generation frequency of the stream data.
Further, the data source adapter is adapted to a plurality of data sources.
In particular, the data source adapter is extensible, i.e., the data source can be extended.
In some embodiments, the data source adapted by the data source adapter may include, but is not limited to: myaql, Hive, Kafka, Mongo, ES, File, etc.
According to the data generation system provided by the embodiment of the invention, metadata is collected through the parameter analyzer and is sent to the metadata analyzer, the metadata analyzer analyzes the metadata according to the preset metadata generation rule to obtain table field metadata and a data construction rule, the data construction engine simulates and generates different types of data according to the data construction rule, the data routing divides the different types of data into batch data and streaming data and outputs the batch data and the streaming data to the data source adapter, the configuration flexibility is strong, the data reading from a text file is supported, a user can conveniently and accurately control field values, the data can be reversely analyzed, the output of a program can be analyzed, and the comparison of automatic test scripts is convenient.
The data generation system provided by the embodiment of the invention carries out platformization on test data, a generation rule, a data type and timing time of batch generation data; powerful grammar, grouping, interval, step length, circulation, randomness, formatting, functions, suffixes and the like are provided, and the configuration flexibility is strong; the data reading from the text file is supported, and the user can conveniently and accurately control the field value; multiplexing the definitions by using a pre-made sequence (sequences), instances (instances), configuration (config) to solve the definition of the complex data format; supporting a plurality of output formats of text, JSON, XML, CSV, SQL, Excel and ProtoBuf; the data can be reversely analyzed, the output of the program can be analyzed, and the automatic test scripts can be conveniently compared.
The data generation system provided by the embodiment of the invention can simulate and generate most common data types for the construction of test data of multiple data sources, can intuitively and more effectively generate performance data in a self-defined manner, and improves the quality assurance for projects through tests. The method comprises the following specific steps:
the first point is as follows: as shown in fig. 2, a user can identify that the table structures of the production environment and the test environment database are not consistent through the system, and obtain a DDL statement corrected by the table structure, so as to ensure that the table structures of the production environment and the test environment are completely consistent;
and a second point: data tracking and rollback are realized, a user needs to add, delete, modify and check data every day, data is tampered due to program bug or misoperation, and the user cannot easily find out when the data is tampered and cannot recover the data due to the previous data tampering. The page of the data generation system provided by the embodiment of the invention can have the functions of data tracking and data rollback, search the whole library, a specific table or a change log of a certain record, and restore the data to the time point before the misoperation according to the influence range of the misoperation.
And a third point: as shown in fig. 3, the data generating system provided by the embodiment of the present invention may be used to monitor the change of the table data.
A fourth point: in the prior art, data is usually found out through SQL, then the data is imported into EXECL, and static charts such as a broken line chart and a pie chart are made, so that the process is relatively complicated. The data generation system provided by the embodiment of the invention can draw the chart based on the SQL result set directly, and can also make a plurality of high-level charts, such as dynamic charts, ring ratios, personalized Tooltip and the like, so that project leaders can finish the work with high quality.
And fifth, the method comprises the following steps: the performance of the database is optimized in real time, and performance optimization of the database in the prior art needs to be based on long-time detailed monitoring records and can be better and effectively optimized and improved by carrying out detailed analysis and abnormal positioning. The data generation system provided by the embodiment of the invention provides second-level monitoring of database performance, and comprises indexes such as select/insert/update/delete, active connection number and network flow, so that the performance fluctuation is avoided. In addition, SQL display can be operated in the database session, session killing is also supported, and session classification statistics can help a user to quickly locate an abnormal SQL source.
And a sixth point: more convenient data operation, the product, PM or business in the prior art need to complete SQL operation through a convenient and comprehensive product, store common operation data and apply to specific business. According to the data generation system provided by the embodiment of the invention, through the function of opening the table, a user can operate the table data in a similar EXECL mode, and the table data can be subjected to addition, deletion, modification, check and statistical analysis without knowing SQL; the business can also customize SQL, and the user can save the commonly used business SQL and directly apply the SQL in managing other databases/instances.
The seventh point is that: SQL is multiplexed, in the prior art, SQL is almost executed when a user accesses a database, SQL is simply queried, but for complex analysis or query with business logic, the cost of rewriting each time is too high, the SQL is stored in a text, the corresponding relation needs to be maintained continuously, and the SQL cannot be used anytime and anywhere. The data generation system provided by the embodiment of the invention can store the common SQL to the system, the SQL multiplexing is not limited by local storage, the multiplexing range is flexible, and the data generation system is not limited by a current database, a current example or all examples.
And eighth point: the visual display of the structure of the base table, when the product designs a new business table or combs the existing business table in the prior art, the structure of all tables in the database is often to be known integrally, and the user can display and paste the tables one by one through commands before, but is not visual enough, and can also display and screen the tables through the tool visualization, but is very inconvenient. According to the data generation system provided by the embodiment of the invention, a user generates a document function through the data generation management platform, can generate a table structure of the whole database by one key, can browse online and can also be led into formats such as Word, Execl and PDF.
An embodiment of the present invention further provides a data generating method, as shown in fig. 4, where the method may be used in a terminal, but is not limited to, the method includes the following steps:
step 101: collecting metadata;
step 102: analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule;
step 103: according to the data construction rule, simulating to generate different types of data;
step 104: different types of data are divided into bulk data and stream data, and the bulk data and the stream data are output to respective data sources.
Further, step 101 collects metadata, including:
and analyzing data generated by interaction of the user and the data platform to obtain metadata.
Further, the step 102 of analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule includes:
extracting target metadata from the metadata according to a metadata source specified by a user;
and analyzing the text content of the target metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule.
Further, the step 104 of dividing the different types of data into batch data and stream data and outputting the batch data and the stream data to each data source includes:
dividing different types of data into batch data and stream data according to different data output types;
and converting the formats of the batch data and the streaming data into the format specified by the user, and outputting the batch data and the streaming data which are converted into the format specified by the user to each data source.
According to the data generation method provided by the embodiment of the invention, the metadata is acquired and analyzed according to the preset metadata generation rule to obtain the table field metadata and the data construction rule, different types of data are generated in a simulation mode according to the data construction rule, the different types of data are divided into batch data and streaming data, and the batch data and the streaming data are output to each data source.
The embodiment of the present invention further provides a readable storage medium, on which an executable program is stored, and when the executable program is executed by a processor, the steps of the data generation method provided by the foregoing embodiment are implemented.
It is to be understood that the system embodiments provided above correspond to the method embodiments described above, and corresponding specific contents may be referred to each other, which are not described herein again.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A data generation system, the system comprising: the system comprises a parameter resolver, a metadata resolver, a data construction engine, a data route and a data source adapter;
the parameter analyzer is used for collecting metadata and sending the metadata to the metadata analyzer;
the metadata analyzer is used for analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule;
the data construction engine is used for simulating and generating different types of data according to the data construction rules;
and the data route is used for dividing the different types of data into batch data and stream data and outputting the batch data and the stream data to a data source adapter.
2. The system of claim 1, wherein the parameter parser is specifically configured to: and analyzing data generated by interaction of the user and the data platform to obtain the metadata.
3. The system of claim 1, wherein the metadata parser is specifically configured to:
extracting target metadata from the metadata according to a metadata source specified by a user;
and analyzing the text content of the target metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule.
4. The system of claim 1, wherein the data routing is specifically configured to:
dividing the different types of data into batch data and stream data according to different data output types;
and converting the formats of the batch data and the streaming data into a format specified by a user, and outputting the batch data and the streaming data which are converted into the format specified by the user to a data source adapter.
5. The system of claim 1, wherein the data source adapter is adapted to accommodate a plurality of data sources.
6. A method of data generation, the method comprising:
collecting metadata;
analyzing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule;
according to the data construction rule, simulating to generate different types of data;
and dividing the different types of data into batch data and stream data, and outputting the batch data and the stream data to each data source.
7. The method of claim 6, wherein the collecting metadata comprises:
and analyzing data generated by interaction of the user and the data platform to obtain the metadata.
8. The method of claim 6, wherein parsing the metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule comprises:
extracting target metadata from the metadata according to a metadata source specified by a user;
and analyzing the text content of the target metadata according to a preset metadata generation rule to obtain table field metadata and a data construction rule.
9. The method of claim 6, wherein the dividing the different types of data into bulk data and stream data and outputting the bulk data and stream data into respective data sources comprises:
dividing the different types of data into batch data and stream data according to different data output types;
and converting the formats of the batch data and the streaming data into the formats specified by the user, and outputting the batch data and the streaming data which are converted into the formats specified by the user to each data source.
10. A readable storage medium having stored thereon an executable program, characterized in that the executable program, when executed by a processor, implements the steps of the data generating method of any one of claims 6-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111475789.2A CN114201490A (en) | 2021-12-06 | 2021-12-06 | Data generation system, method and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111475789.2A CN114201490A (en) | 2021-12-06 | 2021-12-06 | Data generation system, method and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114201490A true CN114201490A (en) | 2022-03-18 |
Family
ID=80650787
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111475789.2A Pending CN114201490A (en) | 2021-12-06 | 2021-12-06 | Data generation system, method and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114201490A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140007056A1 (en) * | 2012-06-28 | 2014-01-02 | Maxim Leizerovich | Metadata-based Test Data Generation |
| CN112199302A (en) * | 2020-12-07 | 2021-01-08 | 望海康信(北京)科技股份公司 | Test data generation method and system, corresponding equipment and storage medium |
| CN113672671A (en) * | 2020-05-15 | 2021-11-19 | 西安京迅递供应链科技有限公司 | Method and device for realizing data processing |
-
2021
- 2021-12-06 CN CN202111475789.2A patent/CN114201490A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140007056A1 (en) * | 2012-06-28 | 2014-01-02 | Maxim Leizerovich | Metadata-based Test Data Generation |
| CN113672671A (en) * | 2020-05-15 | 2021-11-19 | 西安京迅递供应链科技有限公司 | Method and device for realizing data processing |
| CN112199302A (en) * | 2020-12-07 | 2021-01-08 | 望海康信(北京)科技股份公司 | Test data generation method and system, corresponding equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 咋老: "测试数据生成工具datafaker", pages 1 - 3, Retrieved from the Internet <URL:https://zhuanlan.zhihu.com/p/85554716> * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8984485B2 (en) | Analysis of source code changes | |
| CN108038222B (en) | System of entity-attribute framework for information system modeling and data access | |
| CN109254907B (en) | Java-based interface test report generation method and system | |
| CN108959433A (en) | A kind of method and system extracting knowledge mapping and question and answer from software project data | |
| US8417690B2 (en) | Automatically avoiding unconstrained cartesian product joins | |
| CN101996131A (en) | Automatic test method and automatic test platform for graphic user interface (GUI) based on x extensive makeup language (XML) packaging key word | |
| CN106293891B (en) | Multidimensional investment index monitoring method | |
| US10474675B2 (en) | Explain tool for optimizing SPARQL queries | |
| CN111459698A (en) | Database cluster fault self-healing method and device | |
| CN110019116B (en) | Data tracing method, device, data processing equipment and computer storage medium | |
| CN113986241B (en) | Configuration method and device of business rules based on knowledge graph | |
| CN111078729A (en) | Medical data tracing method, device, system, storage medium and electronic equipment | |
| CN109376153A (en) | System and method for writing data into graph database based on NiFi | |
| CN111949540A (en) | Code change risk estimation and verification method and device | |
| CN114253995B (en) | Data tracing method, device, equipment and computer readable storage medium | |
| CN110351131B (en) | Monitoring method and device for distributed link and electronic equipment | |
| CN116661756A (en) | Object analysis method and device based on low-code DSL | |
| CN111966587A (en) | Data acquisition method, device and equipment | |
| CN110909017B (en) | Data analysis method and system | |
| CN109977104B (en) | Data management method and device | |
| CN114201490A (en) | Data generation system, method and readable storage medium | |
| CN115576831A (en) | Test case recommendation method, device, equipment and storage medium | |
| CN115114299A (en) | Method for realizing metadata management based on Flink SQL | |
| CN111159203B (en) | Data association analysis method, platform, electronic equipment and storage medium | |
| CN114385155A (en) | vue project visualization tool generation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |