CN116975060A - Data construction method, device, terminal equipment and storage medium - Google Patents
Data construction method, device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN116975060A CN116975060A CN202310949418.6A CN202310949418A CN116975060A CN 116975060 A CN116975060 A CN 116975060A CN 202310949418 A CN202310949418 A CN 202310949418A CN 116975060 A CN116975060 A CN 116975060A
- Authority
- CN
- China
- Prior art keywords
- data
- configuration information
- task thread
- data table
- constructed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title claims abstract description 103
- 238000003860 storage Methods 0.000 title claims abstract description 13
- 238000004458 analytical method Methods 0.000 claims abstract description 27
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 27
- 238000003780 insertion Methods 0.000 claims description 17
- 230000037431 insertion Effects 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 12
- 238000004519 manufacturing process Methods 0.000 claims description 9
- 238000002360 preparation method Methods 0.000 abstract description 11
- 238000012360 testing method Methods 0.000 abstract description 11
- 238000013461 design Methods 0.000 abstract description 8
- 238000000034 method Methods 0.000 description 30
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 1
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000001351 cycling effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
Abstract
The application discloses a data construction method, a data construction device, terminal equipment and a storage medium, and belongs to the technical field of nonfunctional testing. The application receives the configuration information sent by the front end; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. Aiming at the test data, the application designs an automatic counting tool with an interface based on an intelligent analysis algorithm so as to improve the intelligent level of data construction and complete the efficient preparation of the data by using a unified platform.
Description
Technical Field
The present application relates to the field of testing technologies, and in particular, to a data construction method, a data construction device, a terminal device, and a storage medium.
Background
With the development and popularization of the internet software technology, the number of users accessing through various terminal devices is increased, and higher requirements are put on the processing capacity of a back-end software system. When the system is used by mass users, how to ensure that a software system efficiently and stably provides services is an unavoidable problem for software manufacturers.
For batch data construction of database tables, common methods include Jmeter number of manufacture, code number of manufacture, writing storage process number of manufacture, using number of manufacture tools provided on the web, etc. Besides the number making tools, the method has high requirements on professional skills of users, the execution efficiency difference of different number making schemes is obvious, the data generation rule is not flexible and intelligent enough, and the efficient preparation of data by utilizing a unified platform cannot be realized. The existing various counting tools provide more choices for users, but have larger defects: the compatibility and mobility are greatly compromised due to different coding languages and implementation difficulties, and the compatibility of multiple types of databases is less likely to be realized.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present application and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The application mainly aims to provide a data construction method, a device, terminal equipment and a storage medium, and aims to design an automatic counting tool with an interface so as to improve the intelligent level of data construction and complete efficient preparation of data by using a unified platform.
In order to achieve the above object, the present application provides a data construction method including:
receiving configuration information sent by a front end;
determining a corresponding data table according to the configuration information;
and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
Optionally, before the step of performing data construction on the data table by the task thread constructed in advance according to the configuration information, the method further includes:
judging whether the data table and the corresponding database exist or not;
if the data table and the corresponding database exist, executing the steps of: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
Optionally, the step of performing data construction on the data table through a task thread constructed in advance according to the configuration information includes:
generating a seed table corresponding to the data table through the task thread based on an intelligent analysis algorithm according to the configuration information;
and inserting the data of the seed table into the data table through the task thread.
Optionally, the step of generating, by the task thread, the seed table corresponding to the data table based on the intelligent analysis algorithm according to the configuration information includes:
acquiring the data quantity to be constructed and the number of data cycle insertion times based on the intelligent analysis algorithm according to the expected data quantity in the configuration information and the data quantity of the data table;
and generating the seed table through the task thread according to the data quantity required to be constructed.
Optionally, the step of generating, by the task thread, the seed table according to the data amount to be structured includes:
and copying the data of the data table by the task thread based on a preset number-of-manufacture rule according to the data quantity to be constructed, and generating the seed table.
Optionally, the step of inserting, by the task thread, the data of the seed table into the data table includes:
and according to the data cycle insertion times, inserting the data cycle of the seed table into the data table through the task thread until the data volume of the data table meets the expected data volume, and terminating the data cycle insertion to obtain the data table after data construction.
Optionally, the step of inserting, by the task thread, the data of the seed table into the data table further includes:
and performing sequence self-increment filling on the field names corresponding to the data table according to the field names of the unique processing table in the configuration information.
Optionally, the step of performing data construction on the data table through a task thread constructed in advance according to the configuration information further includes:
storing the execution information of the task thread through a database;
ending the task thread and sending the execution time of the task thread to the front end.
The embodiment of the application also provides a data construction device, which comprises:
the configuration information receiving module is used for receiving the configuration information sent by the front end;
the data table determining module is used for determining a corresponding data table according to the configuration information;
and the data construction module is used for carrying out data construction on the data table through a task thread constructed in advance according to the configuration information.
The embodiment of the application also provides a terminal device, which comprises a memory, a processor and a data construction program stored on the memory and capable of running on the processor, wherein the data construction program realizes the steps of the data construction method when being executed by the processor.
The embodiment of the application also proposes a computer-readable storage medium on which a data construction program is stored, which, when executed by a processor, implements the steps of the data construction method as described above.
The data construction method, the device, the terminal equipment and the storage medium provided by the embodiment of the application receive the configuration information sent by the front end; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. Aiming at the test data, the application designs an automatic counting tool with an interface based on an intelligent analysis algorithm so as to improve the intelligent level of data construction and complete the efficient preparation of the data by using a unified platform.
Drawings
FIG. 1 is a schematic diagram of functional modules of a terminal device to which a data construction apparatus of the present application belongs;
FIG. 2 is a flow chart of a first exemplary embodiment of a data construction method of the present application;
FIG. 3 is a flow chart of a second exemplary embodiment of a data construction method of the present application;
FIG. 4 is a flow chart of a third exemplary embodiment of a data construction method of the present application;
fig. 5 is a schematic overall flow chart of another exemplary embodiment of the data construction method of the present application.
The achievement of the objects, functional features and advantages of the present application will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The main solutions of the embodiments of the present application are: receiving configuration information sent by a front end; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. Aiming at the test data, the application designs an automatic counting tool with an interface based on an intelligent analysis algorithm so as to improve the intelligent level of data construction and complete the efficient preparation of the data by using a unified platform.
The application relates to the technical terms:
SQL: structured Query Language structured query language, SQL for short, is a special purpose programming language, a database query and programming language for accessing data and querying, updating and managing relational database systems; and is also an extension of the database script file. The structured query language is a high-level, non-procedural programming language that allows users to work on high-level data structures. The method does not require the user to specify a data storage method or the user to know a specific data storage mode, so that different database systems with completely different substructures can use the same structured query language as an interface for data input and management. The structured query language statement can be nested, which gives it great flexibility and powerful functionality.
MySQL is a relational database management system, developed by MySQL AB company in Sweden, belongs to Oracle flag products, and is one of the most popular relational database management systems.
The SQL language used by MySQL is the most commonly used standardized language for accessing databases, and MySQL is generally selected as a website database for development of small and medium-sized websites due to the characteristics of small size, high speed, open source codes and the like.
JDBC, java Database Connectivity, java database connection, is a Java API for executing SQL statements, consisting of a set of classes and interfaces written in the Java language. The system can provide unified access to various relational databases, and accordingly can construct higher-level tools and interfaces, so that database developers can write database application programs, all the standard-oriented targets are realized, and the system has simple, strictly-defined type interfaces and high-performance implementation.
IBM DB2, a set of relational database management systems developed by IBM corporation in the united states, has major operating environments of UNIX (including IBM's AIX), linux, IBM i (legacy OS/400), z/OS, and Windows server versions.
DB2 is mainly applied to a large-scale application system, has better scalability, can support from a mainframe to a single-user environment, and is applied to all common server operating system platforms. DB2 provides a high level of data utilization, integrity, security, recoverability, and small to large scale application execution capabilities with platform-independent basic functions and SQL commands. The DB2 adopts a data grading technology, so that mainframe data can be conveniently downloaded to a LAN database server, a client/server user and an application program based on the LAN can access the mainframe data, and the database localization and remote connection are transparent. DB2 is known to possess a very complete query optimizer, whose external connections improve query performance and support multitasking parallel queries. DB2 has good network support capability, each subsystem can be connected with hundreds of thousands of distributed users, thousands of active threads can be activated at the same time, and the method is particularly suitable for large-scale distributed application systems.
DB2 in addition to the mainstream OS/390 and VM operating systems, and the medium-scale AS/400 system, IBM also provides a DB2 product that spans platforms (including UNIX-based LINUX, HP-UX, sunSolaris, and SCOUnixWare; AS well AS OS/2 operating systems for personal computers, and Microsoft Windows 2000 and its early systems). The DB2 database may be accessed by any application using microsoft open database connectivity (ODBC) interface, java database connectivity (JDBC) interface, or CORBA interface proxy.
Oracle, a world-leading information management software developer, is known for its complex relational database product. The Oracle relational database is the first database in the world to support the SQL language.
Mongodb, a database based on distributed file storage, is written in the C++ language. The method aims to provide an extensible high-performance data storage solution for WEB applications, and has the greatest characteristics that the supported query language is very powerful, the grammar of the method is similar to that of an object-oriented query language, almost most functions similar to the single-table query of a relational database can be realized, and the method also supports indexing of data.
The ymal file is a flexible data format, supports annotation and line-feed symbols, is an intuitive data serialization format capable of being identified by a computer, and is mainly applied to information configuration, database information, account information, log files and the like.
Selecting insertion: INSERT INTO SELECT is an SQL statement that copies data from one table to another in a database.
Sequence self-increasing filling: the SEQUENCE self-populating, a mechanism of self-populating in a database, is a database object that generates successive unique values that can be used to provide self-populating digital values for columns in a table.
A sub-table of: seed tables, tables used in databases to initialize or populate data, are used when creating a database or a particular table.
The embodiment of the application considers that the data generation rule of the related technical scheme is not flexible enough, the labor cost is high, and the execution efficiency and the suitability are low.
Based on the above, the embodiment of the application provides a solution, reduces the professional basic knowledge reserve required for constructing a large amount of data, and greatly improves the efficiency of data construction while simplifying the number-making process.
Specifically, referring to fig. 1, fig. 1 is a schematic diagram of functional modules of a terminal device to which a data construction apparatus of the present application belongs. The data construction means may be a device independent of the terminal device, capable of data processing, which may be carried on the terminal device in the form of hardware or software. The terminal equipment can be intelligent mobile equipment with a data processing function such as a mobile phone and a tablet personal computer, and can also be fixed terminal equipment or a server with a data processing function.
In this embodiment, the terminal device to which the data construction apparatus belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.
The memory 130 stores an operating system and a data construction program, and the data construction device may store the received configuration information and constructed data in the memory 130; the output module 110 may be a display screen, a speaker, etc. The communication module 140 may include a WIFI module, a mobile communication module, a bluetooth module, and the like, and communicates with an external device or a server through the communication module 140.
Wherein the data construction program in the memory 130 when executed by the processor performs the steps of:
receiving configuration information sent by a front end;
determining a corresponding data table according to the configuration information;
and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
judging whether the data table and the corresponding database exist or not;
if the data table and the corresponding database exist, executing the steps of: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
generating a seed table corresponding to the data table through the task thread based on an intelligent analysis algorithm according to the configuration information;
and inserting the data of the seed table into the data table through the task thread.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
acquiring the data quantity to be constructed and the number of data cycle insertion times based on the intelligent analysis algorithm according to the expected data quantity in the configuration information and the data quantity of the data table;
and generating the seed table through the task thread according to the data quantity required to be constructed.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
and copying the data of the data table by the task thread based on a preset number-of-manufacture rule according to the data quantity to be constructed, and generating the seed table.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
and according to the data cycle insertion times, inserting the data cycle of the seed table into the data table through the task thread until the data volume of the data table meets the expected data volume, and terminating the data cycle insertion to obtain the data table after data construction.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
and performing sequence self-increment filling on the field names corresponding to the data table according to the field names of the unique processing table in the configuration information.
Further, the data construction program in the memory 130 when executed by the processor also realizes the following steps:
storing the execution information of the task thread through a database;
ending the task thread and sending the execution time of the task thread to the front end.
The embodiment receives the configuration information sent by the front end through the scheme; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. Aiming at the test data, the application designs an automatic counting tool with an interface based on an intelligent analysis algorithm so as to improve the intelligent level of data construction and complete the efficient preparation of the data by using a unified platform.
The method embodiment of the application is proposed based on the above-mentioned terminal equipment architecture but not limited to the above-mentioned architecture.
Referring to fig. 2, fig. 2 is a flowchart illustrating a first exemplary embodiment of a data construction method according to the present application. The data construction method comprises the following steps:
step S10: receiving configuration information sent by a front end;
the user configures the information used for constructing the data through the front end, and the rear end receives the configuration information sent by the front end.
The scheme provides a visual front-end configuration page, the configuration information of the data construction can be configured after a user logs in the front-end configuration page, and the front-end configuration page sends the configuration information to the back-end for further processing.
Specifically, a user logs in a front-end page, selects a database type needing to be counted, and fills in database connection configuration information: IP address, store name, table name, user name and password, appointed data table name, total data volume, number making task name, and the field name which needs to be uniquely processed in the input table. Wherein selecting the type of database requiring the number of builds comprises: the commonly used relational databases, such as Oracle, mysql, mongoDB, DB2, can be arbitrarily switched among a plurality of database types.
The application provides a simplified tool: and inputting database configuration information at a visual interface, reasonably checking the written information such as database type, table field, data volume and the like, and displaying the total data amount of the current table and the time for the data to fall down for a user by the system after the execution is finished. The backend needs to select appropriate tools and libraries to receive and parse the configuration information sent by the front-end according to specific requirements and technology stacks, and may use various ways to receive the configuration information sent by the front-end, such as HTTP (HyperText Transfer Protocol ) requests, webSocket (full duplex communication protocol based on TCP), message queues, and so on.
Step S20: determining a corresponding data table according to the configuration information;
and determining a data table corresponding to the data structure according to the key table of the number of the needed structures in the configuration information.
After receiving the configuration information sent by the front end, the back end service matches the configuration information of the front end with the corresponding. Ymal configuration file of the back end, and selects the corresponding. Ymal configuration file. The purpose of the matching is to generate corresponding connection information in preparation for a subsequent check.
Step S30: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
And according to the configuration information, carrying out data construction on the data table through the task thread.
After the back end receives the configuration information sent by the front end, a corresponding task object is created, and the back end distributes a task thread for the task of the user, so that the multi-user simultaneous online use can be supported, each counting task is only distributed with one thread for processing, and the tasks are not interfered with each other. The specific rules and analysis algorithms for data construction need to be determined according to the needs and specific situations of users, and data construction can be performed on the data table in a manner such as random generation, template filling, rule generation, existing data derivation and data conversion.
The embodiment receives the configuration information sent by the front end through the scheme; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. Aiming at the test data, the application designs an automatic counting tool with an interface based on an intelligent analysis algorithm so as to improve the intelligent level of data construction and complete the efficient preparation of the data by using a unified platform.
Referring to fig. 3, fig. 3 is a flowchart illustrating a second exemplary embodiment of the data construction method according to the present application.
Based on the first embodiment, a second embodiment of the present application is proposed, which differs from the first embodiment in that:
in this embodiment, before the step of performing data construction on the data table by the task thread that is built in advance according to the configuration information, the method further includes:
step S301: judging whether the data table and the corresponding database exist or not;
step S302: if the data table and the corresponding database exist, executing the steps of: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
Specifically, in order to perform data construction on the data table later, first, it is determined whether the data table and the database corresponding to the data table exist.
Finally, if the data table and the database corresponding to the data table exist, executing the steps: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
Specifically, before the task thread performs data construction, the task thread performs database configuration information reading, and the JDBC (Java Database Connectivity, java database connection) is connected to test database connectivity, and then the SELECT COUNT (1) is performed to check the availability of the data table and COUNT the current data amount of the table.
More specifically, the judging whether the data table and the database corresponding to the data table exist or not can be realized by the following steps:
firstly, connecting to the database, and using database connection information in configuration information;
then, inquiring whether the database exists or not, and checking whether the database exists or not by inquiring related commands or interfaces of the database and taking the database name in the configuration information as a parameter;
then, if the query results indicate that the database exists, then execution continues. Otherwise, the database is not existed;
then, inquiring whether the data table exists, and checking whether the data table exists or not in a database by using a similar inquiring command or interface and taking a table name in configuration information as a parameter;
finally, if the query result shows that the data table exists, the data table is indicated to exist, data construction can be performed on the data table, and the SELECT COUNT (1) is executed to check the availability of the data table and COUNT the current data amount of the table. Otherwise, the data table is not existed, and a message that the data table is not existed is returned to the front end.
According to the embodiment, through the scheme, whether the data table and the corresponding database exist or not is judged; if the data table and the corresponding database exist, executing the steps of: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. By judging the existence of the data table and the database, the precondition of the data construction can be satisfied, the data construction operation is ensured to be carried out on a correct target, the error and the safety risk are reduced, and the feasibility and the success rate of the data construction are improved.
Referring to fig. 4, fig. 4 is a flowchart illustrating a third exemplary embodiment of the data construction method according to the present application.
Based on the first embodiment, a third embodiment of the present application is proposed, which differs from the first embodiment in that:
in this embodiment, the step of performing data construction on the data table by a task thread constructed in advance according to the configuration information includes:
step S303: generating a seed table corresponding to the data table through the task thread based on an intelligent analysis algorithm according to the configuration information;
step S304: and inserting the data of the seed table into the data table through the task thread.
Specifically, firstly, according to the content in the configuration information, generating a seed table corresponding to the data table by using an intelligent analysis algorithm through a task thread, wherein the intelligent analysis algorithm can analyze and calculate the configuration information to obtain the parameter value of the generated seed table.
And finally, inserting the data in the seed table into the data table through the task thread.
Further, as an implementation manner, the step of generating, by the task thread, the seed table corresponding to the data table based on the intelligent analysis algorithm according to the configuration information includes:
step S3031: acquiring the data quantity to be constructed and the number of data cycle insertion times based on the intelligent analysis algorithm according to the expected data quantity in the configuration information and the data quantity of the data table;
step S3032: and generating the seed table through the task thread according to the data quantity required to be constructed.
Specifically, first, according to the expected data amount in the configuration information and the data amount of the data table, the data amount and the number of cyclic insertion are analyzed and calculated on the data table by using an intelligent analysis algorithm, so that the calculated data amount and the number of insertion can cover the expected data demand, and the generation and the insertion of the data are reasonably arranged.
And finally, generating a corresponding seed table through the task thread according to the data quantity required to be constructed.
It should be noted that, when generating the data of the seed table, randomness, uniqueness and rationality of the data are considered to ensure that the generated test data has authenticity and representativeness.
Further, as an implementation manner, the step of generating the seed table by the task thread according to the data volume to be constructed includes:
step S30321: and copying the data of the data table by the task thread based on a preset number-of-manufacture rule according to the data quantity to be constructed, and generating the seed table.
Based on a preset number-of-manufacture rule, copying the data of the data table through a task thread to generate a seed table with the corresponding size of the data quantity to be constructed.
Specifically, after the task thread reads the target table to calculate the data amount M to be constructed, the value a of each bit of M is calculated i Then create a corresponding seed table seed i ，seed i Copying one piece of data in the target table, and then performing self-copying to enable the data volume of the seed table to reach 10 i 。
The intelligent analysis algorithm is designed, a construction number flow is intelligently constructed according to the required data quantity, the size of a seed table and the number of executed circulation times are dynamically planned, millions of data construction can be supported, and the total quantity of construction numbers is ensured to have no deviation.
Further, as an embodiment, the step of inserting, by the task thread, the data of the seed table into the data table includes:
step S3041: and according to the data cycle insertion times, inserting the data cycle of the seed table into the data table through the task thread until the data volume of the data table meets the expected data volume, and terminating the data cycle insertion to obtain the data table after data construction.
In particular, using seed i By INSERT INTO SELECT mode writes a to the data table in a round robin manner i Secondary data. Cycling writing a to a data table i Seed table seed is deleted after secondary data i Judging whether the current data volume of the data table meets the expected data volume or not; if yes, the data cycle is ended and the data table is inserted. If not, the next cycle is entered.
Further, as an embodiment, the step of inserting, by the task thread, the data of the seed table into the data table further includes:
step S3042: and performing sequence self-increment filling on the field names corresponding to the data table according to the field names of the unique processing table in the configuration information.
Specifically, when the data cycle insertion is executed, the self-increment filling of the sequence carried by the database is utilized for the table field needing to be uniquely processed, so that the value of each column of the field is ensured to be unique, and the actual service needs are more met: and the writing of completely unique data into the full-table field is supported, and the actual service requirements are further met.
More specifically, when writing the target table according to the seed table INSERT INTO SELECT, the uniquely processed table field is self-increment filled by using the SEQUENCE of the database itself when writing data, so as to ensure the data uniqueness, and meanwhile, when writing data, one column in the table is selected to use a special character mark.
Further, as an implementation manner, the step of performing data construction on the data table through a task thread constructed in advance according to the configuration information further includes:
step S305: storing the execution information of the task thread through a database;
step S306: ending the task thread and sending the execution time of the task thread to the front end.
Specifically, first, the task thread writes the execution information into the local database.
And finally, ending the task thread and sending the execution time of the task thread to the front end.
Wherein, the execution information includes: user identity, information related to the number of the filled structures, task execution state and error log record, which are convenient for user identification and subsequent cleaning, and avoid the influence of dirty data on normal service use.
And constructing an intelligent analysis algorithm to construct data, supporting simultaneous online use of multiple users, arbitrary switching of multiple database types, and recording user use logs to a database, so that the time consumption for preparing data is greatly reduced, and the online quality of software is further ensured.
According to the scheme, the seed table corresponding to the data table is generated through the task thread based on an intelligent analysis algorithm according to the configuration information; and inserting the data of the seed table into the data table through the task thread. The generated data is ensured to accord with the data amount and field definition of the data table, and the accuracy and the effectiveness of the data construction process are ensured.
As another exemplary embodiment, the overall flow of data construction of the data table based on the intelligent analysis algorithm in this embodiment is shown in fig. 5:
the application provides a front-end and back-end combined system, which comprises a front-end page, a background service and a database, and specifically comprises the following steps:
step a: a plurality of users log in the front-end page designed by the application respectively, then select the database type needing to be created according to own needs, fill in corresponding database connection configuration information comprising IP address, library name, table name, user name, password, created number label table, required data volume and uniquely processed table field, and then submit a request;
step b: the background asynchronous task builds a task pool according to the request, creates corresponding tasks for each user request respectively, and each task is executed by adopting a single thread;
step c: each task thread analyzes the filled database types respectively, then checks the corresponding database connection information and target table information, if the database connection check is not passed or the target table does not exist, returns a corresponding error prompt to the front end, and then jumps to step e;
step d: if the configuration check in the step c is passed, each thread creates a seed table and sets the size of the seed table according to the count making rule provided by the application, records the number of times of circulation needed to be executed for calculating the copy data, and then circularly inserts the seed table data number into the target table each time by adopting a INSERT INTO SELECT mode until the data amount of the target table meets the expectations;
step e: writing corresponding thread execution information into a local database respectively, wherein the method comprises the following steps: user identity, information related to the number of times filled in, task execution state and error log record;
step f: ending the thread and returning the thread execution time to the front end.
Specifically, each make thread performs the steps of:
1. firstly, counting the current data quantity of an Org of a target table before executing a counting task, and setting the current data quantity as SUM;
2. then, the current data quantity SUM of the Org is differenced with the expected data quantity (set as P) to obtain the total quantity M=P-SUM of the needed number;
3. then calculate the number of bits in total M, set as
4. Calculating the value of each bit of M to be a i Represents the number of times of loop execution, a i The formula of (2) is as follows:
5. at each a i Generating corresponding seed table seed according to Org table replication under value i ，seed i The total amount of the table data is 10 i ；
6. In the execution of the counting task, the data is constructed by INSERT INTO SELECT, i.e. using seed i Write a to Org Table cycle i The data increment of the Org table in the number making process is expressed as follows:
the present application integrates the ymal files of various relational databases, and the corresponding program branches can be designated only by checking the corresponding data quantity types during the counting, wherein the database types comprise Oracle, mysql, mongoDB, DB, etc.
Further, when seed i Table completion a i Immediately after the sub-loop writes into the Org table, the thread of execution deletes the seed i The table avoids the dirty table generated in the database from affecting normal use. Then the execution thread judges the difference value between the current data volume and the expected data volume P of the Org table and determines whether to generate a i+1 And seed i+1 . Aiming at the aspect of seed table size selection, in order to avoid the influence of the overlarge seed table on the execution efficiency of the whole number-making task, the maximum data magnitude of the seed table is not more than 10 5 。
Further, when the target table contains a field which needs to be uniquely processed, the corresponding field writing data is filled by adopting the SEQUENCE number of the database. Note that the SEQUENCE value is written directly into the database when the field is of NUMBER type; fields of the non-NUMBER type need to be converted into corresponding types, so that unnecessary problems are avoided;
further, in order to facilitate the user to quickly distinguish the structured data, the user needs to mark the table data by adding keywords, before marking the data, analyzing the types of the fields of the target table, and preferentially selecting the CHAR, VARCHAR or TEXT type table fields with the longest length to mark, for example, writing the values of a certain column in the form of PT- +field values, thereby quickly distinguishing which data are structured in batches. If the target table only has the type fields such as NUMBER/TIMESTAMP/DATE, the target table is not marked;
further, the accounting task thread writes execution information to a local database, comprising: user identity, information related to the number of constructs filled in, task execution status, and error log record. The user can inquire the related count record later, and the table data with the marked information can be cleaned timely after the project is finished, so that dirty data is reduced.
The test data construction method comprises a front-end page and a background system, wherein a user logs in the front-end page to select a database type, and fills in database configuration information comprising an IP address, a library name, a table name, a user name and a password; then selecting a key table needing to be manufactured and a target data volume needing to be manufactured, and simultaneously designating column names needing to be uniquely processed in the table and separating the column names by commas; after submitting the count request, the background system intelligently analyzes the number of cycles required by count according to the filled target data volume, selects the seed table size each time to execute the count task, and selects one column in the table to carry out data marking; and finally, the number is built, and the user information and the task execution information are written into a local database.
According to the technical scheme provided by the application, the front-end page is light in weight, namely, the page is simple in design, and a redundancy-free module increases the use burden of a user; the processing flow is intelligent, a user only needs to fill in database connection information and data requirements, the whole counting process is not limited by the user capacity, and meanwhile, the built intelligent analysis algorithm enables data to be generated rapidly according to set rules; back-end processing parallelization supports simultaneous online use of multiple users; and when logging in, recording each user information and the number of creation condition, including filled configuration information, constructed data volume, created task number and the like, so that the table space resources can be conveniently released after the measurement is finished, and the resource cost is saved. The method reduces the reserve of professional basic knowledge required for constructing a large batch, and greatly improves the efficiency of data construction while simplifying the number-making process.
The embodiment receives the configuration information sent by the front end through the scheme; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. Aiming at the test data, the application designs an automatic counting tool with an interface based on an intelligent analysis algorithm so as to improve the intelligent level of data construction and complete the efficient preparation of the data by using a unified platform.
In addition, an embodiment of the present application further provides a data construction apparatus, where the data construction apparatus includes:
the configuration information receiving module is used for receiving the configuration information sent by the front end;
the data table determining module is used for determining a corresponding data table according to the configuration information;
and the data construction module is used for carrying out data construction on the data table through a task thread constructed in advance according to the configuration information.
The principle and implementation process of the data structure are implemented in this embodiment, please refer to the above embodiments, and are not repeated here.
In addition, the embodiment of the application also provides a terminal device, which comprises a memory, a processor and a data construction program stored on the memory and capable of running on the processor, wherein the data construction program realizes the steps of the data construction method when being executed by the processor.
Because the data construction program is executed by the processor and adopts all the technical schemes of all the embodiments, the data construction program at least has all the beneficial effects brought by all the technical schemes of all the embodiments and is not described in detail herein.
Furthermore, an embodiment of the present application also proposes a computer-readable storage medium having stored thereon a data construction program which, when executed by a processor, implements the steps of the data construction method as described above.
Because the data construction program is executed by the processor and adopts all the technical schemes of all the embodiments, the data construction program at least has all the beneficial effects brought by all the technical schemes of all the embodiments and is not described in detail herein.
Compared with the prior art, the data construction method, the device, the terminal equipment and the storage medium provided by the embodiment of the application receive the configuration information sent by the front end; determining a corresponding data table according to the configuration information; and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance. The method and the device effectively solve the problems that the data generation rule in the prior art is not flexible and intelligent enough, the suitability and the efficiency are low, and the efficient preparation of the data can not be completed by utilizing the unified platform, and an automatic counting tool with an interface is designed based on an intelligent analysis algorithm so as to improve the intelligent level of the data construction and enable the efficient preparation of the data to be completed by utilizing the unified platform.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present application.
The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the application, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (11)
1. A data construction method, characterized in that the data construction method comprises the steps of:
receiving configuration information sent by a front end;
determining a corresponding data table according to the configuration information;
and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
2. The data construction method according to claim 1, wherein the step of constructing the data table by a task thread constructed in advance according to the configuration information further comprises, before:
judging whether the data table and the corresponding database exist or not;
if the data table and the corresponding database exist, executing the steps of: and according to the configuration information, carrying out data construction on the data table through a task thread constructed in advance.
3. The data construction method as claimed in claim 1, wherein the step of constructing the data table by a task thread constructed in advance according to the configuration information comprises:
generating a seed table corresponding to the data table through the task thread based on an intelligent analysis algorithm according to the configuration information;
and inserting the data of the seed table into the data table through the task thread.
4. The data construction method as claimed in claim 3, wherein the step of generating the seed table corresponding to the data table by the task thread based on the intelligent analysis algorithm according to the configuration information comprises:
acquiring the data quantity to be constructed and the number of data cycle insertion times based on the intelligent analysis algorithm according to the expected data quantity in the configuration information and the data quantity of the data table;
and generating the seed table through the task thread according to the data quantity required to be constructed.
5. The data construction method according to claim 4, wherein the step of generating the seed table by the task thread according to the amount of data to be constructed comprises:
and copying the data of the data table by the task thread based on a preset number-of-manufacture rule according to the data quantity to be constructed, and generating the seed table.
6. The data construction method according to claim 5, wherein the step of inserting the data of the seed table into the data table by the task thread comprises:
and according to the data cycle insertion times, inserting the data cycle of the seed table into the data table through the task thread until the data volume of the data table meets the expected data volume, and terminating the data cycle insertion to obtain the data table after data construction.
7. The data construction method according to claim 6, wherein the step of inserting the data of the seed table into the data table by the task thread further comprises:
and performing sequence self-increment filling on the field names corresponding to the data table according to the field names of the unique processing table in the configuration information.
8. The data construction method according to claim 7, wherein the step of constructing the data table by a task thread constructed in advance according to the configuration information further comprises:
storing the execution information of the task thread through a database;
ending the task thread and sending the execution time of the task thread to the front end.
9. A data construction apparatus, the apparatus comprising:
the configuration information receiving module is used for receiving the configuration information sent by the front end;
the data table determining module is used for determining a corresponding data table according to the configuration information;
and the data construction module is used for carrying out data construction on the data table through a task thread constructed in advance according to the configuration information.
10. A data construction apparatus, the apparatus comprising: a memory, a processor and a data construction program stored on the memory and executable on the processor, the data construction program being configured to implement the steps of the data construction method as claimed in any one of claims 1 to 8.
11. A storage medium having stored thereon a data construction program which, when executed by a processor, implements the steps of the data construction method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310949418.6A CN116975060A (en) | 2023-07-28 | 2023-07-28 | Data construction method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310949418.6A CN116975060A (en) | 2023-07-28 | 2023-07-28 | Data construction method, device, terminal equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116975060A true CN116975060A (en) | 2023-10-31 |
Family
ID=88479167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310949418.6A Pending CN116975060A (en) | 2023-07-28 | 2023-07-28 | Data construction method, device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116975060A (en) |
-
2023
- 2023-07-28 CN CN202310949418.6A patent/CN116975060A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3446242B1 (en) | Query plan generation and execution in a relational database management system with a temporal-relational database | |
| US20090254881A1 (en) | Code generation techniques for administrative tasks | |
| US6496833B1 (en) | System and method for generating code for query object interfacing | |
| US7743066B2 (en) | Anonymous types for statically typed queries | |
| DuBois | MySQL cookbook: Solutions for database developers and administrators | |
| JPH08339355A (en) | Method and apparatus for access to processing task executionin distributed system | |
| CN111309760A (en) | Data retrieval method, system, device and storage medium | |
| US20040230584A1 (en) | Object oriented query root leaf inheritance to relational join translator method, system, article of manufacture, and computer program product | |
| WO2000075849A2 (en) | Method and apparatus for data access to heterogeneous data sources | |
| US10977262B2 (en) | Data export job engine | |
| US7610292B2 (en) | Systems and methods for storing a dataset having a hierarchical data structure in a database | |
| US6944797B1 (en) | Method and system for tracing | |
| US9977669B2 (en) | Online computing system | |
| US7840603B2 (en) | Method and apparatus for database change management | |
| CN112035443A (en) | Big data execution method, system, equipment and storage medium based on Linux platform | |
| US8407713B2 (en) | Infrastructure of data summarization including light programs and helper steps | |
| CN113254519B (en) | Access method, device, equipment and storage medium of multi-source heterogeneous database | |
| US11080291B2 (en) | Downloading visualization data between computer systems | |
| US20050080759A1 (en) | Transparent interface to a messaging system from a database engine | |
| CN116975060A (en) | Data construction method, device, terminal equipment and storage medium | |
| US7979431B2 (en) | System and method for automatically modifying a query within a web based database system | |
| US20040172382A1 (en) | System and method for generating a request for information about selected objects | |
| AU2019425532B2 (en) | System and methods for loading objects from hash chains | |
| CN112347794A (en) | Data translation method, device, equipment and computer storage medium | |
| CN112163029A (en) | Virtual database system and query method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |