CN113641572A - Massive big data calculation development debugging method based on SQL - Google Patents
Massive big data calculation development debugging method based on SQL Download PDFInfo
- Publication number
- CN113641572A CN113641572A CN202110750626.4A CN202110750626A CN113641572A CN 113641572 A CN113641572 A CN 113641572A CN 202110750626 A CN202110750626 A CN 202110750626A CN 113641572 A CN113641572 A CN 113641572A
- Authority
- CN
- China
- Prior art keywords
- sql
- debugging
- data
- calculation
- base table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/362—Software debugging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a massive big data calculation and development debugging method based on SQL, which comprises the steps of obtaining original SQL sentences and debugging information of calculation index data; the debugging information comprises debugging requirements and debugging requirements; judging whether the obtained original SQL statement needs to be debugged or not according to the debugging information; the method comprises the steps of reforming the acquired original SQL statement, generating debugging SQL and the like. The invention supports the index calculation of formal SQL sentences and debugging SQL simultaneously to obtain index data, and the data in the two modes are written into different ground libraries to achieve the aim that the index data do not interfere with each other, thereby ensuring the elegance of debugging, leading SQL developers to write more conveniently and quickly, having less cost investment, leading the SQL calculation index to be more easily and stably upgraded, avoiding the dirty data interference generated by a data user in the upgrading process, and effectively reducing the hardware cost.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a massive big data calculation and development debugging method based on SQL.
Background
In the process of offline index calculation of big data and more mature stream batch integration index calculation, data calculation engines supporting SQL statement standards, such as hive SQL, spark SQL, flink SQL and the like, are often used for supporting index calculation storage, and an index data set is provided for supporting query of a display system of each data, so that more and more accurate data decision bases and service enabling are provided for enterprises. The SQL calculation in the big data cluster is characterized in that data is obtained from an input source, index data is generated after index definition SQL calculation and is stored in an output source, and then the data is obtained from the output source and is used for displaying reports and charts and providing data analysis. However, in the current SQL compiling and developing process, different data environments generate different source data, and development and use personnel cannot directly develop SQL statements in the production environment for reasons of safety, data isolation and the like when developing calculation indexes; if the SQL statement error cannot be found in time in the development environment, the SQL statement error can be found only after the execution of the production environment fails or the log alarms, and even the phenomenon that the SQL statement error cannot be found and unreasonable error data is displayed occurs, which brings wrong data strategy guidance to analysis and use personnel and causes economic loss. If the correctness of SQL calculation is better ensured, the environmental cost is increased, and more development time and developers are invested.
Disclosure of Invention
The invention aims to solve the problems and provides a massive big data calculation and development debugging method based on SQL (structured query language) so as to check the legality of SQL sentences and the correctness of index data according to real source data in a production environment, protect formal database data from being polluted, improve the development efficiency fundamentally and reduce the input cost of personnel.
The purpose of the invention is realized by the following technical scheme: a massive big data calculation development debugging method based on SQL comprises the following steps:
s1: acquiring original SQL sentences and debugging information of the calculation index data; the debugging information comprises debugging requirements and debugging requirements;
s2: judging whether the obtained original SQL statement needs to be debugged or not according to the debugging information; modifying the acquired original SQL statement to generate debugging SQL, and executing the step S3; otherwise, go to step S3;
s3: submitting the original SQL statement or the debugging SQL generated by the transformation in the step S2 to an SQL calculation engine of the cluster for SQL index calculation; obtaining SQL calculation index data after the original SQL statement is calculated, and obtaining debugging SQL calculation index data after the debugging SQL is calculated;
s4: outputting a calculation result; the SQL calculation index data is written into a formal landing library, and the debugging SQL calculation index data is written into a debugging landing library;
s5: and respectively providing the data in the official landing library and the data in the debugging landing library for different users.
The method for reconstructing the acquired original SQL statement comprises the following steps:
step 1: analyzing the original SQL statement to obtain an input source base table, an output source base table and a target field list of index calculation; meanwhile, acquiring and perfecting the meta information structure and the data source service address information of the input source base table and the output source base table from the meta information data warehouse through the base table name of the input source base table and the base table name of the output source base table;
step 2: judging whether field information and a type list of an output source base table can be acquired from an output source; if yes, executing step 4; if not, executing the step 3;
and step 3: judging whether an aggregation calculation field exists in a target field list in index SQL of an input source; extracting the aggregation calculation field as field information, acquiring a type list from the service address information input into the source base table, and executing the step 4; if not, directly acquiring field information and a type list from the service address information input into the source base table, and executing the step 4;
and 4, step 4: constructing a meta-information structure and a storage form of a debugging output source base table according to the field information and the type list;
and 5: and replacing the output source base table of the original SQL statement with the debugging output source base table to complete transformation, thereby obtaining the debugging SQL.
Compared with the prior art, the invention has the following advantages and beneficial effects:
(1) the invention supports the index calculation of formal SQL sentences and debugging SQL simultaneously to obtain index data, and the data in the two modes are written into different ground libraries to achieve the aim that the index data do not interfere with each other, thereby ensuring the elegance of debugging, leading SQL developers to write more conveniently and quickly, having less cost investment, leading the SQL calculation index to be more easily and stably upgraded, avoiding the dirty data interference generated by a data user in the upgrading process, and effectively reducing the hardware cost.
(2) The invention provides a uniform background service scheme, develops SQL writing in which a user concentrates on index calculation, is suitable for any data production environment, supports SQL operation types of an SQL calculation engine and a stream batch integration under a big data cluster, and saves hardware cost, flow cost and operation and maintenance cost for building a set of environment.
(3) The invention provides a uniform background service scheme, development and use personnel do not need to use various language codes to research and develop file execution packages of various indexes and upload the file execution packages to a cluster system for operation, only need to pay attention to index SQL logic compiling processing, and the development cost of the development personnel is saved.
(4) The invention provides uniform SQL analysis processing, and does not need development and use personnel to consider how to debug SQL to arrange, pull and debug source data and consider the whole debugging scheme and the scheme of landing and warehousing of debugging result data. The scheme of the invention can completely avoid writing in the formal library, achieve effective isolation of the same SQL, ensure the correctness of the formal library and save the efficiency cost of developers and the data construction cost.
(5) The invention provides the SQL analysis processing, and development and use personnel can find the BUG in the SQL development by checking the debugging data result, correct the SQL error, facilitate the SQL development of the development personnel, realize one-stop service index calculation, improve the index development efficiency and effectively control the cost.
Drawings
FIG. 1 is a flowchart illustrating the overall steps of the present invention.
FIG. 2 is a flowchart of the steps of reforming the original SQL statement in the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples, but the embodiments of the present invention are not limited thereto.
Examples
The invention relates to a massive big data calculation and development debugging method based on SQL, which firstly explains the definitions of some key terms:
index data: the data is data of each statistical index calculated by using technical means in the big data calculation process in order to analyze the behaviors of users or events of things.
Inputting a source: and the storage system of the source data provides an accessible URL address or socket port service.
An output source: a storage system receiving the data may be used to provide an accessible URL address or socket port service.
SQL calculation: the method refers to distributed SQL engine calculation under a majority cluster environment, and is different from a synchronous calculation process in which single SQL calculation can obtain calculation results in time. And the method is an asynchronous index calculation process, and calculation result data are often stored in a database system to support users to inquire and read the data.
As shown in fig. 1 and 2, the massive big data calculation and development debugging method based on SQL of the present invention includes the following steps:
s1: acquiring original SQL sentences and debugging information of the calculation index data, namely SQL development and use personnel edit the original SQL sentences of the calculation index data through a front-end editor or a text tool and attach debugging information of whether to be debugged; meanwhile, the original SQL statement and the debugging information are submitted to a back-end service interface executor together. The debugging information comprises debugging requirement and debugging non-requirement. In actual development, SQL written by development and use personnel during data index development is debugged before the SQL is put into formal production calculation; in addition, in the calculation of the currently running index, if the version of the data and the field is newly added or deleted, the new version of the SQL index calculation generally needs to be debugged to ensure that the current SQL index calculation does not interrupt running.
S2: after receiving the SQL and the debugging identification, the back-end service interface executor judges whether the obtained original SQL statement needs to be debugged or not according to the debugging information; and if the debugging information is needed to be debugged, reconstructing the obtained original SQL statement to generate debugging SQL, and executing the step S3. If the debug information is that no debug is required, step S3 is performed directly.
Specifically, the step of reconstructing the obtained original SQL statement comprises the following steps:
step 1: and analyzing the original SQL statement by an SQL analyzer to obtain an input source base table and an output source base table of index calculation and a target field list determined in operation. And simultaneously, acquiring and perfecting the meta information structure and the data source service address information of the input source base table and the output source base table from the meta information data warehouse through the base table name of the input source base table and the base table name of the output source base table.
Step 2: judging whether field information and a type list of an output source base table can be acquired from an output source; if yes, executing step 4; otherwise, step 3 is executed.
And step 3: judging whether an aggregation calculation field exists in a target field list in index SQL of an input source; extracting the aggregation calculation field as field information, acquiring a type list from the service address information input into the source base table, and executing the step 4; and if not, directly acquiring field information and a type list from the service address information input into the source base table, and executing the step 4.
In some data source systems in the SQL computing activity, it is not allowed to create a base table directly in the index computing process, and it is necessary to create base table information by means of a data warehouse management system, for example, relational databases such as mysql, hive, impala, ckickhouse and the like are available, but data storage systems such as hbase, ES, mongodb, redis, kafka and the like allow structural data information to be constructed in the computing process, so there may be a case that specific data structure names and type information cannot be taken in the data warehouse of the output source. The type list information of the field information is supplemented from the data warehouse of the base table information of the input source in the present embodiment.
And 4, step 4: constructing a meta-information structure and a storage form of a debugging output source base table according to the field information and the type list; after the field information and the type list are obtained through the steps, the floor base table and the field structure information of the debugging output source are constructed.
Specifically, a database system (such as ES, HBase, mongoDb, redis, kafka and the like) with a cache is uniformly selected as a floor debugging base table according to the characteristics of index calculation to receive debugging index data; and setting an expiration mechanism for the debugging base table, calculating the data expiration time by adopting the arrival time or the upper limit of the number of pieces, and calculating and deleting the expired data. If the ES storage is used as the ground base table, an index (index) can be independently created for each debugging, the type data of the index contains the field type and the structure information, and the expiration time is set for the index. If HBase is selected, a debugging table is generated for each debugging, rowkey (a row key can be any character string and can mark a unique identifier of a row of records) is set as the table name of an output source plus time, and the format is as follows: test _ debug _ pay _ finish _20210420113332, wherein the field structure information of the data is the floor base table field structure information containing the json string of the data. And (4) injecting base table information of the data source of the debugging ground into the logic executed by the big data SQL execution engine, and applying for registering the connection information of the debugging output source in advance.
And 5: and replacing the output source base table of the original SQL statement with the debugging output source base table to complete transformation, thereby obtaining the debugging SQL.
In the embodiment, SQL is analyzed by using an SQL parser and is decomposed, service connection information of an input source and an output source in an original SQL statement is obtained, meta-information structure data of a source base table of the input source and table meta-information structure data generated by analyzing the output source are read to construct a base table structure of a debugging output source, if no designated field meta-information structure data exists in the current SQL statement, table structure information is obtained by inquiring the base table in the output source and is used for supplementing a field list of the debugging SQL for subsequent debugging output and outputting the debugging SQL statement, so that the difference between the debugging SQL and a formal SQL output source ground base is generated, the data generated by calculating two SQL indexes are not influenced with each other, and the common use of developers and non-developers is met. If the memory cache table exists, corresponding database table caching or memory cache table replacement is also performed for the data cache memory table in the SQL if writing requirements exist.
And (4) delivering the modified debugging SQL statement to a big data SQL computing engine for execution, generating debugging data, and verifying the correctness of the SQL statement developed by a developer. When the use scene of the developer is batch calculation, the input source data is directly read, and the calculation output is directly debugged to generate a data result. When the stream data is calculated, a development user can be enabled to specify the input data in advance, or specified data from a formal input source is written into an intermediate data table or a data storage component and is provided for SQL logic to calculate index data. The method comprises the following steps:
s3: submitting the original SQL statement submitted by the user or the debugging SQL generated by transformation in the step S2 to the SQL calculation engine of the cluster for SQL index calculation; the SQL statement is submitted to an SQL calculation engine of the big data cluster to be executed, and an index result is generated. The SQL calculation engine can be a SQL-removed calculation engine such as hive, impala, sparkSql, flinkSql, Es and the like. The SQL calculation index data is obtained after the original SQL statement is calculated, and the debugging SQL calculation index data is obtained after the debugging SQL is calculated.
S4: outputting a calculation result; the SQL calculation index data is written into a formal landing library, and the debugging SQL calculation index data is written into a debugging landing library.
In the whole calculation process, the output ground base table is changed by the debugging SQL, and the calculated index data of the debugging SQL cannot be written into the ground base table of the non-debugging SQL no matter whether the data is qualified or not, so that the correctness of the production environment data is ensured.
In the activity of debugging index calculation, the upper limit of the running time or the number of processed data pieces can be set for the debugging index calculation task as a debugging termination condition, and development and use personnel can be supported to manually terminate the debugging activity. For example, for the groupId consumption of kafka, in order to prevent kafka data in the index calculation of formal SQL from being consumed again because the kafka data is already consumed and cannot be consumed again because the same groupId consumption is used in the index calculation of the debug SQL, data loss or less calculation is caused. Therefore, in the debugging process, the added means is modified to be inconsistent with the groupId of each debugging; or generate a dedicated debug kafka to buffer the data of the input source for each debugging of the target data. For another example, in the intermediate table of the classified grouping calculation dimension write library, in order to prevent repeated writing, a cache intermediate table is generated to replace the dimension intermediate table in the debugging process, and whether writing has already been performed is checked during synchronous calculation to ensure data correctness.
The original SQL sentences respectively appoint output sources to receive the index data, unified output sources are designed for debugging SQL to store the index data, the output sources are subjected to data expiration calculation according to a time length caching mechanism or a number upper limit setting mechanism, and the expired data is deleted, so that the storage pressure of the unified debugging output sources is relieved, and meanwhile, the correctness of the index data in a formal production environment is guaranteed, and the influence of error data generated by calculation of the debugging SQL is avoided.
S5: and respectively providing the data in the official landing library and the data in the debugging landing library for different users. The development and use personnel analyze and debug the data in the database, and the data analysis personnel use the data in the formal landing library; and the data interface or service is provided for development and use personnel and data analysis personnel to read and view the index debugging data according to different purposes. The SQL development user can acquire debugging data by means of a debugging index data interface or service to show and analyze and verify the correctness of the SQL calculation.
The debugging method of the present embodiment is described below by way of example:
developer development uses the following SQL to debug execution time
Firstly, using SQL parser to parse the output source unidata _ stream _ data _ rd. ads _ mysql _ test _ order _ info table, the input source order _ finished. ads _ kafka _ test _ order _ finished table and the aggregation calculation function count, sum and group.
Secondly, acquiring the output source library table unidata _ stream _ data _ rd. ads _ mysql _ test _ order _ info from the data warehouse management system to obtain the meta information structure data: orderNum is bigint type, orderPrime, processionPrime, totAlPrime, and warTotalPrime is bigint type, and dt is a substring function operation and thus results in a string type.
The third step: selecting HBase to generate an ads _ mysql _ test _ order _ info _ debug ID table for a floor table scheme, wherein rowkey in data is as follows: ads _ mysql _ test _ order _ info + debug generation ID + time + index number.
The data content is the json cluster assembly mode of { orderNum: calculated value, proptionPrime: calculated value, totalprace: calculated value, wartTotalPrime: calculated value, dt: time calculated value }.
The fourth step: the service address of the HBase table and the table name of the ads _ mysql _ test _ order _ info _ debug ID are injected for the SQL execution engine, and the data is written to the HBase in the third step.
The fifth step: the final debug SQL is reformed into the following SQL statement:
insert _ femto _ default _ ads _ mysql _ test _ order _ info _ debug ID
And a sixth step: and submitting the debugging SQL to a big data SQL execution engine for index calculation, and writing the generated result data into a base _ default _ ads _ mysql _ test _ order _ info _ debugging ID, so that no influence is generated on a base table of the unidata _ stream _ data _ rd.
Finally, a debugging data interface or service is provided to let the developer pass
and acquiring the index calculation data of the currently debugged SQL by the ads _ mysql _ test _ order _ info _ debugging ID.
As described above, the present invention can be preferably realized.
Claims (2)
1. A massive big data calculation development debugging method based on SQL is characterized in that: the method comprises the following steps:
s1: acquiring original SQL sentences and debugging information of the calculation index data; the debugging information comprises debugging requirements and debugging requirements;
s2: judging whether the obtained original SQL statement needs to be debugged or not according to the debugging information; modifying the acquired original SQL statement to generate debugging SQL, and executing the step S3; otherwise, go to step S3;
s3: submitting the original SQL statement or the debugging SQL generated by the transformation in the step S2 to an SQL calculation engine of the cluster for SQL index calculation; obtaining SQL calculation index data after the original SQL statement is calculated, and obtaining debugging SQL calculation index data after the debugging SQL is calculated;
s4: outputting a calculation result; the SQL calculation index data is written into a formal landing library, and the debugging SQL calculation index data is written into a debugging landing library;
s5: and respectively providing the data in the official landing library and the data in the debugging landing library for different users.
2. The massive big data calculation and development debugging method based on SQL according to claim 1, characterized in that: the method for reconstructing the acquired original SQL statement comprises the following steps:
step 1: analyzing the original SQL statement to obtain an input source base table, an output source base table and a target field list of index calculation; meanwhile, acquiring and perfecting the meta information structure and the data source service address information of the input source base table and the output source base table from the meta information data warehouse through the base table name of the input source base table and the base table name of the output source base table;
step 2: judging whether field information and a type list of an output source base table can be acquired from an output source; if yes, executing step 4; if not, executing the step 3;
and step 3: judging whether an aggregation calculation field exists in a target field list in index SQL of an input source; extracting the aggregation calculation field as field information, acquiring a type list from the service address information input into the source base table, and executing the step 4; if not, directly acquiring field information and a type list from the service address information input into the source base table, and executing the step 4;
and 4, step 4: constructing a meta-information structure and a storage form of a debugging output source base table according to the field information and the type list;
and 5: and replacing the output source base table of the original SQL statement with the debugging output source base table to complete transformation, thereby obtaining the debugging SQL.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110750626.4A CN113641572B (en) | 2021-07-02 | 2021-07-02 | Debugging method for massive big data computing development based on SQL |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110750626.4A CN113641572B (en) | 2021-07-02 | 2021-07-02 | Debugging method for massive big data computing development based on SQL |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113641572A true CN113641572A (en) | 2021-11-12 |
| CN113641572B CN113641572B (en) | 2023-06-13 |
Family
ID=78416598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110750626.4A Active CN113641572B (en) | 2021-07-02 | 2021-07-02 | Debugging method for massive big data computing development based on SQL |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113641572B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5664173A (en) * | 1995-11-27 | 1997-09-02 | Microsoft Corporation | Method and apparatus for generating database queries from a meta-query pattern |
| CN102981946A (en) * | 2011-09-07 | 2013-03-20 | 阿里巴巴集团控股有限公司 | Extract Transform and Load (ETL) smoke test method |
| US20160062870A1 (en) * | 2014-08-28 | 2016-03-03 | Tamir Menahem | Structured query language debugger |
| US20160147637A1 (en) * | 2014-11-25 | 2016-05-26 | Raghuvira Bhagavan | Contextual debugging of sql queries in database-accessing applications |
| CN106202332A (en) * | 2016-07-01 | 2016-12-07 | 多点生活(中国)网络科技有限公司 | A kind of off-line picking task processing method based on SQLite data base |
| US20170270037A1 (en) * | 2016-03-18 | 2017-09-21 | Salesforce.Com, Inc. | Making production data available for testing in a non-production environment |
| US10162729B1 (en) * | 2016-02-01 | 2018-12-25 | State Farm Mutual Automobile Insurance Company | Automatic review of SQL statement complexity |
| CN109408378A (en) * | 2018-09-30 | 2019-03-01 | 福建星瑞格软件有限公司 | A kind of test method and system quickly positioning SQL parsing mistake under big data quantity |
| CN109508177A (en) * | 2018-08-21 | 2019-03-22 | 深圳乐信软件技术有限公司 | A kind of real-time computing technique, device, server and storage medium |
| CN110020834A (en) * | 2019-04-15 | 2019-07-16 | 浩鲸云计算科技股份有限公司 | A kind of on-line intelligence big data development approach of modeling and debugging fusion |
| WO2019183219A1 (en) * | 2018-03-22 | 2019-09-26 | Snowflake Computing Inc. | Incremental feature development and workload capture in database systems |
| CN110597722A (en) * | 2019-09-17 | 2019-12-20 | 深圳市及响科技有限公司 | Online debugging method for online development platform |
| CN111026779A (en) * | 2019-12-19 | 2020-04-17 | 厦门安胜网络科技有限公司 | Data processing method, device and storage medium based on Flink SQL |
| CN111581088A (en) * | 2020-04-29 | 2020-08-25 | 上海中通吉网络技术有限公司 | Spark-based SQL program debugging method, device, equipment and storage medium |
| CN112650676A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Software testing method, device, equipment and storage medium |
| CN112861495A (en) * | 2020-11-24 | 2021-05-28 | 辽宁振兴银行股份有限公司 | Method for generating impala SQL statement based on Excel template file |
-
2021
- 2021-07-02 CN CN202110750626.4A patent/CN113641572B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5664173A (en) * | 1995-11-27 | 1997-09-02 | Microsoft Corporation | Method and apparatus for generating database queries from a meta-query pattern |
| CN102981946A (en) * | 2011-09-07 | 2013-03-20 | 阿里巴巴集团控股有限公司 | Extract Transform and Load (ETL) smoke test method |
| US20160062870A1 (en) * | 2014-08-28 | 2016-03-03 | Tamir Menahem | Structured query language debugger |
| US20160147637A1 (en) * | 2014-11-25 | 2016-05-26 | Raghuvira Bhagavan | Contextual debugging of sql queries in database-accessing applications |
| US10162729B1 (en) * | 2016-02-01 | 2018-12-25 | State Farm Mutual Automobile Insurance Company | Automatic review of SQL statement complexity |
| US20170270037A1 (en) * | 2016-03-18 | 2017-09-21 | Salesforce.Com, Inc. | Making production data available for testing in a non-production environment |
| CN106202332A (en) * | 2016-07-01 | 2016-12-07 | 多点生活(中国)网络科技有限公司 | A kind of off-line picking task processing method based on SQLite data base |
| WO2019183219A1 (en) * | 2018-03-22 | 2019-09-26 | Snowflake Computing Inc. | Incremental feature development and workload capture in database systems |
| CN109508177A (en) * | 2018-08-21 | 2019-03-22 | 深圳乐信软件技术有限公司 | A kind of real-time computing technique, device, server and storage medium |
| CN109408378A (en) * | 2018-09-30 | 2019-03-01 | 福建星瑞格软件有限公司 | A kind of test method and system quickly positioning SQL parsing mistake under big data quantity |
| CN110020834A (en) * | 2019-04-15 | 2019-07-16 | 浩鲸云计算科技股份有限公司 | A kind of on-line intelligence big data development approach of modeling and debugging fusion |
| CN110597722A (en) * | 2019-09-17 | 2019-12-20 | 深圳市及响科技有限公司 | Online debugging method for online development platform |
| CN111026779A (en) * | 2019-12-19 | 2020-04-17 | 厦门安胜网络科技有限公司 | Data processing method, device and storage medium based on Flink SQL |
| CN111581088A (en) * | 2020-04-29 | 2020-08-25 | 上海中通吉网络技术有限公司 | Spark-based SQL program debugging method, device, equipment and storage medium |
| CN112861495A (en) * | 2020-11-24 | 2021-05-28 | 辽宁振兴银行股份有限公司 | Method for generating impala SQL statement based on Excel template file |
| CN112650676A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Software testing method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 肖顺: "基于Spark的用户行为分析系统研究及实现" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113641572B (en) | 2023-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020233330A1 (en) | Batch testing method, apparatus, and computer-readable storage medium | |
| US8019795B2 (en) | Data warehouse test automation framework | |
| CN110347598B (en) | Test script generation method and device, server and storage medium | |
| CN107844424A (en) | Model-based testing system and method | |
| US10990515B2 (en) | Automated unit testing in a mainframe environment | |
| CN112988782B (en) | Hive-supported interactive query method and device and storage medium | |
| CN111966738B (en) | System for automatically realizing back-end programming by selecting front-end configuration | |
| CN109614315A (en) | A kind of automatic generation method and system of data synchronism detection use-case | |
| CN111078675A (en) | Multidimensional comprehensive database SQL (structured query language) auditing and optimizing method and system | |
| CN114281342A (en) | Automatic code generation method | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| CN108829578A (en) | A kind of CDR association backfill accuracy automated testing method and system | |
| CN113641572A (en) | Massive big data calculation development debugging method based on SQL | |
| CN114969115A (en) | Data management method and system based on standardized metadata system | |
| Rugaber et al. | The transition of application programs from COBOL to a fourth generation language | |
| CN113886205A (en) | Database performance bottleneck positioning analysis method, device and system and storage medium | |
| CN111767222A (en) | Data model verification method and device, electronic equipment and storage medium | |
| CN113836164A (en) | Method, system, device and medium for unifying SQL | |
| Yu et al. | Design and implementation of fault injection based on abstract syntax tree of C Program | |
| CN111723104A (en) | Method, device and system for syntax analysis in data processing system | |
| CN117390055B (en) | JOOQ continuous list sentence generation method, device and medium | |
| CN116755684B (en) | OAS Schema generation method, device, equipment and medium | |
| Zhou | Research on the design of teacher workload management system | |
| KR100656559B1 (en) | Program Automatic Generating Tools | |
| Zhijuan et al. | A Data Warehouse Design Method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |