CN115168399B - Data processing method, device and equipment based on graphical interface and storage medium - Google Patents

Data processing method, device and equipment based on graphical interface and storage medium Download PDF

Info

Publication number
CN115168399B
CN115168399B CN202211085857.9A CN202211085857A CN115168399B CN 115168399 B CN115168399 B CN 115168399B CN 202211085857 A CN202211085857 A CN 202211085857A CN 115168399 B CN115168399 B CN 115168399B
Authority
CN
China
Prior art keywords
data
target
target data
generating
statement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211085857.9A
Other languages
Chinese (zh)
Other versions
CN115168399A (en
Inventor
马云
席小丁
张亚峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Yonghong Tech Co ltd
Original Assignee
Beijing Yonghong Tech Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Yonghong Tech Co ltd filed Critical Beijing Yonghong Tech Co ltd
Priority to CN202211085857.9A priority Critical patent/CN115168399B/en
Publication of CN115168399A publication Critical patent/CN115168399A/en
Application granted granted Critical
Publication of CN115168399B publication Critical patent/CN115168399B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2428—Query predicate definition using graphical user interfaces, including menus and forms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages
    • G06F16/2445—Data retrieval commands; View definitions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database

Abstract

The invention relates to the field of data processing, and discloses a data processing method, a data processing device, data processing equipment and a storage medium based on a graphical interface, which are used for improving the data processing efficiency of the graphical interface. The method comprises the following steps: receiving a data processing request sent by a terminal, and performing request analysis on the data processing request to obtain data source data corresponding to the data processing request; preprocessing data source data to obtain initial data, and setting a data format of the initial data to obtain target data; extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes; matching statement generation rules according to the target data attributes and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rules; and respectively generating associated data corresponding to each query statement according to the plurality of query statements, and generating a target data file according to the associated data corresponding to each query statement.

Description

Data processing method, device and equipment based on graphical interface and storage medium
Technical Field
The present invention relates to the field of data processing, and in particular, to a data processing method, apparatus, device, and storage medium based on a graphical interface.
Background
With the rapid development of computer technology, the data volume is increased rapidly, the traditional relational database is not suitable for processing big data, and for the processing work of the big data, the concept of the relational database is proposed and various big data storage tools are developed.
However, in the existing scheme, because the query sentences are different, in order to facilitate people to use the query sentences to perform data processing on the relational database, a part of storage tools provide a source query mode, but the existing scheme has poor data processing readability and is not beneficial to user writing, so that the data processing efficiency of the existing scheme is low.
Disclosure of Invention
The invention provides a data processing method, a data processing device, data processing equipment and a storage medium based on a graphical interface, which are used for improving the data processing efficiency of the graphical interface.
The invention provides a data processing method based on a graphical interface in a first aspect, which comprises the following steps: receiving a data processing request sent by a terminal, and performing request analysis on the data processing request to obtain data source data corresponding to the data processing request; preprocessing the data source data to obtain initial data, and setting a data format of the initial data to obtain target data; extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes; matching statement generating rules according to the target data attributes and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generating rules; and generating associated data corresponding to each query statement according to the plurality of query statements respectively, and generating a target data file according to the associated data corresponding to each query statement.
Optionally, in a first implementation manner of the first aspect of the present invention, the receiving a data processing request sent by a terminal, and performing request analysis on the data processing request to obtain data source data corresponding to the data processing request includes: receiving a data processing request sent by a terminal; performing request analysis on the data processing request to obtain a data acquisition address; inquiring a target data source according to the data acquisition address; and performing data extraction on the target data source according to a preset data structure type to obtain data source data corresponding to the data processing request.
Optionally, in a second implementation manner of the first aspect of the present invention, the preprocessing the data source data to obtain initial data, and performing data format setting on the initial data to obtain target data includes: carrying out data cleaning on the data source data to obtain data source data after the data cleaning; performing data conversion on the data source data subjected to data cleaning to obtain data source data subjected to data conversion; performing data integration on the data source data after the data conversion to obtain initial data; carrying out data category division on the initial data to obtain a plurality of data categories; generating a data format corresponding to each data category according to the data categories; and setting the data format of the initial data according to the data format corresponding to each data category to obtain target data.
Optionally, in a third implementation manner of the first aspect of the present invention, the extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes includes: acquiring standard attribute information and preset data attribute information from a preset database according to the target data; matching and analyzing a plurality of data in the target data through the standard attribute information to obtain target data attributes; and constructing a data model of the target data according to the target data attribute.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the matching a statement generation rule according to the target data attribute and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rule includes: determining statement generation parameters according to the target data attributes and the data model; inputting the statement generation parameters into a preset statement generation script for command conversion to generate a statement generation command; carrying out rule matching on the target data according to the statement generating command to obtain a statement generating rule; and generating a plurality of query sentences corresponding to the target data according to the sentence generation rule.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the generating associated data corresponding to each query statement according to the plurality of query statements, and generating a target data file according to the associated data corresponding to each query statement respectively includes: respectively reading the plurality of query sentences to obtain associated information corresponding to each query sentence; searching a relevant data item corresponding to the target data according to the relevant information corresponding to each query statement; generating associated data corresponding to each query statement according to the associated data items; and generating a target data file according to the associated data corresponding to each query statement.
Optionally, in a sixth implementation manner of the first aspect of the present invention, the data processing method based on a graphical interface further includes: and calling a preset data conversion model to perform data conversion on the plurality of associated data in the target data file to obtain a conversion target data file.
The second aspect of the present invention provides a data processing apparatus based on a graphical interface, including: the receiving module is used for receiving a data processing request sent by a terminal and analyzing the request of the data processing request to obtain data source data corresponding to the data processing request; the setting module is used for preprocessing the data source data to obtain initial data and setting a data format of the initial data to obtain target data; the extraction module is used for extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes; the processing module is used for matching a statement generation rule according to the target data attribute and the data model and generating a plurality of query statements corresponding to the target data according to the statement generation rule; and the generating module is used for generating associated data corresponding to each query statement according to the plurality of query statements respectively and generating a target data file according to the associated data corresponding to each query statement.
Optionally, in a first implementation manner of the second aspect of the present invention, the receiving module is specifically configured to: receiving a data processing request sent by a terminal; performing request analysis on the data processing request to obtain a data acquisition address; inquiring a target data source according to the data acquisition address; and performing data extraction on the target data source according to a preset data structure type to obtain data source data corresponding to the data processing request.
Optionally, in a second implementation manner of the second aspect of the present invention, the setting module is specifically configured to: carrying out data cleaning on the data source data to obtain data source data after the data cleaning; performing data conversion on the data source data subjected to data cleaning to obtain data source data subjected to data conversion; performing data integration on the data source data after the data conversion to obtain initial data; carrying out data category division on the initial data to obtain a plurality of data categories; generating a data format corresponding to each data category according to the data categories; and setting the data format of the initial data according to the data format corresponding to each data category to obtain target data.
Optionally, in a third implementation manner of the second aspect of the present invention, the extraction module is specifically configured to: acquiring standard attribute information and preset data attribute information from a preset database according to the target data; matching and analyzing a plurality of data in the target data through the standard attribute information to obtain target data attributes; and constructing a data model of the target data according to the target data attribute.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the processing module is specifically configured to: determining statement generation parameters according to the target data attributes and the data model; inputting the statement generation parameters into a preset statement generation script for command conversion to generate a statement generation command; carrying out rule matching on the target data according to the statement generating command to obtain a statement generating rule; and generating a plurality of query statements corresponding to the target data according to the statement generation rule.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the generating module is specifically configured to: respectively reading the plurality of query sentences to obtain associated information corresponding to each query sentence; searching a relevant data item corresponding to the target data according to the relevant information corresponding to each query statement; generating associated data corresponding to each query statement according to the associated data items; and generating a target data file according to the associated data corresponding to each query statement.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the data processing apparatus based on a graphical interface further includes: and the conversion module is used for calling a preset data conversion model to perform data conversion on the plurality of associated data in the target data file to obtain a conversion target data file.
The third aspect of the present invention provides a data processing apparatus based on a graphical interface, comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor calls the instructions in the memory to enable the graphical interface based data processing device to execute the graphical interface based data processing method.
A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, cause the computer to execute the above-mentioned graphical interface-based data processing method.
In the technical scheme provided by the invention, data source data are preprocessed to obtain initial data, data attribute extraction is carried out on target data to obtain target data attributes, a data model of the target data is constructed according to the target data attributes, and self-service data analysis is flexibly carried out; the data model can perform de-duplication, self-circulation listing, sorting, sampling, filtering, grouping, summarizing, association, row-column conversion and the like on target data; matching a statement generation rule according to the target data attribute and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rule, wherein the query statements have low use threshold and high calculation processing performance, and the data query process can generate and execute the query statements with high performance according to the database grammar specification of the data storage; the method has the advantages that the associated data corresponding to each query statement are generated according to the plurality of query statements, the target data file is generated according to the associated data corresponding to each query statement, the calculation compatibility is high, memory calculation operators are provided for data of different sources or calculations which are homologous but cannot be supported, extra operation or processing is not needed, and the data processing efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a graphical interface-based data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of another embodiment of a graphical interface-based data processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an embodiment of a graphical interface based data processing apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another embodiment of a graphical interface based data processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of a data processing device based on a graphical interface in the embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a data processing method, a data processing device, data processing equipment and a storage medium based on a graphical interface, which are used for improving the data processing efficiency of the graphical interface. The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of an embodiment of the present invention is described below, with reference to fig. 1, an embodiment of a data processing method based on a graphical interface in an embodiment of the present invention includes:
101. receiving a data processing request sent by a terminal, and performing request analysis on the data processing request to obtain data source data corresponding to the data processing request;
it is to be understood that the execution subject of the present invention may be a data processing apparatus based on a graphical interface, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
Specifically, a data processing request sent by a terminal is received, an interface path is determined according to the data processing request, wherein the data processing request carries a user account and a data processing identifier, an initial permission obtaining instruction is generated according to the data management request user account, the data processing identifier and the interface path, the initial permission obtaining instruction is sent to a cache module, the initial permission information is analyzed under the condition that the initial permission information fed back by a cache pool is received, target permission information is determined according to an analysis result, and data management processing information corresponding to the target permission information is obtained and fed back to the terminal. The consistency of the terminal data processing permission and the permissions corresponding to the user account, the interface path and the data processing identification is ensured, the data permission is accurately controlled, the data source data corresponding to the data processing request is finally obtained, and the permission control of the data of different levels is realized.
102. Preprocessing data source data to obtain initial data, and setting a data format of the initial data to obtain target data;
specifically, the server constructs a search engine library according to the data source data, and retrieves required initial data from the search engine library; the method comprises the steps of sequentially carrying out data preprocessing and data cleaning on initial data, carrying out feature screening on the initial data after the data cleaning, extracting feature values of the initial data to determine preset index variables, and carrying out format setting on the initial data based on the index variables to obtain target data.
103. Extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes;
specifically, a preset attribute data table is obtained, code table code value matching is conducted on each data column in target data, regular identification is conducted on each data column, the column type of each data column is determined, column feature vectors of each column are extracted, the column feature vectors comprise statistical features of the column data, description features of column names and/or column annotation information and column basic attribute features, the column feature vectors of each column are identified, column labels of each column are determined, matching is conducted on each column of data based on the labels, target data attributes are obtained, a data model of the target data is built according to the target data attributes, accuracy is guaranteed, and meanwhile calculated amount is effectively reduced.
104. Matching statement generation rules according to the target data attributes and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rules;
specifically, the server generates a hierarchical structure matched with the target data attribute and the data model; the method comprises the steps that query key characters determined based on a target grammar rule are arranged on each level except the lowest level in a level structure, the level of each query parameter in a query request in the level structure is determined, the query element corresponding to the query parameter in a target data table is determined by utilizing the binding relation between the query parameter and the target data table and the category of each query parameter, the query statement of each level and the query key characters of the last level are combined according to the target grammar rule from the lowest level to obtain the query statement of the last level until the query statement of the highest level is obtained, the query element corresponding to the query parameter is queried, the query statements of each highest level are combined according to the target grammar rule to obtain the query statement meeting the query request, namely a plurality of query statements corresponding to the target data.
105. And respectively generating associated data corresponding to each query statement according to the plurality of query statements, and generating a target data file according to the associated data corresponding to each query statement.
Specifically, the preset database is dynamically scanned, the server executes each query statement, at least part of the query results are returned to the user for the user to preview, and if the user is satisfied with the previewed query results, all the query results are returned to the user.
In the embodiment of the invention, data source data are preprocessed to obtain initial data, data attribute extraction is carried out on target data to obtain target data attribute, a data model of the target data is constructed according to the target data attribute, and self-service data analysis is flexibly carried out; the data model can perform deduplication, self-circulation listing, sorting, sampling, filtering, grouping, summarizing, associating, line-row conversion and the like on target data; matching a statement generation rule according to the target data attribute and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rule, wherein the query statements have low use threshold and high calculation processing performance, and the data query process can generate and execute the query statements with high performance according to the database grammar specification of the data storage; the method has the advantages that the associated data corresponding to each query statement are generated according to the query statements, the target data file is generated according to the associated data corresponding to each query statement, the calculation compatibility is high, memory calculation operators are provided for data of different sources or calculations which are homologous but cannot be supported, extra operation or processing is not needed, and the data processing efficiency is improved.
Referring to fig. 2, another embodiment of the data processing method based on the graphical interface according to the embodiment of the present invention includes:
201. receiving a data processing request sent by a terminal, and performing request analysis on the data processing request to obtain data source data corresponding to the data processing request;
specifically, a data processing request sent by a terminal is received; performing request analysis on the data processing request to obtain a data acquisition address; inquiring a target data source according to the data acquisition address; and performing data extraction on the target data source according to the preset data structure type to obtain data source data corresponding to the data processing request.
The data processing method includes the steps of receiving a data processing request sent by a terminal, carrying out request analysis on the data processing request to obtain data acquisition addresses, wherein the data acquisition addresses at least comprise two data acquisition addresses, generating a data acquisition address set by a server according to the at least two data acquisition addresses, sorting the data acquisition addresses according to occupied spaces of data pointed by the data acquisition addresses in the data acquisition address set, checking the data acquisition addresses in the data acquisition address set according to checking conditions, and adjusting sorting results of the data acquisition addresses in the data acquisition address set according to the checking results, wherein the checking conditions comprise format checking conditions and/or non-resource content checking conditions, the highest-ranked data acquisition address in the adjusted sorting results is selected as a target data acquisition address, and the server carries out data extraction on a target data source according to a preset data structure type to obtain data source data corresponding to the data processing request, so that success rate and efficiency of acquiring real data addresses are improved.
202. Preprocessing data source data to obtain initial data, and setting a data format of the initial data to obtain target data;
specifically, data cleaning is carried out on data source data to obtain the data source data after the data cleaning; performing data conversion on the data source data subjected to data cleaning to obtain data source data subjected to data conversion; performing data integration on the data source data after the data conversion to obtain initial data; performing data category division on the initial data to obtain a plurality of data categories; respectively generating a data format corresponding to each data category according to the plurality of data categories; and setting the data format of the initial data according to the data format corresponding to each data category to obtain target data.
The data cleaning comprises the steps of judging whether data source data belong to a preset type of cleaning based on a confirmation result, calling the confirmation result corresponding to the data source data if the data source data belong to a preset object of cleaning based on the confirmation result, taking the confirmation result as the data after the data source data are cleaned, cleaning the data source data in sequence according to a plurality of preset data cleaning rules if the data source data do not belong to the preset type of cleaning based on the confirmation result to obtain the data after the data source data are cleaned, then carrying out data conversion on the data source data after the data are cleaned to obtain the data source data after the data conversion, carrying out data integration on the data source data after the data conversion to obtain initial data, and carrying out data category division on the initial data to obtain a plurality of data categories; and respectively generating a data format corresponding to each data category according to the multiple data categories, and setting the data format of the initial data according to the data format corresponding to each data category to obtain target data.
203. Acquiring standard attribute information and preset data attribute information from a preset database according to target data;
204. matching and analyzing a plurality of data in the target data through the standard attribute information to obtain target data attributes;
205. constructing a data model of the target data according to the target data attribute;
specifically, the server acquires target data, automatically corrects abnormal target data, extracts standard attribute data in a standard resource database, establishes an association relationship, accurately matches the target data with a standard resource data target database on the basis of the target data, divides the numbered character strings T into N small blocks, matches the standard attributes in the source database with the data of each small block, finally acquires standard attribute information and preset data attribute information, performs matching analysis on a plurality of data in the target data through a preset similarity calculation method, acquires target data attributes, and constructs a corresponding data model.
206. Matching statement generation rules according to the target data attributes and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rules;
specifically, determining statement generation parameters according to target data attributes and a data model; inputting the statement generation parameters into a preset statement generation script for command conversion to generate a statement generation command; carrying out rule matching on the target data according to the statement generation command to obtain a statement generation rule; and generating a plurality of query sentences corresponding to the target data according to the sentence generation rule.
Determining statement generation parameters according to the target data attributes and the data model; the sentence generation parameters are input into a preset sentence generation script to perform command conversion, a sentence generation command is generated, each command classification rule is obtained, each command classification rule is analyzed and processed according to a preset classification rule base to obtain the feature type of each command classification rule, a target grammar matching rule corresponding to each command classification rule is obtained according to each feature type, a sentence generation rule is obtained based on each command classification rule and the target grammar matching rule corresponding to each command classification rule, a plurality of query sentences corresponding to target data are generated according to the sentence generation rule, and the generation efficiency of the query sentences is improved.
207. And respectively generating associated data corresponding to each query statement according to the plurality of query statements, and generating a target data file according to the associated data corresponding to each query statement.
Specifically, a plurality of query sentences are read respectively to obtain associated information corresponding to each query sentence; searching a relevant data item corresponding to the target data according to the relevant information corresponding to each query statement; generating associated data corresponding to each query statement according to the associated data items; and generating a target data file according to the associated data corresponding to each query statement.
Acquiring a query statement, wherein the query statement comprises a select clause and a from clause; detecting whether the select clause and the from clause accord with a preset arrangement sequence or not, if not, adjusting the select clause and the from clause according to the preset arrangement sequence, sending the adjusted query clauses to a target data storage space for execution, generating associated data corresponding to each query clause according to associated data items, and generating a target data file according to the associated data corresponding to each query clause.
Optionally, a preset data conversion model is called to perform data conversion on the multiple associated data in the target data file, so as to obtain a conversion target data file.
The method comprises the steps of obtaining a plurality of associated data in a target data file according to a source path, determining a source format of the source data, calling a format conversion program matched with the source format and the target format from a preset format converter to convert the source data from the source format to the target format, obtaining the target data, storing a plurality of format conversion programs in the format converter, and storing the target data according to the target path to obtain a conversion target data file, so that the conversion efficiency is improved, and the method also has good universality.
In the embodiment of the invention, data source data are preprocessed to obtain initial data, data attribute extraction is carried out on target data to obtain target data attribute, a data model of the target data is constructed according to the target data attribute, and self-service data analysis is flexibly carried out; the data model can perform deduplication, self-circulation listing, sorting, sampling, filtering, grouping, summarizing, associating, line-row conversion and the like on target data; matching a statement generation rule according to the target data attribute and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rule, wherein the query statements have low use threshold and high calculation processing performance, and the data query process can generate and execute the query statements with high performance according to the database grammar specification of the data storage; the method has the advantages that the associated data corresponding to each query statement are generated according to the plurality of query statements, the target data file is generated according to the associated data corresponding to each query statement, the calculation compatibility is high, memory calculation operators are provided for data of different sources or calculations which are homologous but cannot be supported, extra operation or processing is not needed, and the data processing efficiency is improved.
With reference to fig. 3, the data processing method based on the graphical interface in the embodiment of the present invention is described above, and a data processing apparatus based on the graphical interface in the embodiment of the present invention is described below, where an embodiment of the data processing apparatus based on the graphical interface in the embodiment of the present invention includes:
a receiving module 301, configured to receive a data processing request sent by a terminal, and perform request analysis on the data processing request to obtain data source data corresponding to the data processing request;
a setting module 302, configured to preprocess the data source data to obtain initial data, and set a data format of the initial data to obtain target data;
the extraction module 303 is configured to perform data attribute extraction on the target data to obtain target data attributes, and construct a data model of the target data according to the target data attributes;
a processing module 304, configured to match a statement generation rule according to the target data attribute and the data model, and generate a plurality of query statements corresponding to the target data according to the statement generation rule;
a generating module 305, configured to generate associated data corresponding to each query statement according to the multiple query statements, and generate a target data file according to the associated data corresponding to each query statement.
In the embodiment of the invention, data source data are preprocessed to obtain initial data, data attribute extraction is carried out on target data to obtain target data attribute, a data model of the target data is constructed according to the target data attribute, and self-service data analysis is flexibly carried out; the data model can perform deduplication, self-circulation listing, sorting, sampling, filtering, grouping, summarizing, associating, line-row conversion and the like on target data; matching statement generating rules according to target data attributes and data models, and generating a plurality of query statements corresponding to target data according to the statement generating rules, wherein the query statements have low use threshold and high calculation processing performance, and the data query process can generate and execute the query statements with high performance according to the database grammar specification of data storage; the method has the advantages that the associated data corresponding to each query statement are generated according to the plurality of query statements, the target data file is generated according to the associated data corresponding to each query statement, the calculation compatibility is high, memory calculation operators are provided for data of different sources or calculations which are homologous but cannot be supported, extra operation or processing is not needed, and the data processing efficiency is improved.
Referring to fig. 4, another embodiment of the data processing apparatus based on graphical interface according to the embodiment of the present invention includes:
a receiving module 301, configured to receive a data processing request sent by a terminal, and perform request analysis on the data processing request to obtain data source data corresponding to the data processing request;
a setting module 302, configured to pre-process the data source data to obtain initial data, and perform data format setting on the initial data to obtain target data;
an extraction module 303, configured to perform data attribute extraction on the target data to obtain target data attributes, and construct a data model of the target data according to the target data attributes;
a processing module 304, configured to match a statement generation rule according to the target data attribute and the data model, and generate a plurality of query statements corresponding to the target data according to the statement generation rule;
a generating module 305, configured to generate associated data corresponding to each query statement according to the plurality of query statements, and generate a target data file according to the associated data corresponding to each query statement.
Optionally, the receiving module 301 is specifically configured to: receiving a data processing request sent by a terminal; performing request analysis on the data processing request to obtain a data acquisition address; inquiring a target data source according to the data acquisition address; and performing data extraction on the target data source according to a preset data structure type to obtain data source data corresponding to the data processing request.
Optionally, the setting module 302 is specifically configured to: performing data cleaning on the data source data to obtain data source data after the data cleaning; performing data conversion on the data source data subjected to data cleaning to obtain data source data subjected to data conversion; performing data integration on the data source data after the data conversion to obtain initial data; carrying out data category division on the initial data to obtain a plurality of data categories; generating a data format corresponding to each data category according to the data categories; and setting the data format of the initial data according to the data format corresponding to each data category to obtain target data.
Optionally, the extracting module 303 is specifically configured to: acquiring standard attribute information and preset data attribute information from a preset database according to the target data; matching and analyzing a plurality of data in the target data through the standard attribute information to obtain target data attributes; and constructing a data model of the target data according to the target data attribute.
Optionally, the processing module 304 is specifically configured to: determining statement generation parameters according to the target data attributes and the data model; inputting the statement generation parameters into a preset statement generation script for command conversion to generate a statement generation command; carrying out rule matching on the target data according to the statement generating command to obtain a statement generating rule; and generating a plurality of query sentences corresponding to the target data according to the sentence generation rule.
Optionally, the generating module 305 is specifically configured to: respectively reading the plurality of query sentences to obtain associated information corresponding to each query sentence; searching a relevant data item corresponding to the target data according to the relevant information corresponding to each query statement; generating associated data corresponding to each query statement according to the associated data items; and generating a target data file according to the associated data corresponding to each query statement.
Optionally, the data processing apparatus based on a graphical interface further includes:
the conversion module 306 is configured to invoke a preset data conversion model to perform data conversion on multiple associated data in the target data file, so as to obtain a conversion target data file.
In the embodiment of the invention, data source data are preprocessed to obtain initial data, data attribute extraction is carried out on target data to obtain target data attribute, a data model of the target data is constructed according to the target data attribute, and self-service data analysis is flexibly carried out; the data model can perform deduplication, self-circulation listing, sorting, sampling, filtering, grouping, summarizing, associating, line-row conversion and the like on target data; matching statement generating rules according to target data attributes and data models, and generating a plurality of query statements corresponding to target data according to the statement generating rules, wherein the query statements have low use threshold and high calculation processing performance, and the data query process can generate and execute the query statements with high performance according to the database grammar specification of data storage; the method has the advantages that the associated data corresponding to each query statement are generated according to the plurality of query statements, the target data file is generated according to the associated data corresponding to each query statement, the calculation compatibility is high, memory calculation operators are provided for data of different sources or calculations which are homologous but cannot be supported, extra operation or processing is not needed, and the data processing efficiency is improved.
Fig. 3 and fig. 4 above describe the data processing apparatus based on the graphical interface in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the data processing apparatus based on the graphical interface in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of a data processing apparatus based on a graphical interface according to an embodiment of the present invention, where the data processing apparatus 500 based on a graphical interface may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a sequence of instruction operations for the graphical interface based data processing apparatus 500. Still further, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the graphical interface based data processing device 500.
The graphical interface based data processing apparatus 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows Server, mac OS X, unix, linux, freeBSD, and the like. Those skilled in the art will appreciate that the graphical interface based data processing device architecture shown in fig. 5 does not constitute a limitation of the graphical interface based data processing device and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
The invention also provides a data processing device based on the graphical interface, which comprises a memory and a processor, wherein the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to execute the steps of the data processing method based on the graphical interface in the above embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and may also be a volatile computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the instructions cause the computer to execute the steps of the graphical interface-based data processing method.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (6)

1. A data processing method based on a graphical interface is characterized by comprising the following steps:
receiving a data processing request sent by a terminal, and performing request analysis on the data processing request to obtain data source data corresponding to the data processing request;
preprocessing the data source data to obtain initial data, and setting a data format of the initial data to obtain target data; the preprocessing the data source data to obtain initial data, and performing data format setting on the initial data to obtain target data includes: carrying out data cleaning on the data source data to obtain data source data after the data cleaning; performing data conversion on the data source data subjected to data cleaning to obtain data source data subjected to data conversion; performing data integration on the data source data after the data conversion to obtain initial data; carrying out data category division on the initial data to obtain a plurality of data categories; generating a data format corresponding to each data category according to the multiple data categories respectively; setting a data format of the initial data according to the data format corresponding to each data category to obtain target data;
extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes; the data attribute extraction of the target data to obtain target data attributes, and the construction of the data model of the target data according to the target data attributes comprise: acquiring standard attribute information and preset data attribute information from a preset database according to the target data; matching and analyzing a plurality of data in the target data through the standard attribute information to obtain target data attributes; constructing a data model of the target data according to the target data attribute; specifically, a preset attribute data table is obtained, code table code value matching is carried out on each data column in target data, regular identification is carried out on each data column, the column type of each data column is determined, column characteristic vectors of each column are extracted, the column characteristic vectors comprise statistical characteristics of the column data, description characteristics of column names and/or column annotation information and column basic attribute characteristics, the column characteristic vectors of each column are identified, column labels of each column are determined, matching is carried out on each column of data based on the labels, target data attributes are obtained, and a data model of the target data is constructed according to the target data attributes;
matching statement generating rules according to the target data attributes and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generating rules; matching a statement generation rule according to the target data attribute and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rule, wherein the generating includes: determining statement generation parameters according to the target data attributes and the data model; inputting the statement generation parameters into a preset statement generation script for command conversion to generate a statement generation command; carrying out rule matching on the target data according to the statement generating command to obtain a statement generating rule; generating a plurality of query sentences corresponding to the target data according to the sentence generation rule; determining statement generation parameters according to the target data attributes and the data model; inputting statement generation parameters into a preset statement generation script for command conversion, generating a statement generation command, acquiring each command classification rule, analyzing each command classification rule according to a preset classification rule base to acquire the feature type of each command classification rule, acquiring a target grammar matching rule corresponding to each command classification rule according to each feature type, acquiring a statement generation rule based on each command classification rule and the target grammar matching rule corresponding to each command classification rule, and generating a plurality of query statements corresponding to target data according to the statement generation rule;
generating associated data corresponding to each query statement according to the plurality of query statements respectively, and generating a target data file according to the associated data corresponding to each query statement; wherein, the generating the associated data corresponding to each query statement according to the plurality of query statements respectively, and generating the target data file according to the associated data corresponding to each query statement comprises: respectively reading the plurality of query sentences to obtain associated information corresponding to each query sentence; searching a relevant data item corresponding to the target data according to the relevant information corresponding to each query statement; generating associated data corresponding to each query statement according to the associated data items; and generating a target data file according to the associated data corresponding to each query statement.
2. The graphical interface-based data processing method according to claim 1, wherein the receiving of the data processing request sent by the terminal and the request parsing of the data processing request to obtain data source data corresponding to the data processing request comprises:
receiving a data processing request sent by a terminal;
performing request analysis on the data processing request to obtain a data acquisition address;
inquiring a target data source according to the data acquisition address;
and performing data extraction on the target data source according to a preset data structure type to obtain data source data corresponding to the data processing request.
3. The graphical interface-based data processing method according to claim 1 or 2, wherein the graphical interface-based data processing method further comprises:
and calling a preset data conversion model to perform data conversion on the plurality of associated data in the target data file to obtain a conversion target data file.
4. A data processing device based on a graphical interface is characterized by comprising:
the receiving module is used for receiving a data processing request sent by a terminal and analyzing the request of the data processing request to obtain data source data corresponding to the data processing request;
the setting module is used for preprocessing the data source data to obtain initial data and setting the data format of the initial data to obtain target data; the preprocessing the data source data to obtain initial data, and performing data format setting on the initial data to obtain target data includes: performing data cleaning on the data source data to obtain data source data after the data cleaning; performing data conversion on the data source data subjected to data cleaning to obtain data source data subjected to data conversion; performing data integration on the data source data after the data conversion to obtain initial data; carrying out data category division on the initial data to obtain a plurality of data categories; generating a data format corresponding to each data category according to the multiple data categories respectively; setting a data format of the initial data according to the data format corresponding to each data category to obtain target data;
the extraction module is used for extracting data attributes of the target data to obtain target data attributes, and constructing a data model of the target data according to the target data attributes; the data attribute extraction of the target data to obtain target data attributes, and the construction of the data model of the target data according to the target data attributes includes: acquiring standard attribute information and preset data attribute information from a preset database according to the target data; matching and analyzing a plurality of data in the target data through the standard attribute information to obtain target data attributes; constructing a data model of the target data according to the target data attribute; specifically, a preset attribute data table is obtained, code table code value matching is carried out on each data column in target data, regular identification is carried out on each data column, the column type of each data column is determined, column characteristic vectors of each column are extracted, wherein each column characteristic vector comprises the statistical characteristics of column data, the description characteristics of column names and/or column annotation information and column basic attribute characteristics, the column characteristic vectors of each column are identified, column labels of each column are determined, each column data is matched based on the labels, target data attributes are obtained, and a data model of the target data is constructed according to the target data attributes;
the processing module is used for matching a statement generation rule according to the target data attribute and the data model and generating a plurality of query statements corresponding to the target data according to the statement generation rule; matching a statement generation rule according to the target data attribute and the data model, and generating a plurality of query statements corresponding to the target data according to the statement generation rule, wherein the generating includes: determining statement generation parameters according to the target data attributes and the data model; inputting the statement generation parameters into a preset statement generation script for command conversion to generate a statement generation command; performing rule matching on the target data according to the statement generation command to obtain a statement generation rule; generating a plurality of query sentences corresponding to the target data according to the sentence generation rule; determining statement generation parameters according to the target data attributes and the data model; inputting statement generation parameters into a preset statement generation script for command conversion, generating a statement generation command, acquiring each command classification rule, analyzing each command classification rule according to a preset classification rule base to acquire the feature type of each command classification rule, acquiring a target grammar matching rule corresponding to each command classification rule according to each feature type, acquiring a statement generation rule based on each command classification rule and the target grammar matching rule corresponding to each command classification rule, and generating a plurality of query statements corresponding to target data according to the statement generation rule;
the generating module is used for generating associated data corresponding to each query statement according to the plurality of query statements respectively and generating a target data file according to the associated data corresponding to each query statement; wherein, the generating the associated data corresponding to each query statement according to the plurality of query statements respectively, and generating the target data file according to the associated data corresponding to each query statement comprises: respectively reading the plurality of query sentences to obtain associated information corresponding to each query sentence; searching a relevant data item corresponding to the target data according to the relevant information corresponding to each query statement; generating associated data corresponding to each query statement according to the associated data items; and generating a target data file according to the associated data corresponding to each query statement.
5. A graphical interface based data processing device, comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the graphical interface based data processing apparatus to perform the graphical interface based data processing method of any of claims 1-3.
6. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the graphical interface based data processing method according to any one of claims 1-3.
CN202211085857.9A 2022-09-06 2022-09-06 Data processing method, device and equipment based on graphical interface and storage medium Active CN115168399B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211085857.9A CN115168399B (en) 2022-09-06 2022-09-06 Data processing method, device and equipment based on graphical interface and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211085857.9A CN115168399B (en) 2022-09-06 2022-09-06 Data processing method, device and equipment based on graphical interface and storage medium

Publications (2)

Publication Number Publication Date
CN115168399A CN115168399A (en) 2022-10-11
CN115168399B true CN115168399B (en) 2022-12-06

Family

ID=83480877

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211085857.9A Active CN115168399B (en) 2022-09-06 2022-09-06 Data processing method, device and equipment based on graphical interface and storage medium

Country Status (1)

Country Link
CN (1) CN115168399B (en)

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9135583B2 (en) * 2008-07-16 2015-09-15 Business Objects S.A. Systems and methods to create continuous queries associated with push-type and pull-type data
CN106897060A (en) * 2017-02-15 2017-06-27 中国保险信息技术管理有限责任公司 Based on patterned data processing method and device
CN113420044A (en) * 2021-06-30 2021-09-21 平安国际智慧城市科技股份有限公司 Data query method, device, equipment and storage medium
CN113672781A (en) * 2021-08-20 2021-11-19 平安国际智慧城市科技股份有限公司 Data query method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN115168399A (en) 2022-10-11

Similar Documents

Publication Publication Date Title
US6665661B1 (en) System and method for use in text analysis of documents and records
US8832133B2 (en) Answering web queries using structured data sources
CN104199965B (en) Semantic information retrieval method
CN112035599A (en) Query method and device based on vertical search, computer equipment and storage medium
CN110532347A (en) A kind of daily record data processing method, device, equipment and storage medium
CN111562920A (en) Method and device for determining similarity of small program codes, server and storage medium
CN107341152B (en) Parameter input method and device
JP5780036B2 (en) Extraction program, extraction method and extraction apparatus
CN111984673B (en) Fuzzy retrieval method and device for tree structure of power grid electric energy metering system
CN113297251A (en) Multi-source data retrieval method, device, equipment and storage medium
CN111104422B (en) Training method, device, equipment and storage medium of data recommendation model
CN112579604A (en) Test system number making method, device, equipment and storage medium
CN115147020B (en) Decoration data processing method, device, equipment and storage medium
CN115168399B (en) Data processing method, device and equipment based on graphical interface and storage medium
CN111651514A (en) Data import method and device
CN111460114A (en) Retrieval method, device, equipment and computer readable storage medium
CN116469500A (en) Data quality control method and system based on post-structuring of medical document
CN111831286A (en) User complaint processing method and device
JP2020071678A (en) Information processing device, control method, and program
CN115408419A (en) Data extraction method and related device
JP2019200582A (en) Search device, search method, and search program
CN115858742A (en) Question text expansion method, device, equipment and storage medium
CN113901839A (en) User video information auditing method, device, equipment and storage medium
CN113344023A (en) Code recommendation method, device and system
CN115203057B (en) Low code test automation method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
PE01 Entry into force of the registration of the contract for pledge of patent right

Denomination of invention: Data processing methods, devices, devices, and storage media based on graphical interfaces

Granted publication date: 20221206

Pledgee: Beijing first financing Company limited by guarantee

Pledgor: BEIJING YONGHONG TECH CO.,LTD.

Registration number: Y2024980011596