CN112883049A - Real-time data calculation method and device and storage medium - Google Patents
Real-time data calculation method and device and storage medium Download PDFInfo
- Publication number
- CN112883049A CN112883049A CN201911210038.0A CN201911210038A CN112883049A CN 112883049 A CN112883049 A CN 112883049A CN 201911210038 A CN201911210038 A CN 201911210038A CN 112883049 A CN112883049 A CN 112883049A
- Authority
- CN
- China
- Prior art keywords
- data
- distributed
- structured query
- computation
- query language
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 37
- 230000000007 visual effect Effects 0.000 claims description 21
- 238000005111 flow chemistry technique Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 7
- 230000009471 action Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005034 decoration Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Stored Programmes (AREA)
Abstract
The application discloses a method, a device and a storage medium for real-time calculation of data, wherein the method comprises the following steps: the method comprises the steps of receiving a structured query language instruction which is input by a user and used for processing data, generating a plurality of data computing units which can be processed by a distributed computing engine according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data, and performing distributed computing on the plurality of data computing units by the distributed computing engine. By the embodiment, the calculation efficiency of data real-time calculation can be improved.
Description
Technical Field
The present application relates to the field of big data, and in particular, to a method, an apparatus, and a storage medium for real-time computation of data.
Background
With the rapid development of internet technology, the concept of real-time computation has been created based on a large amount of data generated in real time, and solutions to real-time computation of many data have been developed at the same time. Most of the existing data real-time calculation methods are based on basic technical frameworks such as chord, spark streaming and the like, and the real-time calculation methods based on the frameworks have high requirements on code writers and low data operation efficiency, and cannot effectively meet high-requirement data calculation.
The embodiment of the disclosure provides a data real-time computing method, a data real-time computing device and a storage medium, so as to improve the computing efficiency of data real-time computing.
Disclosure of Invention
The embodiment of the disclosure provides a data real-time computing method, a data real-time computing device and a storage medium, so as to improve the computing efficiency of data real-time computing.
To solve the above technical problem, the embodiment of the present invention is implemented as follows:
in a first aspect, an embodiment of the present disclosure provides a method for computing data in real time, including:
receiving a structured query language instruction input by a user and used for processing data;
generating a plurality of data computation units capable of being processed by a distributed computation engine in accordance with the structured query language instructions, wherein the distributed computation engine supports real-time computation of data; and
performing distributed computation on the plurality of data computation units using the distributed computation engine.
In a second aspect, the disclosed embodiments also provide a storage medium, which includes a stored program, wherein the data real-time computing method according to the first aspect is executed by a processor when the program runs.
In a third aspect, there is also provided a real-time data computing apparatus according to an embodiment of the present disclosure, including:
the language instruction receiving module is used for receiving a structured query language instruction which is input by a user and used for processing data;
a data unit generation module for generating a plurality of data computation units capable of being processed by a distributed computation engine according to the structured query language instruction, wherein the distributed computation engine supports real-time computation of data; and
and the data unit calculation module is used for performing distributed calculation on the plurality of data calculation units by utilizing the distributed calculation engine.
In a fourth aspect, an embodiment of the present disclosure further provides a real-time data computing apparatus, including:
a processor; and
a memory coupled to the processor for providing instructions to the processor for processing the following processing steps:
receiving a structured query language instruction input by a user and used for processing data;
generating a plurality of data computation units capable of being processed by a distributed computation engine in accordance with the structured query language instructions, wherein the distributed computation engine supports real-time computation of data; and
performing distributed computation on the plurality of data computation units using the distributed computation engine.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine through the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by using the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the disclosure and together with the description serve to explain the disclosure and not to limit the disclosure. In the drawings:
fig. 1 is a block diagram of a hardware structure of a computing device for implementing a real-time data computing method according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of a real-time data calculation method according to an embodiment of the disclosure;
FIG. 3 is a schematic flow chart of a real-time data calculation method according to another embodiment of the present disclosure
FIG. 4 is a schematic diagram of a data real-time computing device according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of a data real-time computing device according to another embodiment of the disclosure.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. It is to be understood that the described embodiments are merely exemplary of some, and not all, of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
According to the present embodiment, there is also provided an embodiment of a method for real-time computation of data, it is noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order different than here.
The method embodiments provided by the present embodiment may be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Fig. 1 is a hardware block diagram of a computing device for implementing a real-time data computing method according to an embodiment of the present disclosure. As shown in fig. 1, the computing device may include one or more processors (which may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the electronic device. For example, the computing device may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
It should be noted that the one or more processors and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuitry may be a single, stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computing device. As referred to in the disclosed embodiments, the data processing circuit acts as a processor control (e.g., selection of a variable resistance termination path connected to the interface).
The memory may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the data real-time computing method in the embodiments of the present disclosure, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the data real-time computing method of the application program. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, which may be connected to the computing device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device is used for receiving or transmitting data via a network. Specific examples of such networks may include wireless networks provided by communication providers of the computing devices. In one example, the transmission device includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computing device.
It should be noted here that in some alternative embodiments, the computing device shown in fig. 1 described above may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that FIG. 1 is only one example of a particular specific example and is intended to illustrate the types of components that may be present in a computing device as described above.
In the above operating environment, the present embodiment provides a real-time data calculation method. Fig. 2 is a schematic flow chart of a real-time data computing method according to an embodiment of the present disclosure, and referring to fig. 2, the method includes:
s202: receiving a structured query language instruction input by a user and used for processing data;
s204: generating a plurality of data computing units capable of being processed by a distributed computing engine according to the structured query language instructions, wherein the distributed computing engine supports real-time computing of data;
s206: and performing distributed computation on the plurality of data computing units by using the distributed computing engine.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine by the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
In the above action S202, a structured query language instruction for processing data, which is input by a user, is received, that is, a structured query language instruction written by the user according to the data on a provided visual interface is received, where the structured query language instruction is SQL (structured query language).
In the above action S204, a plurality of data computing units capable of being processed by the distributed computing engine are generated according to the structured query language instruction, where the distributed computing engine supports real-time computing of data, and first receives a submission operation of the structured query language instruction operated by a user on a visual interface, and the computing engine parses the submitted structured query language instruction and calculates a corresponding instruction execution task, and generates a plurality of data computing units capable of being processed by the distributed computing engine according to the instruction execution task.
In the above-described operation S206, distributed computation is performed on the plurality of data computing units by the distributed computation engine. The distributed engine performs distributed computation on the plurality of data computing units, so that the computing efficiency of the data can be greatly improved.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine by the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
Further, the data is a data table created by a user at a preset visual operation interface, and before receiving a structured query language instruction input by the user for processing the data, the method further comprises the following steps:
(a1) acquiring a continuous data stream generated by a user terminal;
(a2) and receiving a data table created by a user on a preset visual operation interface according to the data stream.
In the above action (a1), a continuous data stream generated by the user terminal is obtained, where the data stream includes data in the format of kafka, hbase, mysql, elastic search, or the like, and is not limited herein.
In the action (a2), a data table created by a user on a preset visual operation interface according to the data stream is received. The user creates a data table corresponding to the data stream type in the visual operation interface, and the data table may be one or both of a streaming data table and a dimension data table, which is not particularly limited herein. For example, a streaming data table is constructed according to kafka data, and a dimension data table is constructed according to mysql data.
Further, generating a plurality of data computing units capable of being processed by a distributed computing engine from the structured query language instructions, comprising:
(b1) debugging the structured query language instruction by using a preset data logic grammar;
(b2) a plurality of data computing units capable of being processed by the distributed computing engine are generated from the debugged structured query language instructions.
In the above actions (b1) and (b2), the structured query language instruction is debugged by using the preset data logic syntax, and a plurality of data computing units capable of being processed by the distributed computing engine are generated according to the debugged structured query language instruction. The method comprises the steps of carrying out grammar and logic debugging on a structured query language instruction according to a preset data logic grammar, feeding back a debugging result, generating a plurality of data computing units which can be processed by a distributed computing engine from the structured query language instruction which passes the debugging, and feeding back the result of the structured query language instruction which does not pass the debugging to a visual interface which can be received by a user.
Further, the distributed computing engine comprises a plurality of servers, and performs distributed computing on a plurality of data computing units by using the distributed computing engine, and comprises:
(c1) distributing the data calculation units in a plurality of servers of the distributed calculation engine, and respectively carrying out logic operation on the data calculation units distributed in each server;
(c2) and performing logic operation on the plurality of servers to obtain a data calculation result corresponding to the structured query language instruction.
In the above-described operation (c1), the data calculation units are distributed among the plurality of servers of the distributed calculation engine, the data calculation units distributed among the servers are each logically operated, when continuous real-time data is received, the plurality of data calculation units are generated from the received continuous real-time data, the plurality of data calculation units are distributed among the plurality of servers of the distributed calculation engine, and the data calculation units distributed among the servers collectively perform distributed calculation, thereby improving the calculation efficiency of performing real-time calculation on continuously generated data.
In the above-described operation (c2), the plurality of servers perform logical operations to obtain data calculation results corresponding to the structured query language instruction. And performing logic operation on the structured query language instruction input according to the continuously generated data through a plurality of servers to obtain a calculation result of the corresponding data.
Further, the embodiment of the present invention further includes: the method is applied in an open source flow processing framework. The embodiment of the invention discloses a data real-time computing method based on Apache Flink open source framework development.
Further, debugging the structured query language instruction by using a preset data logic grammar comprises the following steps:
(d1) receiving a debugging instruction of the structured query language instruction sent by a preset visual interface;
(d2) and debugging the structured query statement by using a preset data logic grammar.
In the above actions (d1) and (d2), a debugging instruction for the structured query language instruction sent by the preset visualization interface is received, and the structured query statement is debugged by using the preset data logic syntax. In this embodiment, a one-key debugging function is provided on a preset visual interface, a debugging instruction for a structured query language instruction, which is operated on the preset visual interface by a user, is received, and a preset data logic grammar is used to perform grammar and logic debugging on a structured query statement.
Further, the embodiment of the present invention further includes: and storing the data calculation result into a database corresponding to the data. And storing the data calculation result into a database corresponding to the received data according to a preset structured query language business rule. For example, after the data source receiving mysql is calculated in real time, the calculated result is stored in the mysql database, and after the data source receiving hbase is calculated in real time, the calculated result is stored in the hbase database.
Fig. 3 is a schematic flow chart of a real-time data computing method according to another embodiment of the present disclosure, and as shown in fig. 3, the embodiment of the present invention includes the following steps;
s301: a user constructs a streaming data table and a dimension data table on a preset visual interface according to received data;
s302: a user inputs an SQL language instruction supporting the correlation calculation of the streaming data table and the dimension data table on a preset visual interface;
s303: debugging the input SQL language instruction according to a preset data logic grammar;
s304: receiving and analyzing the debugged SQL language instruction by the computing engine to generate a plurality of data computing units which can be processed by the distributed computing engine;
s305: performing distributed logic calculation on the plurality of data calculation units to obtain calculation results of corresponding data;
s306: and storing the calculation result of the data into a database corresponding to the data.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine by the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
Further, referring to fig. 1, according to a second aspect of the present embodiment, there is provided a storage medium. The storage medium includes a stored program, wherein the data real-time computing method of any one of the above is executed by a processor when the program is executed.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine by the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
The storage medium provided by the embodiment of the present application can implement the processes in the foregoing method embodiments, and achieve the same functions and effects, which are not repeated here.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
Example 2
Fig. 4 is a schematic diagram of a real-time data computing apparatus according to an embodiment of the present disclosure, and the apparatus 400 corresponds to a real-time data computing method according to embodiment 1. Referring to fig. 4, the apparatus 400 includes:
a language instruction receiving module 401, configured to receive a structured query language instruction input by a user and used for processing data;
a data unit generation module 402 for generating a plurality of data computation units capable of being processed by a distributed computation engine according to the structured query language instruction, wherein the distributed computation engine supports real-time computation of data; and
a data unit calculating module 403, configured to perform distributed calculation on the plurality of data calculating units by using the distributed calculating engine.
Optionally, a data table creating module 404 is further included, where the data is a data table created by a user on a preset visual operation interface, and before receiving a structured query language instruction input by the user for processing data:
acquiring a continuous data stream generated by a user terminal; and
and receiving the data table created by the user on the preset visual operation interface according to the data stream.
Optionally, the data unit generating module 402 is specifically configured to:
debugging the structured query language instruction by using a preset data logic grammar;
and generating a plurality of data computing units which can be processed by a distributed computing engine according to the debugged structured query language instruction.
Optionally, the data unit calculating module 403 is specifically configured to:
the distributed computing engine comprises a plurality of servers, the data computing units are distributed in the servers of the distributed computing engine, and the data computing units distributed in the servers are subjected to logic operation respectively;
and obtaining a data calculation result corresponding to the structured query language instruction after the plurality of servers perform logic operation.
Optionally, a specific framework application module 405 is also included: the device is applied to an open source flow processing framework.
Optionally, the data unit generating module 402 is further specifically configured to:
receiving a debugging instruction of the structured query language instruction sent by the preset visual interface;
and debugging the structured query statement by using the preset data logic grammar.
Optionally, the data result saving module 406 is further included: and the data calculation result is stored in a database corresponding to the data.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine by the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
The data real-time calculation method and device provided by the embodiment of the application can realize each process in the method embodiments and achieve the same functions and effects, and are not repeated here.
Example 3
Fig. 5 is a schematic diagram of a data real-time computing apparatus 500 according to another embodiment of the present disclosure, where the apparatus 500 corresponds to the method according to the first aspect of embodiment 1. Referring to fig. 5, the apparatus 500 includes: a processor 510; and a memory 520 coupled to processor 510 for providing processor 510 with instructions to process the following process steps:
receiving a structured query language instruction input by a user and used for processing data;
generating a plurality of data computation units capable of being processed by a distributed computation engine in accordance with the structured query language instructions, wherein the distributed computation engine supports real-time computation of data; and
performing distributed computation on the plurality of data computation units using the distributed computation engine.
Optionally, the data is a data table created by a user on a preset visual operation interface, and before receiving a structured query language instruction input by the user for processing the data, the memory 520 is further configured to provide the processor 510 with an instruction for processing the following processing steps:
acquiring a continuous data stream generated by a user terminal; and
and receiving the data table created by the user on the preset visual operation interface according to the data stream.
Optionally, generating a plurality of data computing units capable of being processed by a distributed computing engine according to the structured query language instructions comprises:
debugging the structured query language instruction by using a preset data logic grammar;
and generating a plurality of data computing units which can be processed by a distributed computing engine according to the debugged structured query language instruction.
Optionally, the distributed computing engine includes a plurality of servers, and performing distributed computing on the plurality of data computing units by using the distributed computing engine, including:
distributing the data computing units in a plurality of servers of the distributed computing engine, and respectively carrying out logic operation on the data computing units distributed in the servers;
and obtaining a data calculation result corresponding to the structured query language instruction after the plurality of servers perform logic operation.
Optionally, the apparatus is applied in an open source flow processing framework.
Optionally, debugging the structured query language instruction by using a preset data logic syntax includes:
receiving a debugging instruction of the structured query language instruction sent by the preset visual interface;
and debugging the structured query statement by using the preset data logic grammar.
Optionally, the memory 520 is further configured to provide the processor 510 with instructions to process the following process steps: and storing the data calculation result into a database corresponding to the data.
In the embodiment of the invention, a structured query language instruction which is input by a user and used for processing data is received, and a plurality of data computing units which can be processed by a distributed computing engine are generated according to the structured query language instruction, wherein the distributed computing engine supports real-time computing of the data; and performing distributed computation on the plurality of data computation units by using the distributed computation engine. The invention generates a plurality of data computing units which can be processed by the distributed computing engine by the structured query language instruction input by the user, and simultaneously computes the plurality of data computing units by the distributed computing engine, thereby improving the computing efficiency of data real-time computation.
The data real-time computing device provided by the embodiment of the application can realize each process in the foregoing method embodiments, and achieve the same functions and effects, which are not repeated here.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method for real-time computation of data, comprising:
receiving a structured query language instruction input by a user and used for processing data;
generating a plurality of data computation units capable of being processed by a distributed computation engine in accordance with the structured query language instructions, wherein the distributed computation engine supports real-time computation of data; and
performing distributed computation on the plurality of data computation units using the distributed computation engine.
2. The method according to claim 1, wherein the data is a data table created by a user in a preset visual operation interface, and before receiving a structured query language instruction input by the user for processing the data, the method further comprises:
acquiring a continuous data stream generated by a user terminal; and
and receiving the data table created by the user on the preset visual operation interface according to the data stream.
3. The method of claim 1, wherein generating a plurality of data computation elements from the structured query language instructions that are capable of being processed by a distributed computation engine comprises:
debugging the structured query language instruction by using a preset data logic grammar;
and generating a plurality of data computing units which can be processed by a distributed computing engine according to the debugged structured query language instruction.
4. The method of claim 1, wherein the distributed computing engine comprises a plurality of servers, and wherein performing distributed computing on the plurality of data computing units using the distributed computing engine comprises:
distributing the data computing units in a plurality of servers of the distributed computing engine, and respectively carrying out logic operation on the data computing units distributed in the servers;
and obtaining a data calculation result corresponding to the structured query language instruction after the plurality of servers perform logic operation.
5. The method of claim 1, further comprising: the method is applied in an open source flow processing framework.
6. The method of claim 3, wherein debugging the structured query language instructions using a predetermined data logic syntax comprises:
receiving a debugging instruction of the structured query language instruction sent by the preset visual interface;
and debugging the structured query statement by using the preset data logic grammar.
7. The method of claim 1, further comprising: and storing the data calculation result into a database corresponding to the data.
8. A storage medium comprising a stored program, wherein the data real-time computing method of any one of claims 1 to 7 is performed by a processor when the program is run.
9. A real-time computing device for data, comprising:
the language instruction receiving module is used for receiving a structured query language instruction which is input by a user and used for processing data;
a data unit generation module for generating a plurality of data computation units capable of being processed by a distributed computation engine according to the structured query language instruction, wherein the distributed computation engine supports real-time computation of data; and
and the data unit calculation module is used for performing distributed calculation on the plurality of data calculation units by utilizing the distributed calculation engine.
10. A real-time computing device for data, comprising:
a processor; and
a memory coupled to the processor for providing instructions to the processor for processing the following processing steps:
receiving a structured query language instruction input by a user and used for processing data;
generating a plurality of data computation units capable of being processed by a distributed computation engine in accordance with the structured query language instructions, wherein the distributed computation engine supports real-time computation of data; and
performing distributed computation on the plurality of data computation units using the distributed computation engine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911210038.0A CN112883049A (en) | 2019-11-29 | 2019-11-29 | Real-time data calculation method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911210038.0A CN112883049A (en) | 2019-11-29 | 2019-11-29 | Real-time data calculation method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112883049A true CN112883049A (en) | 2021-06-01 |
Family
ID=76039453
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911210038.0A Pending CN112883049A (en) | 2019-11-29 | 2019-11-29 | Real-time data calculation method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112883049A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090138437A1 (en) * | 2007-11-26 | 2009-05-28 | Microsoft Corporation | Converting sparql queries to sql queries |
| CN107391719A (en) * | 2017-07-31 | 2017-11-24 | 南京邮电大学 | Distributed stream data processing method and system in a kind of cloud environment |
| CN107562943A (en) * | 2017-09-22 | 2018-01-09 | 马上消费金融股份有限公司 | The method and system that a kind of data calculate |
| CN108694221A (en) * | 2017-04-12 | 2018-10-23 | 中国移动通信集团福建有限公司 | Data real-time analysis method, module, equipment and device |
| CN109799976A (en) * | 2019-01-11 | 2019-05-24 | 上海凯岸信息科技有限公司 | Real-time air control variable calculation method based on distributive type computing engines |
| CN110334070A (en) * | 2019-05-21 | 2019-10-15 | 中国人民财产保险股份有限公司 | Data processing method, system, equipment and storage medium |
-
2019
- 2019-11-29 CN CN201911210038.0A patent/CN112883049A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090138437A1 (en) * | 2007-11-26 | 2009-05-28 | Microsoft Corporation | Converting sparql queries to sql queries |
| CN108694221A (en) * | 2017-04-12 | 2018-10-23 | 中国移动通信集团福建有限公司 | Data real-time analysis method, module, equipment and device |
| CN107391719A (en) * | 2017-07-31 | 2017-11-24 | 南京邮电大学 | Distributed stream data processing method and system in a kind of cloud environment |
| CN107562943A (en) * | 2017-09-22 | 2018-01-09 | 马上消费金融股份有限公司 | The method and system that a kind of data calculate |
| CN109799976A (en) * | 2019-01-11 | 2019-05-24 | 上海凯岸信息科技有限公司 | Real-time air control variable calculation method based on distributive type computing engines |
| CN110334070A (en) * | 2019-05-21 | 2019-10-15 | 中国人民财产保险股份有限公司 | Data processing method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209309B (en) | Method, device and equipment for determining processing result of data flow graph and storage medium | |
| CN112559301B (en) | Service processing method, storage medium, processor and electronic device | |
| US10878006B2 (en) | Systems to interleave search results and related methods therefor | |
| CN108228754A (en) | flow generation method and terminal device | |
| CN107273140A (en) | Scaffold management method, device and electronic equipment | |
| CN112784155A (en) | Method, device and storage medium for matching service side | |
| CN111178937B (en) | User rewarding method and device for application, electronic equipment and readable storage medium | |
| CN109542398B (en) | Business system generation method and device and computer readable storage medium | |
| CN109828759A (en) | Code compiling method, device, computer installation and storage medium | |
| CN109657317A (en) | A kind of method, system and the equipment of CPLD pin assignments | |
| CN112883049A (en) | Real-time data calculation method and device and storage medium | |
| CN112947954A (en) | Interface updating method and device and storage medium | |
| CN110597717A (en) | Code testing method, device and storage medium | |
| CN111026995A (en) | Method and device for information association between applications and storage medium | |
| CN114817389A (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN110297748A (en) | The method, apparatus and computer readable storage medium of error are called in a kind of positioning | |
| CN113691403A (en) | Topological node configuration method, related device and computer program product | |
| CN109783134B (en) | Front-end page configuration method and device and electronic equipment | |
| CN111966734A (en) | Data processing method and electronic equipment of spreadsheet combined with RPA and AI | |
| CN110647543A (en) | Data aggregation method, device and storage medium | |
| CN112863475B (en) | Speech synthesis method, apparatus and medium | |
| CN110325966A (en) | Arithmetic inertia mark for emulation indicates | |
| CN110096255A (en) | Regular degradation processing method, apparatus and system, data processing method | |
| CN114283012A (en) | Market data pushing method and device and storage medium | |
| CN112685491B (en) | Log processing method, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |