CN110083625A - Realtime stream processing method, equipment, data processing equipment and medium - Google Patents
Realtime stream processing method, equipment, data processing equipment and medium Download PDFInfo
- Publication number
- CN110083625A CN110083625A CN201910205086.4A CN201910205086A CN110083625A CN 110083625 A CN110083625 A CN 110083625A CN 201910205086 A CN201910205086 A CN 201910205086A CN 110083625 A CN110083625 A CN 110083625A
- Authority
- CN
- China
- Prior art keywords
- processing
- configuration
- real
- sentence
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Devices For Executing Special Programs (AREA)
Abstract
The invention discloses a kind of realtime stream processing method, equipment, data processing equipment and media.A kind of realtime stream processing method, comprising: receive the normal data processing sentence write according to specific primitive rule that user provides；The normal data processing sentence is parsed by grammar parser, obtains input configuration, output configuration and processing configuration；And based on input configuration, output configuration and processing configuration, generate real time processing tasks.It can be by simply configuring input by the present invention in that obtaining user, analysis result can be generated based on time series data, it is programmed without user oneself to develop big data processing task, and also do not need the familiar computing engines of user oneself and storage engines to generate analytical statement.As a result, user only needs to write sentence using general normal data Processing Interface, real-time stream process can be realized, so that the present invention can substantially reduce the exploitation threshold of real-time stream process task.
Description
Technical field
The present invention relates to real-time streams processing technique more particularly to realtime stream processing method, equipment, data processing equipment and Jie
Matter.
Background technique
With big data, the development of processing in real time (such as calculating in real time), various real-time processing big data technology layers go out not
?.The programming interface of these real-time stream process engines is each
Different, user, which needs to pay very big learning cost, could be familiar with and use.
As a result, in order to enable user is easy to develop and handle real-time stream process task, need a kind of simple, effective, general
Real-time streams processing scheme.
Summary of the invention
One of in order to solve problem above, the present invention provides a kind of realtime stream processing method, equipment, data processing equipments
And medium.
For problem described in background technique, the present invention is considered as a kind of general normal data processing across language
Interface rather than handle real-time stream process task using above-mentioned different real-time stream process engine programming interface.By
This, the present invention proposes a kind of more common real-time streams handling implement.
An exemplary embodiment according to the present invention, provides a kind of realtime stream processing method, comprising: receives user and provides
Write according to specific primitive rule normal data processing sentence；It is parsed by grammar parser at the normal data
Sentence is managed, input configuration, output configuration and processing configuration are obtained；And based on input configuration, output configuration and add
Work configuration, generates real-time stream process task.
Optionally, above-mentioned realtime stream processing method further include: by the way that the real-time stream process task generated is published to collection
On group, is started the real-time stream process task by cluster and executed.
Optionally, the normal data that user provides is parsed by grammar parser and handles sentence, obtained resolution file, utilize
The resolution file, by the normal data processing sentence be converted into abstract syntax tree, with obtain input configuration, output configuration and
Processing configuration.
Optionally, the normal data processing sentence includes structured query language SQL statement.
Optionally, the primitive rule include: 1) for define about input configure field structure, data source types,
The information of access mode；2) for define about output configure field structure, data source types, access mode information；3)
For defining the information of the data handling procedure about processing configuration.
An exemplary embodiment according to the present invention, provides a kind of real-time device for processing streams, comprising: reception device, quilt
It is configured to receive the normal data processing sentence write according to specific primitive rule that user provides；Grammar parser, quilt
It is configured to parse the normal data processing sentence, obtains input configuration, output configuration and processing configuration；And task is raw
At device, it is configured for the input configuration, output configuration and processing configuration, generates real-time stream process task.
Optionally, above-mentioned real-time device for processing streams further include: task execution device is configured to the reality that will be generated
When stream process task be published on cluster, started the real-time stream process task by cluster and executed.
Optionally, the normal data that grammar parser parsing user provides handles sentence, obtains resolution file, utilizes the solution
File is analysed, normal data processing sentence is converted into abstract syntax tree, to obtain input configuration, output configuration and processing
Configuration.
Optionally, the normal data processing sentence includes structured query language SQL statement.
Optionally, the primitive rule include: 1) for define about input configure field structure, data source types,
The information of access mode；2) for define about output configure field structure, data source types, access mode information；3)
For defining the information of the data handling procedure about processing configuration.
An exemplary embodiment according to the present invention, provides a kind of data processing equipment, comprising: processor；And it deposits
Reservoir is stored thereon with executable code, when the executable code is executed by the processor, executes the processor
One of above method.
An exemplary embodiment according to the present invention, provides a kind of non-transitory machinable medium, deposits thereon
Executable code is contained, when the executable code is executed by processor, the processor is made to execute one of above method.
Through the invention, user only needs to write sentence using general normal data Processing Interface, and reality can be realized
When flow real-time stream process so that the present invention can substantially reduce the exploitation threshold of real-time stream process task.
Detailed description of the invention
Disclosure illustrative embodiments are described in more detail in conjunction with the accompanying drawings, the disclosure above-mentioned and its
Its purpose, feature and advantage will be apparent, wherein in disclosure illustrative embodiments, identical appended drawing reference
Typically represent same parts.
Fig. 1 gives the schematic flow chart of the realtime stream processing method of an exemplary embodiment according to the present invention.
Fig. 2 gives the structural representation for the abstract syntax tree being parsed into from SQL statement.
Content needed for Fig. 3 shows the abstract definition for real-time stream process and its relevant more detailed configuration.
Fig. 4 gives the schematic block diagram of the real-time device for processing streams of an exemplary embodiment according to the present invention.
Fig. 5 gives the schematic flow of the realtime stream processing method of another exemplary embodiment according to the present invention
Figure.
Fig. 6 gives the schematic block diagram of the real-time device for processing streams of another exemplary embodiment according to the present invention.
Fig. 7 gives the schematic block diagram of the data processing equipment of an exemplary embodiment according to the present invention.
Specific embodiment
The preferred embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Preferred embodiment, however, it is to be appreciated that may be realized in various forms the disclosure without the embodiment party that should be illustrated here
Formula is limited.On the contrary, these embodiments are provided so that this disclosure will be more thorough and complete, and can be by the disclosure
Range is completely communicated to those skilled in the art.What needs to be explained here is that number, serial number and attached drawing in the application
Mark it is merely for convenience description and occur, for step of the invention, sequence etc. be not limited in any way, unless
The execution that step has been explicitly pointed out in specification has specific sequencing.
As discussed in the summary of the invention section, the present invention be considered as a kind of general normal data Processing Interface across language and
It is not to handle real-time stream process using above-mentioned different real-time stream process engine programming interface described in background technology
Task.The present invention proposes a kind of more common real-time streams handling implement as a result,.Through the invention, user only needs using logical
Normal data Processing Interface writes sentence, real-time stream process can be realized, so that the present invention can substantially reduce
The exploitation threshold of real-time stream process task.
For example, structured query language SQL (Structured Query Language) can be used as the present invention optionally
A kind of general normal data Processing Interface across language, because it is almost required for each Software Development Engineer
Technical ability.
However, it is noted that the present invention is not limited in using SQL statement, as long as but any suitable logical across language
Normal data Processing Interface can use.
For easy understanding and for the sake of description, the present invention will be described by taking SQL statement as an example below.
Real-time stream process task is write in order to use SQL statement, and the present invention is considered first to common real-time
Stream process carries out abstract processing, it can real-time stream process task is abstracted into input (Input), process (Process) and
(Output) these three stages of output.
Wherein, processing (Process) process is complex, involves how to generate distributed treatment times by SQL statement
Business.In the present invention, it is considered as big data and executes frame (such as Spark/Flink) to realize to generate by SQL statement and divide
The function of cloth processing task.
As a result, the present invention is based on input (Input), processing (Process) and output (Output) these three stages,
A set of SQL primitive rule is defined, user directly can simply write SQL statement according to SQL primitive rule, then can make
With SQL resolver, the SQL statement that user is write is parsed into the abstract syntax tree of input, processing, output, according still further to big data
Frame (such as Spark Streaming, Flink) grammer is executed, abstract syntax tree is parsed into specific data processing task
To execute.
Firstly, for example, can to define SQL primitive rule as follows.
For example, primitive rule may include:
1) for define about input configure field structure, data source types, access mode information；
2) for define about output configure field structure, data source types, access mode information；And
3) for defining the information of the data handling procedure about processing configuration.
By taking SQL statement as an example, the primitive rule of SQL statement can be as follows.
1)Input:create input table input_table_name(column1,column2)with(url
=" xxx ", type=" kafka ") and/* be used to define the primitive rule * of " input "/
Should " input " SQL statement primitive rule meaning are as follows: create field structure as column1, column2,
The flow table that data source types are kafka, access mode is xxx.
The flow table reads data from real-time streaming data for defining how, for example defines the visit for reading real-time streaming data
Data structure etc. after asking mode and parsing.
2)Output:create output table output_table_name(column1,column2)with
The primitive rule * of (url=" xxx ", type=" hive ")/* definition " output "/
Should " output " SQL statement primitive rule meaning are as follows: create field structure as column1, column2,
The result table that data source types are hive, access mode is xxx.
The result table for defining how to write data to downstream storage, such as downstream storage access mode and write
The data structure etc. entered.
3)Process:insert into output_table_name select*frominput_table_name/*
The primitive rule * of definition " processing "/
It is somebody's turn to do the meaning of the primitive rule of " processing " SQL statement are as follows: be inserted into the data of flow table input_table_name
As a result in table output_table_name.Wherein, the definition " select* of the data handling procedure in " processing " SQL statement
From input_table_name " is realized by Spark SQL syntax.
Downstream storage could be written for defining how to convert after data are read from flow table by being somebody's turn to do " processing " SQL statement.
It note that above-mentioned primitive rule is only an example, it will be appreciated by those skilled in the art that, the present invention should not
It is restricted to above-mentioned example.
Next, specific exemplary embodiment will be enumerated below to illustrate it in order to enable the present invention is easier to understand
Realization process.
Fig. 1 gives the schematic flow chart of the realtime stream processing method of an exemplary embodiment according to the present invention.
As shown in Figure 1, receiving the standard universal number write according to specific primitive rule that user provides in step S110
According to processing sentence (for example, the SQL statement write according to specific SQL primitive rule).
Then, in step S120, the upper predicate of user's offer is parsed by grammar parser (such as SQL resolver)
Sentence parses input configuration (Input), output configuration (Output) and processing configuration (Process).
Here it is possible to be parsed sentence (for example, SQL statement) by means of the grammar parser Antlr of such as open source, word is done
The parsing of method, grammer generates Antlr resolution file (that is, the rule file that grammar parser antlr is defined, that is, SQL are former
The sequencing of language rule is expressed), to parse the configuration of the input in SQL statement (Input), output configuration (Output) and process
It configures (Process).
For example, by Antlr resolution file, the SQL statement of user can be converted into abstract syntax tree shown in Fig. 2.
Wherein, that shown in Fig. 2 is from left to right respectively " input " (Input), " output " (Output) and " processing "
(Process) in abstract syntax tree, each branch being made of the english word of the segment in SQL statement is according to them
What corresponding grammer was parsed.Specific explanations will be carried out to each abstract syntax tree of Fig. 2 below.
1) abstract syntax tree of " input " (Input) shows above-mentioned input configuration, that is, a field is column1,
The flow table input_table_name of column2, attribute url=xxx, type=kafka.
2) abstract syntax tree of " output " (Output) shows above-mentioned output configuration, that is, a field is column1,
The result table output_table_name of column2, attribute url=xxx, type=kafka.
3) abstract syntax tree of " processing " (Process) shows processing configuration, that is, can pass through SQL statement for all
The data that " select*from input_table_name " is checked out are all inserted into result table output_table_name.
Then, in step S130, the input configuration generated based on grammar parser, output configuration and processing configuration are raw
At real-time stream process task.
Specifically, it can be configured based on the input of Fig. 2, export configuration and processing configuration, generate big data processing task
(such as Spark Streaming task).For example, can carry out in accordance with the following steps:
1) grammer (such as the Spark of stream frame can be used according to the field in input configuration, attribute, flow table name
The grammer of Streaming or Flink etc.) Lai Dingyi flow table；
2) grammer (such as the Spark of stream frame can be used according to the field in output configuration, attribute, result table name
The grammer of Streaming or Flink etc.) generate result table；And
3) real-time stream process task can be generated according to the processing sentence in processing configuration.
It can be seen that the present invention is using a kind of general normal data Processing Interface across language rather than such as the prior art
Different real-time stream process engine programming interface is used, like that handle real-time stream process task.The present invention provides as a result,
A kind of more common and simple real-time streams handling implement.Through the invention, user only needs using general normal data
Processing Interface writes sentence, and real-time stream process can be realized, and appoints so that the present invention can substantially reduce real-time stream process
The exploitation threshold of business.
In order to further be easy to understand, relevant configuration needed for Fig. 3 shows the abstract definition for real-time stream process, this
A little configurations are defined in the standard universal data processing statement (such as SQL statement etc.) of user's input.
1) input (Input) configuration: wherein may include for field structure, data stream type (for example, Kafka,
ActiveMQ, RocketMQ etc.), the definition of flow table name etc..
In the present invention, can based on input configuration, using stream frame grammar (here, flow frame may include Spark,
Flink, Storm etc.) Lai Shengcheng flow table, the data in flow table will read when executing real-time stream process task from real-time streams.
2) processing (Process) configuration: including for defining how the processing sentence of processing real-time streams.
In the present invention, real-time stream process task can be generated, the real-time stream process according to the processing sentence in processing configuration
Task is used to handle the data in flow table according to the processing sentence in processing configuration, and data processed result is stored in
In following result tables.
3) output (Output) configuration: it is similar with input configuration, wherein may include for field structure, data stream type
The definition of (for example, Kafka, ActiveMQ, RocketMQ etc.), result table name etc..
In the present invention, real-time stream process task can be generated, the real-time stream process according to the processing sentence in processing configuration
Task is used to handle, and data processed result is saved the data in flow table according to the processing sentence in processing configuration
In result table.
Fig. 4 gives the schematic block diagram of the real-time device for processing streams of an exemplary embodiment according to the present invention.
As shown in figure 4, device for processing streams 100 may include reception device 110, grammar parser 120, task generation in real time
Device 130.
Wherein, reception device 110 can be configured for receiving that user provides writes according to specific primitive rule
Normal data handles sentence.
Grammar parser 120 can be configured for parsing the normal data processing sentence, obtain input configuration, output
Configuration and processing configuration；And
Task generating device 130 can be configured for the input configuration, output configuration and processing configuration, raw
At real-time stream process task.
Here, the operation and step shown in FIG. 1 of reception device 110, grammar parser 120, task generating device 130
The operation of S110, S120, S130 are similar, and details are not described herein.
Through the invention, user only needs to write sentence using general normal data Processing Interface, and reality can be realized
When stream process so that the present invention can substantially reduce the exploitation threshold of real-time stream process task.
Fig. 5 gives the schematic flow of the realtime stream processing method of another exemplary embodiment according to the present invention
Figure.
Wherein, step S210-S230 shown in fig. 5 is similar with step S110-S130 shown in FIG. 1, and details are not described herein.
It further, can also include step S140, for executing the real-time stream process task generated in step S230.
For example, can be by the way that the real-time stream process task generated to be published on cluster, then cluster starts the real-time streams
Processing task simultaneously executes.
Fig. 6 gives the schematic block diagram of the real-time device for processing streams of another exemplary embodiment according to the present invention.
As shown in fig. 6, device for processing streams 200 may include reception device 210, grammar parser 220, task generation in real time
Device 230 and task execution device 240.
Wherein, reception device 210 shown in fig. 6, grammar parser 220, task generating device 230 and task execution
Device 240 is similar with reception device 110 shown in fig. 5, grammar parser 120, task generating device 130, and details are not described herein.
In addition, task execution device 240 can be configured to the real-time stream process task generated being published to collection
On group, is started the real-time stream process task by cluster and executed.
Here, the operation of task execution device 140 is similar with the operation of step S140 shown in Fig. 4, and details are not described herein.
Through the invention, user only needs to write sentence using general normal data Processing Interface, and reality can be realized
When stream process so that the present invention can substantially reduce the exploitation threshold of real-time stream process task.
Referring to Fig. 7, data processing equipment 1 includes memory 10 and processor 20.
Processor 20 can be the processor of a multicore, also may include multiple processors.In some embodiments, locate
Reason device 20 may include a general primary processor and one or more special coprocessors, such as graphics processor
(GPU), digital signal processor (DSP) etc..In some embodiments, the circuit realization of customization can be used in processor 20,
Such as application-specific IC (ASIC, Application Specific Integrated Circuit) or scene can
Programmed logic gate array (FPGA, Field Programmable Gate Arrays).
Memory 10 may include various types of storage units, such as Installed System Memory, read-only memory (ROM), and forever
Long storage device.Wherein, ROM can store the static data of other modules needs of processor 20 or computer or refer to
It enables.Permanent storage can be read-write storage device.Permanent storage can be after computer circuit breaking not
The non-volatile memory device of the instruction and data of storage can be lost.In some embodiments, permanent storage device uses
Mass storage device (such as magnetically or optically disk, flash memory) is used as permanent storage.In other embodiment, permanently deposit
Storage device can be removable storage equipment (such as floppy disk, CD-ROM drive).Installed System Memory can be read-write storage equipment or
The read-write storage equipment of volatibility, such as dynamic random access memory.Installed System Memory can store some or all processors
The instruction and data needed at runtime.In addition, memory 10 may include the combination of any computer readable storage medium, packet
Include various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read only memory), disk and/or light
Disk can also use.In some embodiments, memory 10 may include removable storage equipment that is readable and/or writing,
Such as it is laser disc (CD), read-only digital versatile disc (such as DVD-ROM, DVD-dual layer-ROM), read-only Blu-ray Disc, super
Disc density, flash card (such as SD card, min SD card, Micro-SD card etc.), magnetic floppy disc etc..Computer-readable storage
Medium does not include carrier wave and the momentary electron signal by wirelessly or non-wirelessly transmitting.
Code can be handled by being stored on memory 10, when that can handle code by the processing of processor 20, can make processor
20 execute the method addressed above.
In addition, being also implemented as a kind of computer program or computer program product, the meter according to the method for the present invention
Calculation machine program or computer program product include the calculating for executing the above steps limited in the above method of the invention
Machine program code instruction.
Alternatively, the present invention can also be embodied as a kind of (or the computer-readable storage of non-transitory machinable medium
Medium or machine readable storage medium), it is stored thereon with executable code (or computer program or computer instruction code),
When the executable code (or computer program or computer instruction code) by electronic equipment (or calculate equipment, server
Deng) processor execute when, so that the processor is executed each step according to the above method of the present invention.
Those skilled in the art will also understand is that, various illustrative logical blocks, mould in conjunction with described in disclosure herein
Block, circuit and algorithm steps may be implemented as the combination of electronic hardware, computer software or both.
What flow chart and block diagram in attached drawing etc. showed the system and method for multiple embodiments according to the present invention can
The architecture, function and operation being able to achieve.In this regard, each box in flowchart or block diagram can represent a mould
A part of block, program segment or code, a part of the module, section or code include one or more for realizing rule
The executable instruction of fixed logic function.It should also be noted that in some implementations as replacements, the function of being marked in box
It can also be occurred with being different from the sequence marked in attached drawing.For example, two continuous boxes can actually be substantially in parallel
It executes, they can also be executed in the opposite order sometimes, and this depends on the function involved.It is also noted that block diagram and/
Or the combination of each box in flow chart and the box in block diagram and or flow chart, can with execute as defined in function or
The dedicated hardware based system of operation is realized, or can be realized using a combination of dedicated hardware and computer instructions.
Various embodiments of the present invention are described above, above description is exemplary, and non-exclusive, and
It is not limited to disclosed each embodiment.Without departing from the scope and spirit of illustrated each embodiment, for this skill
Many modifications and changes are obvious for the those of ordinary skill in art field.The selection of term used herein, purport
In the principle, practical application or improvement to the technology in market for best explaining each embodiment, or make the art
Other those of ordinary skill can understand each embodiment disclosed herein.
Claims (14)
1. a kind of realtime stream processing method characterized by comprising
Receive the normal data processing sentence write according to specific primitive rule that user provides；
The normal data processing sentence is parsed by grammar parser, is obtained input configuration, output configuration and processing and is matched
It sets；And
Based on input configuration, output configuration and processing configuration, real-time stream process task is generated.
2. realtime stream processing method according to claim 1, which is characterized in that further include:
By the way that the real-time stream process task generated to be published on cluster, is started the real-time stream process task by cluster and executed.
3. realtime stream processing method according to claim 1, which is characterized in that it is described by grammar parser to parse
Normal data processing sentence is stated, the operation for obtaining input configuration, output configuration and processing configuration includes:
The normal data that user provides is parsed by grammar parser and handles sentence, obtains resolution file, using the resolution file,
Normal data processing sentence is converted into abstract syntax tree, is matched with parsing the input in the normal data processing sentence
It sets, export configuration and processing configuration.
4. realtime stream processing method according to claim 1, which is characterized in that the normal data processing sentence includes knot
Structure query language SQL statement.
5. realtime stream processing method according to claim 1, which is characterized in that based on input configuration, output configuration
And processing configuration, the operation for generating real-time stream process task include:
According to field, the data stream type, flow table name in the input configuration, flow table, the flow table are generated using stream frame grammar
In data will when executing real-time stream process task from real-time streams read；
According to field, the data stream type, result table name in the output configuration, result table is generated using stream frame grammar, it should
As a result the data processed result of real-time stream process task will be written into table；And
According to the processing sentence in the processing configuration, real-time stream process task is generated, which is used for institute
The data stated in flow table are handled according to the processing sentence, and data processed result is stored in the result table.
6. realtime stream processing method according to any one of claims 1 to 5, which is characterized in that the primitive rule
Then include:
1) for define about input configure field structure, data source types, access mode information；
2) for define about output configure field structure, data source types, access mode information；
3) for defining the information of the data handling procedure about processing configuration.
7. a kind of real-time device for processing streams characterized by comprising
Reception device is configured for receiving the normal data processing language write according to specific primitive rule that user provides
Sentence；
Grammar parser is configured for parsing the normal data processing sentence, obtains input configuration, output configuration and adds
Work configuration；And
Task generating device is configured for the input configuration, output configuration and processing configuration, generates at real-time streams
Reason task.
8. real-time device for processing streams according to claim 7, which is characterized in that further include:
Task execution device is configured to for the real-time stream process task generated being published on cluster, be started by cluster
The real-time stream process task simultaneously executes.
9. real-time device for processing streams according to claim 7, which is characterized in that the grammar parser parsing user provides
Normal data handle sentence, obtain resolution file, using the resolution file, normal data processing sentence be converted into taking out
As syntax tree, to obtain input configuration, output configuration and processing configuration.
10. real-time device for processing streams according to claim 7, which is characterized in that the normal data handles sentence and includes
Structured query language SQL statement.
11. the real-time device for processing streams according to any one in claim 7~10, which is characterized in that the primitive
Rule includes:
1) for define about input configure field structure, data source types, access mode information；
2) for define about output configure field structure, data source types, access mode information；
3) for defining the information of the data handling procedure about processing configuration.
12. real-time device for processing streams according to claim 7, which is characterized in that task generating device is based on the input
Configuration, output configuration and processing configuration, the operation for generating real-time stream process task include:
According to field, the data stream type, flow table name in the input configuration, flow table, the flow table are generated using stream frame grammar
In data will when executing real-time stream process task from real-time streams read；
According to field, the data stream type, result table name in the output configuration, result table is generated using stream frame grammar, it should
As a result the data processed result of real-time stream process task will be written into table；And
According to the processing sentence in the processing configuration, real-time stream process task is generated, which is used for institute
The data stated in flow table are handled according to the processing sentence, and data processed result is stored in the result table.
13. a kind of data processing equipment, comprising:
Processor；And
Memory is stored thereon with executable code, when the executable code is executed by the processor, makes the processing
Device executes the method as described in any one of claim 1~6.
14. a kind of non-transitory machinable medium, is stored thereon with executable code, when the executable code is located
When managing device execution, the processor is made to execute the method as described in any one of claim 1~6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910205086.4A CN110083625A (en) | 2019-03-18 | 2019-03-18 | Realtime stream processing method, equipment, data processing equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910205086.4A CN110083625A (en) | 2019-03-18 | 2019-03-18 | Realtime stream processing method, equipment, data processing equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110083625A true CN110083625A (en) | 2019-08-02 |
Family
ID=67412476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910205086.4A Pending CN110083625A (en) | 2019-03-18 | 2019-03-18 | Realtime stream processing method, equipment, data processing equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110083625A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704474A (en) * | 2019-09-24 | 2020-01-17 | 杭州玳数科技有限公司 | Real-time SQL extension processing method and device based on Flink |
| CN111240693A (en) * | 2020-01-17 | 2020-06-05 | 北京三快在线科技有限公司 | Real-time data processing method, device, equipment and storage medium |
| CN111309757A (en) * | 2020-05-14 | 2020-06-19 | 深圳市赢时胜信息技术股份有限公司 | SQL interpreter and optimization method of HBase |
| CN111666296A (en) * | 2020-04-28 | 2020-09-15 | 中国平安财产保险股份有限公司 | SQL data real-time processing method and device based on Flink, computer equipment and medium |
| CN113468184A (en) * | 2021-07-16 | 2021-10-01 | 北京明略软件系统有限公司 | Configuration-based general stream type index calculation method and system |
| CN111666296B (en) * | 2020-04-28 | 2024-07-09 | 中国平安财产保险股份有限公司 | SQL data real-time processing method and device based on Flink, computer equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101158975A (en) * | 2007-11-21 | 2008-04-09 | 金蝶软件（中国）有限公司 | Method and system for accessing distinct type data-base |
| CN103064876A (en) * | 2012-10-30 | 2013-04-24 | 中国标准化研究院 | Method for obtaining spatial data servitization facing geographic information application |
| CN105045607A (en) * | 2015-09-02 | 2015-11-11 | 广东创我科技发展有限公司 | Method for achieving uniform interface of multiple big data calculation frames |
| CN105279286A (en) * | 2015-11-27 | 2016-01-27 | 陕西艾特信息化工程咨询有限责任公司 | Interactive large data analysis query processing method |
| CN106648674A (en) * | 2016-12-28 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Big data computing management method and system |
| CN106708854A (en) * | 2015-11-13 | 2017-05-24 | 博雅网络游戏开发（深圳）有限公司 | Data exporting method and apparatus |
-
2019
- 2019-03-18 CN CN201910205086.4A patent/CN110083625A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101158975A (en) * | 2007-11-21 | 2008-04-09 | 金蝶软件（中国）有限公司 | Method and system for accessing distinct type data-base |
| CN103064876A (en) * | 2012-10-30 | 2013-04-24 | 中国标准化研究院 | Method for obtaining spatial data servitization facing geographic information application |
| CN105045607A (en) * | 2015-09-02 | 2015-11-11 | 广东创我科技发展有限公司 | Method for achieving uniform interface of multiple big data calculation frames |
| CN106708854A (en) * | 2015-11-13 | 2017-05-24 | 博雅网络游戏开发（深圳）有限公司 | Data exporting method and apparatus |
| CN105279286A (en) * | 2015-11-27 | 2016-01-27 | 陕西艾特信息化工程咨询有限责任公司 | Interactive large data analysis query processing method |
| CN106648674A (en) * | 2016-12-28 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Big data computing management method and system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704474A (en) * | 2019-09-24 | 2020-01-17 | 杭州玳数科技有限公司 | Real-time SQL extension processing method and device based on Flink |
| CN111240693A (en) * | 2020-01-17 | 2020-06-05 | 北京三快在线科技有限公司 | Real-time data processing method, device, equipment and storage medium |
| CN111666296A (en) * | 2020-04-28 | 2020-09-15 | 中国平安财产保险股份有限公司 | SQL data real-time processing method and device based on Flink, computer equipment and medium |
| CN111666296B (en) * | 2020-04-28 | 2024-07-09 | 中国平安财产保险股份有限公司 | SQL data real-time processing method and device based on Flink, computer equipment and medium |
| CN111309757A (en) * | 2020-05-14 | 2020-06-19 | 深圳市赢时胜信息技术股份有限公司 | SQL interpreter and optimization method of HBase |
| CN113468184A (en) * | 2021-07-16 | 2021-10-01 | 北京明略软件系统有限公司 | Configuration-based general stream type index calculation method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110083625A (en) | Realtime stream processing method, equipment, data processing equipment and medium | |
| CN108228166B (en) | Template-based back-end code generation method and system | |
| US9239710B2 (en) | Programming language transformations with abstract syntax tree extensions | |
| US8694962B2 (en) | Aspect-oriented parallel programming language extensions | |
| CN114625732B (en) | Query method and system based on structured query language SQL | |
| AU2014315619B2 (en) | Methods and systems of four-valued simulation | |
| US8584103B2 (en) | Reducing parallelism of computer source code | |
| CN108984155A (en) | Flow chart of data processing setting method and device | |
| CN110673839B (en) | Distributed tool configuration construction generation method and system | |
| von Pilgrim et al. | Constructing and visualizing transformation chains | |
| US20180129712A1 (en) | Data provenance and data pedigree tracking | |
| CN110955714A (en) | Method and device for converting unstructured text into structured text | |
| CN112860730A (en) | SQL statement processing method and device, electronic equipment and readable storage medium | |
| US11573790B2 (en) | Generation of knowledge graphs based on repositories of code | |
| CN105824647A (en) | Form page generating method and device | |
| Ritter et al. | Hardware accelerated application integration processing: Industry paper | |
| CN112860264B (en) | Method and device for reconstructing abstract syntax tree | |
| CN110109990A (en) | Time series data processing method, equipment, data processing equipment and medium | |
| CN105867886B (en) | Method and device for writing table | |
| CN117112608A (en) | Antlr 4-based database statement conversion method and device | |
| Schubert et al. | Automata theoretic account of proof search | |
| CN112883088A (en) | Data processing method, device, equipment and storage medium | |
| CN109491643B (en) | Flow description, analysis and execution method and device, data processing equipment and medium | |
| US10509659B1 (en) | Input processing logic to produce outputs for downstream systems using configurations | |
| Lukavsky | Building Big Data Pipelines with Apache Beam: Use a single programming model for both batch and stream data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190802 |