CN106991166A - A kind of big data processing method and processing device - Google Patents
A kind of big data processing method and processing device Download PDFInfo
- Publication number
- CN106991166A CN106991166A CN201710209069.9A CN201710209069A CN106991166A CN 106991166 A CN106991166 A CN 106991166A CN 201710209069 A CN201710209069 A CN 201710209069A CN 106991166 A CN106991166 A CN 106991166A
- Authority
- CN
- China
- Prior art keywords
- big data
- data system
- operation task
- target
- pending program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24549—Run-time optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiments of the invention provide a kind of big data processing method and processing device, methods described includes：Obtain pending program；According to the pending program, the first operation task that the pending program is performed is determined；First operation task is converted into the second operation task；Second operation task is the operation task that can be performed in the big data system being adapted in advance；The big data system being adapted in advance is to include multiple big data systems including target big data system；Search the target big data system；Second operation task is performed in the target big data system.Pass through big data processing method and processing device provided in an embodiment of the present invention, it is possible to achieve the complexity of reduction program development and the use threshold of user.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of big data processing method and processing device.
Background technology
As computer technology is socially reintegrated life comprehensively, information explosion has run up to one and has started to trigger what is changed
Degree.21 century is the epoch of data message great development, and mobile Internet, ecommerce, social networks etc. is greatly enlarged mutually
The border of networking and application, various data expand rapidly and become big.Internet, mobile Internet, security monitoring, gold
Melt etc. and all to produce data in madness.In order to obtain the value in big data, big data analytical technology is arisen at the historic moment, also expedited the emergence of
Many big data analysis systems, such as Hadoop, Spark, Tez etc..The characteristics of different big data analysis systems possesses different,
Such as Hadoop is good at batch processing task, and Spark is good at real time processing tasks, and different big data analysis systems all has respective
Application.These big data analysis systems have respective framework and developer's interface, the exploitation that different systems is used
Language is different.
When solving a specific big data problem analysis, user may use different systems, for example, using
During Hadoop and Spark systems, because Hadoop system uses Java language, Spark framework uses Scala languages
Speech, if user wants to carry out big data analysis using these systems, it is necessary to grasp different development languages simultaneously, needs simultaneously
Understand the framework of each system.As can be seen here, in the prior art, user wants to carry out when a big data is analyzed, it is necessary to grasp
Develop ability it is more, use threshold high.
The content of the invention
The purpose of the embodiment of the present invention is to provide a kind of big data processing method and processing device, by the operation of pending program
Task Switching is the operation task that can be performed in the big data system of user's request, to realize the complexity of reduction program development
With the use threshold of user.Concrete technical scheme is as follows：
The invention provides a kind of big data processing method, methods described includes：
Obtain pending program；
According to the pending program, the first operation task that the pending program is performed is determined；
First operation task is converted into the second operation task；Second operation task can be adapted in advance
The operation task performed in big data system；The big data system being adapted in advance be comprising target big data system including
Multiple big data systems；
Search the target big data system；
Second operation task is performed in the target big data system.
Specifically, the pending program is the pending program based on sql like language.
Specifically, it is described according to the pending program, determine the first operation task that the pending program is performed, bag
Include：
Character string in the pending program is converted into word sequence, the first analysis result is obtained；Described first
Analysis result is the pending program operation to be performed；According to first analysis result, the pending program is determined
The first operation task performed.
Specifically, described be converted to the second operation task by the first operation task, including：
First operation task is sent to big data system with unified interface, being adapted in advance, it is described to have
Unified interface, the big data system that is adapted in advance are by first operation task, and being converted to can be in the target big data system
Second operation task performed in system.
Specifically, the lookup target big data system, including：
Second operation task is subjected to morphology parsing, identification includes the sentence of the target big data system, obtained
To the second analysis result；
According to second analysis result, the target big data system is searched.
Specifically, it is described perform second operation task in the target big data system after, methods described is also
Including：
The result that second operation task is performed in the target big data system is stored in the big number of the target
According in system.
Present invention also offers a kind of big data processing unit, described device includes：
Acquisition module, for obtaining pending program；
Parsing module, for determining the first operation task that the pending program is performed；
Modular converter, for first operation task to be converted into the second operation task；Second operation task is
The operation task that can be performed in the big data system being adapted in advance；The big data system being adapted in advance is big comprising target
Multiple big data systems including data system；
Searching modul, for searching the target big data system；
Performing module, for performing second operation task in the target big data system.
Specifically, the acquisition module, specifically for obtaining the pending program based on sql like language.
Specifically, the parsing module, including：
First analyzing sub-module, for the character string in the pending program to be converted into word sequence, obtain
One analysis result；First analysis result is the pending program operation to be performed；
Second analyzing sub-module, for according to first analysis result, determining the pending program is performed first
Operation task.
Specifically, the modular converter, with unified interface, pre- specifically for first operation task is sent to
The big data system being first adapted to, with unified interface, the big data system that is adapted in advance by first operation task,
Be converted to second operation task that can be performed in the target big data system.
Specifically, the modular converter, including：
First transform subblock, for second operation task to be carried out into syntax parsing, identification includes the target
The sentence of big data system, obtains the second analysis result；
Second transform subblock, for according to second analysis result, searching the target big data system.
Specifically, the performing module, specifically for performing second operation in the target big data system and appointing
After business, described device also includes：
Sub-module stored, for the result that second operation task is performed in the target big data system to be stored
In the target big data system.
Big data processing method and processing device provided in an embodiment of the present invention, energy is converted to by the operation task of pending program
The operation task performed in the big data system of user's request, to realize the complexity of reduction program development and the use of user
Threshold.Certainly, any product or method for implementing the present invention it is not absolutely required to while reaching all the above advantage.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is the flow chart of big data processing method of the embodiment of the present invention；
Fig. 2 is the flow chart that the embodiment of the present invention determines the first operation task that pending program is performed；
Fig. 3 is the flow chart that the embodiment of the present invention searches target big data system；
Fig. 4 performs frame diagram for a kind of system provided in an embodiment of the present invention；
Fig. 5 is the structural representation of big data processing unit of the embodiment of the present invention；
Fig. 6 is the structural representation of parsing module of the embodiment of the present invention；
Fig. 7 is the structural representation of modular converter of the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
As computer technology is socially reintegrated life comprehensively, information explosion has run up to one and has started to trigger what is changed
Degree.21 century is the epoch of data message great development, and mobile Internet, ecommerce, social networks etc. is greatly enlarged mutually
The border of networking and application, various data expand rapidly and become big.Internet, mobile Internet, security monitoring, gold
Melt etc. and all to produce data in madness.In order to obtain the value in big data, big data analytical technology is arisen at the historic moment, also expedited the emergence of
Many big data analysis systems, such as Hadoop, Spark, Tez etc..The characteristics of different big data analysis systems possesses different,
Such as Hadoop is good at batch processing task, and Spark is good at real time processing tasks, and different big data analysis systems all has respective
Application.These big data analysis systems have respective framework and developer's interface, the exploitation that different systems is used
Language is different.
When solving a specific big data problem analysis, user may use different systems, for example, using
During Hadoop and Spark systems, because Hadoop system uses Java language, Spark framework uses Scala languages
Speech, if user wants to carry out big data analysis using these systems, it is necessary to grasp different development languages simultaneously, needs simultaneously
Understand the framework of each system.As can be seen here, in the prior art, user wants to carry out when a big data is analyzed, it is necessary to grasp
Develop ability it is more, use threshold high.
In order to reduce the use threshold of user and the complexity of big data exploitation, the invention provides a kind of processing of big data
Method and device, is described in detail individually below.
Fig. 1 is the flow chart of big data processing method of the embodiment of the present invention, 1 pair of reference picture big data processing method of the present invention
It is described in detail, specifically includes following steps：
Step 101, pending program is obtained；
One section of pending program is obtained, wherein, pending program bag contains what the big data system not parsed to be performed
Operation；
Step 102, according to pending program, the first operation task that pending program is performed is determined；
For the pending program acquired, pending program institute operation to be performed task is parsed, analysis result is
First operation task.
Step 103, the first operation task is converted into the second operation task；
Second operation task is the operation task that can be performed in the big data system being adapted in advance；The big number being adapted in advance
It is to include multiple big data systems including target big data system according to system；
First operation task is converted to the second operation task that can be performed in the big data system being adapted in advance, in advance
The big data system of adaptation is a system framework included including multiple big data systems, passes through the big data being adapted in advance
System framework, can be entered in different big data systems, big according to the target that the second operation task enters user's request
In data system.
Step 104, target big data system is searched；
Execution task in the second operation task, search target big data system, the second operation task except comprising
Beyond pending program operation to be performed, the performing environment of pending PROGRAMMED REQUESTS, i.e. target big data system are also included.
Step 105, the second operation task is performed in target big data system；
Find after required target big data system, according to the content of the second operation task in target big data system
Operation is performed, big data processing task is completed.
Big data processing method provided in an embodiment of the present invention, the operation task of pending program is converted to can be in user
The operation task performed in the big data system of demand, to realize the complexity of reduction program development and the use threshold of user,
Wherein, the big data system of user's request is in adapted big data system.Pending program solution of the invention by acquisition
The operation task that can be performed in the big data system of user's request, i.e. target big data system is analysed into, is completed to big data
Processing, the operation maintenance personnel of big data system, it is not necessary to different interfaces are provided for different big data systems, big number is can be achieved with
According to processing task, the complexity of exploitation is greatly reduced, user is easier to operate to.
Specifically, pending program is the pending program based on sql like language；
, it is necessary to which the major function of application is the operation such as extraction, conversion to data in big data analysis, it is substantially based on
Database table is operated, and these operations can use sql like language to be operated, because the theoretical foundation of sql like language is relational algebra,
User is using fairly simple, so many big data analysis engines provide SQL interfaces, the pending program of the embodiment of the present invention
Method for optimizing be the program for being taken based on sql like language, be illustrated below：
Sql like language programs simple case：
1 select k,v,d from table1；
2 Project[k,v,d]；
3 Relation[k,v,d,other]table1；
4 Scan table1withFields[k,v,d]；
5 SparkInterface.scan(“table1”)withFields[k,v,d]；
This section of code is meant, k is selected from table table1, and this operation projection is built a work by v, the value that d tri- is arranged
Journey Project, scan table table1, and k, v are taken out, the value that d tri- is arranged, the meaning of the 5th line program is to use Spark system conducts
The big data system that bottom is performed, just in Spark system scan table table1, and takes out k, v, the value that d tri- is arranged, and tie extracting
Fruit is submitted in the cluster of Spark systems；If it is desired that the big data system performed with Tez systems as bottom, then the 5th stroke
Sequence code is changed into TezInterface.scan (" table1 ") withFields [k, v, d], just in Tez system scan tables
Table1, and k, v, the value that d tri- is arranged are taken out, and be submitted to result is extracted in the cluster of Tez systems.The embodiment of the present invention is preferential
Using the program based on sql like language, it is easily understood, it is not necessary to grasp the language of corresponding system.Certainly, the embodiment of the present invention also may be used
To use other language, such as Java, Scala, can equally reach identical effect, but can be to the development ability of user
It there are certain requirements.
According to pending program, the first operation task that pending program is performed, specific steps are determined as shown in Fig. 2 bag
Include：
Step 201, the character string in pending program is converted into word sequence, obtains the first analysis result；
Step 202, according to the first analysis result, the first operation task that pending program is performed is determined.
First analysis result is pending program operation to be performed, and pending program is carried out into morphology parsing, morphology solution
Analysis is that the character string in pending program language is converted into word sequence, including keyword, character string, identifier, computing
Symbol, the definition of comment line, determine the pending program operation to be carried out, for example, the sql like language programming simple case of above-mentioned offer
In, computer reads this section of code and meant, k is selected from table table1, and v, the value that d tri- is arranged determines pending program
The first operation task, that is to say, that this is the parsing of morphology, and the content of the first analysis result is exactly the first operation task.
Specifically, the first operation task is converted into the second operation task, including：
First operation task is sent to big data system with unified interface, being adapted in advance, with unified interface
, the big data system being adapted in advance by the first operation task, be converted to the second behaviour that can be performed in target big data system
Make task.
Big data processing method provided in an embodiment of the present invention, by the big data system being adapted in advance, wherein including
The target big data system of big data technical problem is handled, the big data system being adapted in advance includes multiple big data systems,
When handling a big data problem, it may be necessary to use multiple big data systems, the embodiment of the present invention is big what is be adapted in advance
There is provided a unified interface on data system, by this interface, order is performed in target big data system.System
One interface, when needing to operate in multiple big data systems so as to user, realizes flexibly switching.
Specifically, searching the process of target big data system as shown in figure 3, comprising the following steps：
Step 301, the second operation task is subjected to syntax parsing, identification includes the language of the target big data system
Sentence, obtains the second analysis result；
Step 302, according to the second analysis result, target big data system is searched.
Syntax parsing is to be based on morphology resolution rules, identifies the particular statement of pending program, is obtained according to step 202
The second operation task, by the second operation task carry out syntax parsing, the second analysis result, the second analysis result are obtained after parsing
In include the target big data system of pending order.
Fig. 4 performs frame diagram for a kind of system provided in an embodiment of the present invention, and above-mentioned Fig. 2 and Fig. 3 is combined, big with target
Illustrated exemplified by data system.
Step 401, pending program is obtained；
Step 402, configuration processor is treated to be parsed；
Step 403, performed in target big data system.
After pending program is got, configuration processor is treated respectively and carries out morphology parsing and syntax parsing, analysis result
To obtain pending program operation to be performed and performing the target big data system of operation, target big data system is fitted in advance
In the big data system matched somebody with somebody, the operation that performs of pending program is performed in target big data system, the processing of big data is completed.
Perform after the second operation task, specifically also include in target big data system：
The result that the second operation task is performed in target big data system is stored in target big data system.
After the second operation task is performed in target big data system, target big data system can store the execution knot
Really, and by the implementing result submit in the cluster for performing big data system, to be subsequently managed to result.Knot will be performed
Fruit is stored in the advantage of target big data system, when computer receives identical again performs order, for example, from table
K, v are selected in table1, during the value that d tri- is arranged, computer directly can read the implementing result in target big data system, storage
Implementing result can improve operating efficiency when analyzing same big data problem.
Big data processing method provided in an embodiment of the present invention, the operation task of pending program is converted to can be in user
The operation task performed in the big data system of demand, to realize the complexity of reduction program development and the use threshold of user,
Wherein, the big data system of user's request is in adapted big data system.The present invention is excellent by the pending program of acquisition
Sql like language is first used, because sql like language is easily understood, as interface, is connected in different big data systems, further drop
The low use threshold of user, pending program is parsed into can be in the big data system of user's request, i.e. target big data system
The operation task performed in system, completes the processing to big data；Implementing result is stored in target big data system, Ke Yi
When analyzing same big data problem, result is directly read, operating efficiency is improved.Big data processing provided in an embodiment of the present invention
Method, it is not necessary to different interfaces are provided for different big data systems, just big data task can be handled, greatly reduce exploitation
Complexity, user is easier to operate to.
Present invention also offers a kind of big data processing unit, Fig. 5 is the knot of big data processing unit of the embodiment of the present invention
Structure schematic diagram, including：
Acquisition module 501, for obtaining pending program；
Parsing module 502, for determining the first operation task that pending program is performed；
Modular converter 503, for the first operation task to be converted into the second operation task；Second operation task is can be pre-
The operation task performed in the big data system being first adapted to；The big data system being adapted in advance is to exist comprising target big data system
Interior multiple big data systems；
Searching modul 504, for searching target big data system；
Performing module 505, for performing the second operation task in target big data system.
Big data processing unit provided in an embodiment of the present invention, the operation task of pending program is converted to can be in user
The operation task performed in the big data system of demand, to realize the complexity of reduction program development and the use threshold of user,
Wherein, the big data system of user's request is in adapted big data system.Pending program solution of the invention by acquisition
The operation task that can be performed in the big data system of user's request, i.e. target big data system is analysed into, is completed to big data
Processing, the operation maintenance personnel of big data system, it is not necessary to different interfaces are provided for different big data systems, can be achieved with big
Data processing task, greatly reduces the complexity of exploitation, and user is easier to operate to.
Specifically, acquisition module 501, for obtaining the pending program based on sql like language.
Specifically, the apparatus structure schematic diagram of parsing module 502 as shown in fig. 6, including：
First analyzing sub-module 601, for the character string in pending program to be converted into word sequence, obtains first
Analysis result；First analysis result is the pending program operation to be performed；
Second analyzing sub-module 602, for according to the first analysis result, determining that the first operation that pending program is performed is appointed
Business.
Modular converter 503, specifically for the first operation task is sent into big number with unified interface, being adapted in advance
According to system, the big data system with unified interface, being adapted in advance is by the first operation task, and being converted to can be big in target
The second operation task performed in data system.
Specifically, the apparatus structure schematic diagram of modular converter 503 as shown in fig. 7, comprises：
First transform subblock 701, for the second operation task to be carried out into syntax parsing, it is big that identification includes the target
The sentence of data system, obtains the second analysis result；
Second transform subblock 702, for according to the second analysis result, searching target big data system.
Specifically, performing module 505, for performing the second operation task in target big data system after, in addition to：
Sub-module stored is big for the result that the second operation task is performed in target big data system to be stored in into target
In data system.
It should be noted that the device of the embodiment of the present invention is the device using above-mentioned big data processing method, then it is above-mentioned
All embodiments of big data processing method are applied to the device, and can reach same or analogous beneficial effect.
It should be noted that herein, such as first and second or the like relational terms are used merely to a reality
Body or operation make a distinction with another entity or operation, and not necessarily require or imply these entities or deposited between operating
In any this actual relation or order.Moreover, term " comprising ", "comprising" or its any other variant are intended to
Nonexcludability is included, so that process, method, article or equipment including a series of key elements not only will including those
Element, but also other key elements including being not expressly set out, or also include being this process, method, article or equipment
Intrinsic key element.In the absence of more restrictions, the key element limited by sentence "including a ...", it is not excluded that
Also there is other identical element in process, method, article or equipment including the key element.
Each embodiment in this specification is described by the way of related, identical similar portion between each embodiment
Divide mutually referring to what each embodiment was stressed is the difference with other embodiment.It is real especially for system
Apply for example, because it is substantially similar to embodiment of the method, so description is fairly simple, related part is referring to embodiment of the method
Part explanation.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent substitution and improvements made within the spirit and principles in the present invention etc., are all contained in protection scope of the present invention
It is interior.
Claims (12)
1. a kind of big data processing method, it is characterised in that methods described includes：
Obtain pending program；
According to the pending program, the first operation task that the pending program is performed is determined；
First operation task is converted into the second operation task；Second operation task is can be in the big number being adapted in advance
According to the operation task performed in system；The big data system being adapted in advance is comprising multiple including target big data system
Big data system；
Search the target big data system；
Second operation task is performed in the target big data system.
2. according to the method described in claim 1, it is characterised in that the pending program is based on the pending of SQ L language
Program.
3. method according to claim 2, it is characterised in that described according to the pending program, it is determined that described wait to hold
The first operation task that line program is performed, including：
Character string in the pending program is converted into word sequence, the first analysis result is obtained；First parsing
As a result it is the pending program operation to be performed；
According to first analysis result, the first operation task that the pending program is performed is determined.
4. method according to claim 3, it is characterised in that described that first operation task is converted into the second operation times
Business, including：
First operation task is sent to big data system with unified interface, being adapted in advance, it is described that there is unification
Interface, the big data system that is adapted in advance are by first operation task, and being converted to can be in the target big data system
Second operation task performed.
5. method according to claim 4, it is characterised in that the lookup target big data system, including：
Second operation task is subjected to syntax parsing, identification includes the sentence of the target big data system, obtains the
Two analysis results；
According to second analysis result, the target big data system is searched.
6. the method according to any one of claim 1 to 5, it is characterised in that described in the target big data system
Perform after second operation task, methods described also includes：
The result that second operation task is performed in the target big data system is stored in the target big data system
In system.
7. a kind of big data processing unit, it is characterised in that described device includes：
Acquisition module, for obtaining pending program；
Parsing module, for determining the first operation task that the pending program is performed；
Modular converter, for first operation task to be converted into the second operation task；Second operation task is can be
The operation task performed in the big data system being adapted in advance；The big data system being adapted in advance is comprising target big data
Multiple big data systems including system；
Searching modul, for searching the target big data system；
Performing module, for performing second operation task in the target big data system.
8. device according to claim 7, it is characterised in that the acquisition module, specifically for obtaining and being based on SQL languages
The pending program of speech.
9. device according to claim 8, it is characterised in that the parsing module, including：
First analyzing sub-module, word sequence is converted into by the character string in the pending program, obtains the first parsing knot
Really；First analysis result is the pending program operation to be performed；
Second analyzing sub-module, for according to first analysis result, determining the first operation that the pending program is performed
Task.
10. device according to claim 9, it is characterised in that the modular converter, specifically for described first is operated
Task is sent to big data system with unified interface, being adapted in advance, it is described it is with unified interface, be adapted in advance it is big
Data system is by first operation task, and second operation that being converted to can perform in the target big data system is appointed
Business.
11. device according to claim 10, it is characterised in that the modular converter, including：
First transform subblock, for second operation task to be carried out into syntax parsing, identification includes the big number of the target
According to the sentence of system, the second analysis result is obtained；
Second transform subblock, for according to second analysis result, searching the target big data system.
12. the device according to any one of claim 7 to 11, it is characterised in that the performing module, specifically in institute
State after performing second operation task in target big data system, described device also includes：
Sub-module stored, for the result that second operation task is performed in the target big data system to be stored in into institute
State in target big data system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710209069.9A CN106991166A (en) | 2017-03-31 | 2017-03-31 | A kind of big data processing method and processing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710209069.9A CN106991166A (en) | 2017-03-31 | 2017-03-31 | A kind of big data processing method and processing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106991166A true CN106991166A (en) | 2017-07-28 |
Family
ID=59415335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710209069.9A Pending CN106991166A (en) | 2017-03-31 | 2017-03-31 | A kind of big data processing method and processing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106991166A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110333941A (en) * | 2019-06-28 | 2019-10-15 | 苏宁消费金融有限公司 | A kind of real-time computing platform of big data based on sql and method |
| CN111506399A (en) * | 2020-03-05 | 2020-08-07 | 百度在线网络技术（北京）有限公司 | Task migration method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103150200A (en) * | 2013-04-08 | 2013-06-12 | 西安电子科技大学 | Computer language transformation system and transformation method from C language to MSVL (Modeling, Simulation and Verification Language) |
| CN103440303A (en) * | 2013-08-21 | 2013-12-11 | 曙光信息产业股份有限公司 | Heterogeneous cloud storage system and data processing method thereof |
| CN106095398A (en) * | 2016-05-10 | 2016-11-09 | 深圳前海信息技术有限公司 | Big data mining application process based on DSL and device |
| CN106383886A (en) * | 2016-09-21 | 2017-02-08 | 深圳市博瑞得科技有限公司 | Big data distribution type programming framework-based big data pre-statistics system and method |
-
2017
- 2017-03-31 CN CN201710209069.9A patent/CN106991166A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103150200A (en) * | 2013-04-08 | 2013-06-12 | 西安电子科技大学 | Computer language transformation system and transformation method from C language to MSVL (Modeling, Simulation and Verification Language) |
| CN103440303A (en) * | 2013-08-21 | 2013-12-11 | 曙光信息产业股份有限公司 | Heterogeneous cloud storage system and data processing method thereof |
| CN106095398A (en) * | 2016-05-10 | 2016-11-09 | 深圳前海信息技术有限公司 | Big data mining application process based on DSL and device |
| CN106383886A (en) * | 2016-09-21 | 2017-02-08 | 深圳市博瑞得科技有限公司 | Big data distribution type programming framework-based big data pre-statistics system and method |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110333941A (en) * | 2019-06-28 | 2019-10-15 | 苏宁消费金融有限公司 | A kind of real-time computing platform of big data based on sql and method |
| CN110333941B (en) * | 2019-06-28 | 2021-08-24 | 苏宁消费金融有限公司 | Big data real-time calculation method based on sql |
| CN111506399A (en) * | 2020-03-05 | 2020-08-07 | 百度在线网络技术（北京）有限公司 | Task migration method and device, electronic equipment and storage medium |
| WO2021174791A1 (en) * | 2020-03-05 | 2021-09-10 | 百度在线网络技术（北京）有限公司 | Task migration method and apparatus, and electronic device and storage medium |
| US11822957B2 (en) | 2020-03-05 | 2023-11-21 | Baidu Online Network Technology (Beijing) Co., Ltd. | Task migration method, apparatus, electronic device and storage medium |
| CN111506399B (en) * | 2020-03-05 | 2024-03-22 | 百度在线网络技术（北京）有限公司 | Task migration method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106168965B (en) | Knowledge graph construction system | |
| CN106104533B (en) | Handle the data set in large data repository | |
| US9122540B2 (en) | Transformation of computer programs and eliminating errors | |
| CN109614432B (en) | System and method for acquiring data blood relationship based on syntactic analysis | |
| CN108694221B (en) | Data real-time analysis method, module, equipment and device | |
| EP3671526A1 (en) | Dependency graph based natural language processing | |
| CN112860727B (en) | Data query method, device, equipment and medium based on big data query engine | |
| CN109145009B (en) | SQL-based method for searching ElasticSearch | |
| CN109947770A (en) | A kind of data base query method, terminal device and storage medium | |
| CN111008020A (en) | Method for analyzing logic expression into general query statement | |
| CN113672628A (en) | Data blood margin analysis method, terminal device and medium | |
| CN111079408A (en) | Language identification method, device, equipment and storage medium | |
| CN114817243A (en) | Method, device and equipment for establishing database joint index and storage medium | |
| CN106991166A (en) | A kind of big data processing method and processing device | |
| CN104424399B (en) | A kind of method, apparatus of the knowledge navigation based on virus protein body | |
| US9208134B2 (en) | Methods and systems for tokenizing multilingual textual documents | |
| CN104462461B (en) | The method and device of investigation processing empty value is carried out to list | |
| CN107679055B (en) | Information retrieval method, server and readable storage medium | |
| CN106570058A (en) | Searching method and search engine | |
| CN113568969B (en) | Information extraction method, apparatus, device and computer readable storage medium | |
| CN111159213A (en) | Data query method, device, system and storage medium | |
| CN111221841A (en) | Real-time processing method and device based on big data | |
| CN114064601B (en) | Storage process conversion method, device, equipment and storage medium | |
| CN105930453A (en) | Repeatability analyzing method and device | |
| CN112527880B (en) | Method, device, equipment and medium for collecting metadata information of big data cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170728 |