CN110489475A - A kind of multi-source heterogeneous data processing method, system and relevant apparatus - Google Patents
A kind of multi-source heterogeneous data processing method, system and relevant apparatus Download PDFInfo
- Publication number
- CN110489475A CN110489475A CN201910749404.3A CN201910749404A CN110489475A CN 110489475 A CN110489475 A CN 110489475A CN 201910749404 A CN201910749404 A CN 201910749404A CN 110489475 A CN110489475 A CN 110489475A
- Authority
- CN
- China
- Prior art keywords
- source heterogeneous
- data
- heterogeneous data
- data processing
- pretreatment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A kind of multi-source heterogeneous data processing method provided herein, comprising: the multi-source heterogeneous data of acquisition target service system end, and establish corresponding index information；Inquiry pretreatment component knowledge base, and determine pretreatment component knowledge base in the highest case of index information similarity；After determining the corresponding pretreatment process of case in pretreatment process library, multi-source heterogeneous data are pre-processed according to pretreatment process accordingly, and pretreated multi-source heterogeneous data are sent to terminal and carry out data processing.This method inquire and determine pretreatment component knowledge base in the highest case of index information similarity；After determining the corresponding pretreatment process of case, multi-source heterogeneous data are pre-processed according to pretreatment process accordingly, can be improved the treatment effeciency of multi-source heterogeneous data, and then improve user experience.The application also provides a kind of multi-source heterogeneous data processing system, server cluster and computer readable storage medium, all has above-mentioned beneficial effect.
Description
Technical field
This application involves multi-source heterogeneous data processing field, in particular to a kind of multi-source heterogeneous data processing method, system,
Server cluster and computer readable storage medium.
Background technique
User be king instantly, user experience is that living in peace for many enterprises is basic.Promote the pass of user experience
One of key point videlicet when, solve the problems, such as user at any time, more and more enterprises increase " client in service software thus
The functional modules such as service " carry out customer in response demand at any time.Customer service functions module proposes keyword search accurately and in time
Business demand.Since electric power enterprise business processing is complicated, each inside, the framework of external system are different, the multiplicity of data format,
Lead to the inefficiency of multi-source heterogeneous data processing in the related technology, user experience is poor.
Therefore, the treatment effeciency of multi-source heterogeneous data how is improved, and then improving user experience is those skilled in the art
The technical issues of urgent need to resolve.
Summary of the invention
The purpose of the application is to provide a kind of multi-source heterogeneous data processing method, system, server cluster and computer can
Storage medium is read, can be improved the treatment effeciency of multi-source heterogeneous data, and then improve user experience.
In order to solve the above technical problems, the application provides a kind of multi-source heterogeneous data processing method, comprising:
The multi-source heterogeneous data of target service system end are acquired, and establish corresponding index information；
Inquiry pretreatment component knowledge base, and determine in the pretreatment component knowledge base with the index information similarity
Highest case；
After determining the corresponding pretreatment process of the case in pretreatment process library, according to the pretreatment process to institute
It states multi-source heterogeneous data to be pre-processed accordingly, and pretreated multi-source heterogeneous data is sent to terminal and are carried out at data
Reason.
It is preferably, described that pretreated multi-source heterogeneous data are sent to after terminal progress data processing, further includes:
Multi-source heterogeneous data after data processing are sent to server cluster, and according to the multi-source after the data processing
The data characteristics of isomeric data carries out relationship type or the storage of non-relational data；
GirdFS data in MongoDB are synchronized to SOLR engine.
Preferably, the GirdFS data by MongoDB are synchronized to SOLR engine, comprising:
After the synchronous regime for judging the GirdFS data and the SOLR engine, by not synchronous GirdFS data into
Row GirdFS reading data and the GirdFS data for being converted to string format；
The GirdFS data of the string format are synchronized to the SOLR engine, and update the synchronous regime.
Preferably, the multi-source heterogeneous data of the acquisition target service system end, and corresponding index information is established, it wraps
It includes:
After acquiring the multi-source heterogeneous data, data type, the data volume size of the multi-source heterogeneous data are carried out
Label obtains mark information, and using the mark information as the index information.
Preferably, the inquiry pre-processes component knowledge base, and determine in the pretreatment component knowledge base with the rope
Fuse ceases the highest case of similarity, comprising:
The pretreatment component knowledge base is inquired, and is determined according to the data type, the data volume size described pre-
Handle component knowledge base in the highest case of the index information similarity.
Preferably, if being matched to the case that the similarity is lower than preset threshold in the pretreatment component knowledge base,
Then the multi-source heterogeneous data are handled according to unknown structure mode, and corresponding index information, pretreatment process are divided
It does not store to the pretreatment component knowledge base, the pretreatment process library.
Preferably, the multi-source heterogeneous data processing method further include:
The pretreatment component knowledge base, the pretreatment process library are executed and update operation.
The application also provides a kind of multi-source heterogeneous data processing system, comprising:
Index information establishes module, for acquiring the multi-source heterogeneous data of target service system end, and establishes corresponding rope
Fuse breath；
Case determining module, for inquiring pretreatment component knowledge base, and determine in the pretreatment component knowledge base and
The highest case of index information similarity；
Preprocessing module, after determining the corresponding pretreatment process of the case in pretreatment process library, according to institute
It states pretreatment process to pre-process the multi-source heterogeneous data accordingly, and pretreated multi-source heterogeneous data is sent
Data processing is carried out to terminal.
The application also provides a kind of server cluster, comprising:
Memory and processor；Wherein, the memory is for storing computer program, and the processor is for executing institute
The step of multi-source heterogeneous data processing method described above is realized when stating computer program.
The application also provides a kind of computer readable storage medium, and the computer-readable recording medium storage has computer
The step of program, the computer program realizes multi-source heterogeneous data processing method described above when being executed by processor.
A kind of multi-source heterogeneous data processing method provided herein, comprising: the multi-source of acquisition target service system end
Isomeric data, and establish corresponding index information；Inquiry pretreatment component knowledge base, and determine the pretreatment component knowledge base
In with the highest case of index information similarity；The corresponding pretreatment process of the case is determined in pretreatment process library
Afterwards, the multi-source heterogeneous data are pre-processed accordingly according to the pretreatment process, and pretreated multi-source is different
Structure data are sent to terminal and carry out data processing.
This method inquiry pretreatment component knowledge base, and determine in the pretreatment component knowledge base with the index information
The highest case of similarity；After determining the corresponding pretreatment process of the case in pretreatment process library, according to the pre- place
Reason process pre-processes the multi-source heterogeneous data accordingly, can be improved the treatment effeciency of multi-source heterogeneous data, in turn
Improve user experience.The application also provides a kind of multi-source heterogeneous data processing system, server cluster and computer-readable storage
Medium all has above-mentioned beneficial effect, and details are not described herein.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of application for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of flow chart of multi-source heterogeneous data processing method provided by the embodiment of the present application；
Fig. 2 is a kind of structural block diagram of multi-source heterogeneous data processing system provided by the embodiment of the present application.
Specific embodiment
The core of the application is to provide a kind of multi-source heterogeneous data processing method, can be improved the processing of multi-source heterogeneous data
Efficiency, and then improve user experience.Another core of the application is to provide a kind of multi-source heterogeneous data processing system, server set
Group and computer readable storage medium.
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is
Some embodiments of the present application, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall in the protection scope of this application.
User be king instantly, user experience is that living in peace for many enterprises is basic.Promote the pass of user experience
One of key point videlicet when, solve the problems, such as user at any time, more and more enterprises increase " client in service software thus
The functional modules such as service " carry out customer in response demand at any time.Customer service functions module proposes keyword search accurately and in time
Business demand.Since electric power enterprise business processing is complicated, each inside, the framework of external system are different, the multiplicity of data format,
Lead to the inefficiency of multi-source heterogeneous data processing in the related technology, user experience is poor.A kind of multi-source provided by the present application is different
Structure data processing method can be improved the treatment effeciency of multi-source heterogeneous data, and then improve user experience.It is specific referring to FIG. 1,
Fig. 1 is a kind of flow chart of multi-source heterogeneous data processing method provided by the embodiment of the present application, the multi-source heterogeneous data processing
Method specifically includes:
S101, the multi-source heterogeneous data for acquiring target service system end, and establish corresponding index information；
The embodiment of the present application is not especially limited the acquisition equipment of multi-source heterogeneous data and corresponding acquisition mode,
Corresponding setting should be made according to the actual situation by those skilled in the art.Believe herein for establishing the index of multi-source heterogeneous data
Breath is also not especially limited, as long as meet demand.Above-mentioned target service system end is a kind of operation system end, is specifically needed
Depending on according to the actual situation.
Further, the multi-source heterogeneous data of above-mentioned acquisition target service system end, and corresponding index information is established, lead to
It often include: that data type, the data volume size of multi-source heterogeneous data are marked and are marked after acquiring multi-source heterogeneous data
Remember information, and using mark information as index information.
S102, inquiry pretreatment component knowledge base, and determine pretreatment component knowledge base in index information similarity most
High case；
Pretreatment component knowledge base is not especially limited herein, should be made according to the actual situation by those skilled in the art
It is corresponding out to set, the corresponding case of various index informations is stored in the pretreatment component knowledge base.In the embodiment of the present application really
The standard of the standing wire fuse breath highest case of similarity is that numerical bias summation is minimum.
Further, above-mentioned inquiry pre-processes component knowledge base, and determine in pretreatment component knowledge base with index information
The highest case of similarity, generally includes: inquiry pretreatment component knowledge base, and determines according to data type, data volume size
Pre-process component knowledge base in the highest case of index information similarity.
In one embodiment, if being matched to the case that similarity is lower than preset threshold in pretreatment component knowledge base,
Then multi-source heterogeneous data are handled according to unknown structure mode, and corresponding index information, pretreatment process are deposited respectively
Storage extremely pretreatment component knowledge base, pretreatment process library.Preset threshold is not especially limited herein, it should be by art technology
Personnel make corresponding setting according to the actual situation.
S103, after determining the corresponding pretreatment process of case in pretreatment process library, according to pretreatment process to multi-source
Isomeric data is pre-processed accordingly, and pretreated multi-source heterogeneous data are sent to terminal and carry out data processing.
Pretreatment process library is not especially limited herein, phase should be made according to the actual situation by those skilled in the art
The setting answered stores the corresponding pretreatment process of various cases in the pretreatment process library, and then can be according to pretreated stream
Journey pre-processes multi-source heterogeneous data accordingly, and pretreated multi-source heterogeneous data are sent to terminal and carry out data
Processing.Preferably, entire Data processing uses micro services mode, and extending transversely according to business load progress.
Further, it is above-mentioned by pretreated multi-source heterogeneous data be sent to terminal carry out data processing after, usually
It can also include: that the multi-source heterogeneous data after data processing are sent to server cluster, and according to the multi-source after data processing
The data characteristics of isomeric data carries out relationship type or the storage of non-relational data；GirdFS data in MongoDB are synchronized to
SOLR engine.
SOLR engine is a high-performance, is developed using Java5, the full-text search server based on Lucene, is the world
The upper search for being permitted great internet sites and navigation feature provide support, with high reliability, scalability and fault-tolerance,
It can provide distributed index, duplication and load balancing inquiry, the advantages such as automatic fault transfer and recovery, centralized configuration.MongoDB
It is the PostgreSQL database system based on distributed document storage write by C Plus Plus, supports loose data structure,
Complicated data type can be stored, supports powerful query language, the exhausted big of similarity relation database list table inquiry may be implemented
Partial function, and support to index data foundation, with the features such as high-performance, easily deployment, easily use, storing data facilitates.
Further, the above-mentioned GirdFS data by MongoDB are synchronized to SOLR engine, generally include: judging
After GirdFS data and the synchronous regime of SOLR engine, not synchronous GirdFS data are subjected to GirdFS reading data and are converted
For the GirdFS data of string format；The GirdFS data of string format are synchronized to SOLR engine, and update synchronous shape
State.Key search is carried out in such a way that MongoDB is in conjunction with SOLR, realizes the GirdFS of MongoDB under java environment
Data are synchronous with the automation of SOLR, improve retrieval accuracy, and the user experience is improved spends.The embodiment of the present application can be effective
The time for handling multi-source heterogeneous data is reduced, and realizes MongoDB and automatic synchronization of the SOLR engine under big file, is realized
The beneficial effect quickly handled at the terminal, greatly improves efficiency.
Further, which usually can also include: to pretreatment component knowledge base, pre- place
It manages process library and executes and update operation.
The deployment framework designed in entire method is data collection terminal-data prediction end-data receiver, data acquisition
End is the mass data acquisition middleware that operation system end is mounted on using heart pattern；Data prediction end refers to and is distributed
The server cluster of the high-volume pretreatment business datum of formula deployment；Data receiver is data after the completion of storage pretreatment, and
Carry out the server cluster of database and search engine Timing Synchronization.The scene that data exchange occurs is that data collection terminal is located to pre-
Data receiver is arrived at reason end again, can prejudge out the type of pending data to be processed, structure by pre-processing link,
Processing method etc. can be handled with reference to knowledge base (mode of past experience and historical accumulation).The application is implemented
Traditional data tupe is changed to transmitting terminal to pretreatment again to receiving end by transmitting terminal to receiving end in example, can effectively be added
Fast data-handling efficiency, the beneficial effect that realization is quickly handled at the terminal, and MongoDB and SOLR search engine is combined,
Power industry retrieval dictionary is established, realizes the retrieval of power industry Chinese word segmentation, and due to expanding in treatment process using dynamic
The mode of exhibition can effectively solve processing bottleneck problem.
It below can to a kind of multi-source heterogeneous data processing system provided by the embodiments of the present application, server cluster and computer
It reads storage medium to be introduced, multi-source heterogeneous data processing system, server cluster and computer-readable storage described below
Medium can correspond to each other reference with above-described multi-source heterogeneous data processing method.
Referring to FIG. 2, Fig. 2 is a kind of structural frames of multi-source heterogeneous data processing system provided by the embodiment of the present application
Figure；The multi-source heterogeneous data processing system includes:
Index information establishes module 201, for acquiring the multi-source heterogeneous data of target service system end, and establishes corresponding
Index information；
Case determining module 202, for inquire pretreatment component knowledge base, and determine pretreatment component knowledge base in rope
Fuse ceases the highest case of similarity；
Preprocessing module 203, after determining the corresponding pretreatment process of case in pretreatment process library, according to pre- place
Reason process pre-processes multi-source heterogeneous data accordingly, and pretreated multi-source heterogeneous data are sent to terminal and are carried out
Data processing.
Based on the above embodiment, the multi-source heterogeneous data processing system of this in the present embodiment usually can also include:
Data memory module, for the multi-source heterogeneous data after data processing to be sent to server cluster, and according to number
According to treated, the data characteristics of multi-source heterogeneous data carries out relationship type or the storage of non-relational data；
Data simultaneous module, for the GirdFS data in MongoDB to be synchronized to SOLR engine.
Based on the above embodiment, data simultaneous module in the present embodiment, generally includes:
Format conversion unit will be not synchronous for after the synchronous regime for judging GirdFS data and SOLR engine
GirdFS data carry out GirdFS reading data and are converted to the GirdFS data of string format；
Data synchronisation unit for the GirdFS data of string format to be synchronized to SOLR engine, and updates synchronous shape
State.
Based on the above embodiment, index information establishes module 201 in the present embodiment, generally includes:
Index information establishes unit, is used for after acquiring multi-source heterogeneous data, data type, number to multi-source heterogeneous data
It is marked to obtain mark information according to amount size, and using mark information as index information.
Based on the above embodiment, case determining module 202 in the present embodiment, generally includes:
Case determination unit determines in advance for inquiring pretreatment component knowledge base, and according to data type, data volume size
Handle component knowledge base in the highest case of index information similarity.
Based on the above embodiment, the multi-source heterogeneous data processing system of this in the present embodiment usually can also include:
Operation executing module is updated, updates operation for executing to pretreatment component knowledge base, pretreatment process library.
The application also provides a kind of server cluster, comprising: memory and processor；Wherein, memory is based on storing
Calculation machine program, processor are used to realize the multi-source heterogeneous data processing method of above-mentioned any embodiment when executing computer program
Step.
The application also provides a kind of computer readable storage medium, and computer-readable recording medium storage has computer journey
Sequence, the step of multi-source heterogeneous data processing method of above-mentioned any embodiment is realized when computer program is executed by processor.
The computer readable storage medium may include: USB flash disk, mobile hard disk, read-only memory (Read-Only
Memory, ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. is various to deposit
Store up the medium of program code.
Each embodiment is described in a progressive manner in specification, the highlights of each of the examples are with other realities
The difference of example is applied, the same or similar parts in each embodiment may refer to each other.For embodiment provide system and
Speech, since it is corresponding with the method that embodiment provides, so being described relatively simple, related place is referring to method part illustration
.
Professional further appreciates that, unit described in conjunction with the examples disclosed in the embodiments of the present disclosure
And algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These
Function is implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Profession
Technical staff can use different methods to achieve the described function each specific application, but this realization is not answered
Think beyond the scope of this invention.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
Above to a kind of multi-source heterogeneous data processing method, system, server cluster and computer provided herein
Readable storage medium storing program for executing is described in detail.Specific case used herein carries out the principle and embodiment of the application
It illustrates, the description of the example is only used to help understand the method for the present application and its core ideas.It should be pointed out that for this
For the those of ordinary skill of technical field, under the premise of not departing from the application principle, the application can also be carried out several
Improvement and modification, these improvement and modification are also fallen into the protection scope of the claim of this application.
Claims (10)
1. a kind of multi-source heterogeneous data processing method characterized by comprising
The multi-source heterogeneous data of target service system end are acquired, and establish corresponding index information；
Inquiry pretreatment component knowledge base, and determine in the pretreatment component knowledge base with the index information similarity highest
Case；
After determining the corresponding pretreatment process of the case in pretreatment process library, according to the pretreatment process to described more
Source isomeric data is pre-processed accordingly, and pretreated multi-source heterogeneous data are sent to terminal and carry out data processing.
2. multi-source heterogeneous data processing method according to claim 1, which is characterized in that described by pretreated multi-source
Isomeric data is sent to after terminal progress data processing, further includes:
Multi-source heterogeneous data after data processing are sent to server cluster, and according to multi-source heterogeneous after the data processing
The data characteristics of data carries out relationship type or the storage of non-relational data；
GirdFS data in MongoDB are synchronized to SOLR engine.
3. multi-source heterogeneous data processing method according to claim 2, which is characterized in that it is described will be in MongoDB
GirdFS data are synchronized to SOLR engine, comprising:
After judging the synchronous regime of the GirdFS data and the SOLR engine, not synchronous GirdFS data are carried out
GirdFS reading data and the GirdFS data for being converted to string format；
The GirdFS data of the string format are synchronized to the SOLR engine, and update the synchronous regime.
4. multi-source heterogeneous data processing method according to claim 1, which is characterized in that the acquisition target service system
The multi-source heterogeneous data at end, and establish corresponding index information, comprising:
After acquiring the multi-source heterogeneous data, data type, the data volume size of the multi-source heterogeneous data are marked
Mark information is obtained, and using the mark information as the index information.
5. multi-source heterogeneous data processing method according to claim 4, which is characterized in that the inquiry pretreatment component is known
Know library, and determine in the pretreatment component knowledge base with the highest case of index information similarity, comprising:
The pretreatment component knowledge base is inquired, and determines the pretreatment according to the data type, the data volume size
In component knowledge base with the highest case of the index information similarity.
6. multi-source heterogeneous data processing method according to claim 1, which is characterized in that if knowing in the pretreatment component
Know and be matched to the case that the similarity is lower than preset threshold in library, then according to unknown structure mode to the multi-source heterogeneous data
It is handled, and corresponding index information, pretreatment process is stored respectively to the pretreatment component knowledge base, the pre- place
Manage process library.
7. multi-source heterogeneous data processing method according to any one of claims 1 to 6, which is characterized in that further include:
The pretreatment component knowledge base, the pretreatment process library are executed and update operation.
8. a kind of multi-source heterogeneous data processing system characterized by comprising
Index information establishes module, for acquiring the multi-source heterogeneous data of target service system end, and establishes corresponding index letter
Breath；
Case determining module, for inquiring pretreatment component knowledge base, and determine in the pretreatment component knowledge base with it is described
The highest case of index information similarity；
Preprocessing module, after determining the corresponding pretreatment process of the case in pretreatment process library, according to described pre-
Process flow pre-processes the multi-source heterogeneous data accordingly, and pretreated multi-source heterogeneous data are sent to end
End carries out data processing.
9. a kind of server cluster characterized by comprising
Memory and processor；Wherein, the memory is for storing computer program, the processor by execute it is described based on
The step of realizing multi-source heterogeneous data processing method as described in any one of claim 1 to 7 when calculation machine program.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer journey
Sequence, the computer program realize multi-source heterogeneous data processing as described in any one of claim 1 to 7 when being executed by processor
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910749404.3A CN110489475B (en) | 2019-08-14 | 2019-08-14 | Multi-source heterogeneous data processing method, system and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910749404.3A CN110489475B (en) | 2019-08-14 | 2019-08-14 | Multi-source heterogeneous data processing method, system and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110489475A true CN110489475A (en) | 2019-11-22 |
| CN110489475B CN110489475B (en) | 2021-01-26 |
Family
ID=68550984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910749404.3A Active CN110489475B (en) | 2019-08-14 | 2019-08-14 | Multi-source heterogeneous data processing method, system and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110489475B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110990390A (en) * | 2019-12-02 | 2020-04-10 | 东莞中国科学院云计算产业技术创新与育成中心 | Data cooperative processing method and device, computer equipment and storage medium |
| CN111431967A (en) * | 2020-02-25 | 2020-07-17 | 天宇经纬（北京）科技有限公司 | Multi-source heterogeneous data representation and distribution method and device based on business rules |
| CN112270600A (en) * | 2020-10-29 | 2021-01-26 | 广东通莞科技股份有限公司 | Multi-source data processing method, system and related device |
| CN112883096A (en) * | 2021-03-11 | 2021-06-01 | 广东工业大学 | Data preprocessing method |
| CN113111503A (en) * | 2021-04-01 | 2021-07-13 | 重庆传晟酷德大数据科技有限公司 | Multi-source heterogeneous data construction method based on CAD |
| CN117195054A (en) * | 2023-09-15 | 2023-12-08 | 苏州优鲜生网络科技有限公司 | Cross-node data identification method and system based on clusters |
Citations (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070055011A1 (en) * | 2003-05-16 | 2007-03-08 | Basf Aktiengesellschaft | Method for producing aqueous polymer dispersions |
| CN101079034A (en) * | 2006-07-10 | 2007-11-28 | 腾讯科技（深圳）有限公司 | System and method for eliminating redundancy file of file storage system |
| CN101089843A (en) * | 2006-06-15 | 2007-12-19 | 王刘忠 | Search method only for product or service supply information |
| CN101094173A (en) * | 2007-06-28 | 2007-12-26 | 上海交通大学 | Integrated system of data interchange under distributed isomerical environment |
| CN101136038A (en) * | 2007-10-18 | 2008-03-05 | 中国兵器工业第五九研究所 | Plasticity forming numerical modeling method |
| CN101202902A (en) * | 2007-12-11 | 2008-06-18 | 西安交通大学 | Method for designing P2P stream medium network transferring structure with number copyright management |
| CN101286161A (en) * | 2008-05-28 | 2008-10-15 | 华中科技大学 | Intelligent Chinese request-answering system based on concept |
| CN101387582A (en) * | 2008-10-24 | 2009-03-18 | 西北工业大学 | Failure diagnosis system and method based on PDA |
| CN101534021A (en) * | 2009-04-27 | 2009-09-16 | 北京四方继保自动化股份有限公司 | Multimode data acquisitions and processing method applied to power automation system |
| CN101742228A (en) * | 2008-11-19 | 2010-06-16 | 新奥特硅谷视频技术有限责任公司 | Preprocessing method and system applied to digital court |
| CN101853291A (en) * | 2010-05-24 | 2010-10-06 | 合肥工业大学 | Data flow based car fault diagnosis method |
| CN102073646A (en) * | 2009-11-23 | 2011-05-25 | 北京科技大学 | Blog group-oriented subject propensity processing method and system |
| CN102098799A (en) * | 2011-01-26 | 2011-06-15 | 北京邮电大学 | Intelligent cognitive wireless network system for realizing heterogeneous network convergence |
| CN102254030A (en) * | 2011-08-02 | 2011-11-23 | 中国科学院计算机网络信息中心 | Global change research-oriented automatic space science data gathering method |
| CN102495892A (en) * | 2011-12-09 | 2012-06-13 | 北京大学 | Webpage information extraction method |
| CN102609512A (en) * | 2012-02-07 | 2012-07-25 | 北京中机科海科技发展有限公司 | System and method for heterogeneous information mining and visual analysis |
| CN102765643A (en) * | 2012-05-31 | 2012-11-07 | 天津大学 | Elevator fault diagnosis and early-warning method based on data drive |
| CN102855600A (en) * | 2012-07-23 | 2013-01-02 | 电子科技大学 | Selective recommendation method for isomerism ability of mobile internet |
| CN104679902A (en) * | 2015-03-20 | 2015-06-03 | 湘潭大学 | Information abstract extraction method in conjunction with cross-media fuse |
| US20150378567A1 (en) * | 2010-12-17 | 2015-12-31 | Microsoft Technology Licensing, Llc | Data Feed Having Customizable Analytic and Visual Behavior |
| CN105760495A (en) * | 2016-02-17 | 2016-07-13 | 扬州大学 | Method for carrying out exploratory search for bug problem based on knowledge map |
| CN106372079A (en) * | 2015-07-22 | 2017-02-01 | 中国化工信息中心 | Patent information processing and retrieval method |
| CN106528786A (en) * | 2016-11-08 | 2017-03-22 | 国网山东省电力公司电力科学研究院 | Method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase |
| CN106611053A (en) * | 2016-12-26 | 2017-05-03 | 河南信安通信技术股份有限公司 | Data cleaning and indexing method |
| CN106611046A (en) * | 2016-12-16 | 2017-05-03 | 武汉中地数码科技有限公司 | Big data technology-based space data storage processing middleware framework |
| CN106980618A (en) * | 2016-01-15 | 2017-07-25 | 航天信息股份有限公司 | File memory method and system based on MongoDB distributed type assemblies frameworks |
| CN107330125A (en) * | 2017-07-20 | 2017-11-07 | 云南电网有限责任公司电力科学研究院 | The unstructured distribution data integrated approach of magnanimity of knowledge based graphical spectrum technology |
| CN107609154A (en) * | 2017-09-23 | 2018-01-19 | 浪潮软件集团有限公司 | Method and device for processing multi-source heterogeneous data |
| CN108121828A (en) * | 2018-01-17 | 2018-06-05 | 清华大学 | A kind of multi-source heterogeneous data managing method and system based on key-value pair data storehouse |
| CN108446363A (en) * | 2018-03-13 | 2018-08-24 | 北京奇安信科技有限公司 | A kind of data processing method and device of KV engines |
| CN109033387A (en) * | 2018-07-26 | 2018-12-18 | 广州大学 | A kind of Internet of Things search system, method and storage medium merging multi-source data |
| CN109063063A (en) * | 2018-07-20 | 2018-12-21 | 泰华智慧产业集团股份有限公司 | Data processing method and device based on multi-source data |
| CN109165202A (en) * | 2018-07-04 | 2019-01-08 | 华南理工大学 | A kind of preprocess method of multi-source heterogeneous big data |
| US10176259B1 (en) * | 2009-05-15 | 2019-01-08 | Donald Newton Cohen | Use of virtual database technology for internet search and data integration |
| CN109471884A (en) * | 2018-09-12 | 2019-03-15 | 国网浙江省电力有限公司嘉兴供电公司 | The relevant multi-source heterogeneous data processing method of distributed new |
| US10289629B1 (en) * | 2016-06-22 | 2019-05-14 | Amazon Technologies, Inc. | Techniques for interruption-free partitioning |
| CN109918472A (en) * | 2019-02-27 | 2019-06-21 | 北京百度网讯科技有限公司 | Method, apparatus, equipment and the medium of storage and inquiry data |
| CN110069495A (en) * | 2019-03-13 | 2019-07-30 | 中科恒运股份有限公司 | Date storage method, device and terminal device |
-
2019
- 2019-08-14 CN CN201910749404.3A patent/CN110489475B/en active Active
Patent Citations (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070055011A1 (en) * | 2003-05-16 | 2007-03-08 | Basf Aktiengesellschaft | Method for producing aqueous polymer dispersions |
| CN101089843A (en) * | 2006-06-15 | 2007-12-19 | 王刘忠 | Search method only for product or service supply information |
| CN101079034A (en) * | 2006-07-10 | 2007-11-28 | 腾讯科技（深圳）有限公司 | System and method for eliminating redundancy file of file storage system |
| CN101094173A (en) * | 2007-06-28 | 2007-12-26 | 上海交通大学 | Integrated system of data interchange under distributed isomerical environment |
| CN101136038A (en) * | 2007-10-18 | 2008-03-05 | 中国兵器工业第五九研究所 | Plasticity forming numerical modeling method |
| CN101202902A (en) * | 2007-12-11 | 2008-06-18 | 西安交通大学 | Method for designing P2P stream medium network transferring structure with number copyright management |
| CN101286161A (en) * | 2008-05-28 | 2008-10-15 | 华中科技大学 | Intelligent Chinese request-answering system based on concept |
| CN101387582A (en) * | 2008-10-24 | 2009-03-18 | 西北工业大学 | Failure diagnosis system and method based on PDA |
| CN101742228A (en) * | 2008-11-19 | 2010-06-16 | 新奥特硅谷视频技术有限责任公司 | Preprocessing method and system applied to digital court |
| CN101534021A (en) * | 2009-04-27 | 2009-09-16 | 北京四方继保自动化股份有限公司 | Multimode data acquisitions and processing method applied to power automation system |
| US10176259B1 (en) * | 2009-05-15 | 2019-01-08 | Donald Newton Cohen | Use of virtual database technology for internet search and data integration |
| CN102073646A (en) * | 2009-11-23 | 2011-05-25 | 北京科技大学 | Blog group-oriented subject propensity processing method and system |
| CN101853291A (en) * | 2010-05-24 | 2010-10-06 | 合肥工业大学 | Data flow based car fault diagnosis method |
| US20150378567A1 (en) * | 2010-12-17 | 2015-12-31 | Microsoft Technology Licensing, Llc | Data Feed Having Customizable Analytic and Visual Behavior |
| CN102098799A (en) * | 2011-01-26 | 2011-06-15 | 北京邮电大学 | Intelligent cognitive wireless network system for realizing heterogeneous network convergence |
| CN102254030A (en) * | 2011-08-02 | 2011-11-23 | 中国科学院计算机网络信息中心 | Global change research-oriented automatic space science data gathering method |
| CN102495892A (en) * | 2011-12-09 | 2012-06-13 | 北京大学 | Webpage information extraction method |
| CN102609512A (en) * | 2012-02-07 | 2012-07-25 | 北京中机科海科技发展有限公司 | System and method for heterogeneous information mining and visual analysis |
| CN102765643A (en) * | 2012-05-31 | 2012-11-07 | 天津大学 | Elevator fault diagnosis and early-warning method based on data drive |
| CN102855600A (en) * | 2012-07-23 | 2013-01-02 | 电子科技大学 | Selective recommendation method for isomerism ability of mobile internet |
| CN104679902A (en) * | 2015-03-20 | 2015-06-03 | 湘潭大学 | Information abstract extraction method in conjunction with cross-media fuse |
| CN106372079A (en) * | 2015-07-22 | 2017-02-01 | 中国化工信息中心 | Patent information processing and retrieval method |
| CN106980618A (en) * | 2016-01-15 | 2017-07-25 | 航天信息股份有限公司 | File memory method and system based on MongoDB distributed type assemblies frameworks |
| CN105760495A (en) * | 2016-02-17 | 2016-07-13 | 扬州大学 | Method for carrying out exploratory search for bug problem based on knowledge map |
| US10289629B1 (en) * | 2016-06-22 | 2019-05-14 | Amazon Technologies, Inc. | Techniques for interruption-free partitioning |
| CN106528786A (en) * | 2016-11-08 | 2017-03-22 | 国网山东省电力公司电力科学研究院 | Method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase |
| CN106611046A (en) * | 2016-12-16 | 2017-05-03 | 武汉中地数码科技有限公司 | Big data technology-based space data storage processing middleware framework |
| CN106611053A (en) * | 2016-12-26 | 2017-05-03 | 河南信安通信技术股份有限公司 | Data cleaning and indexing method |
| CN107330125A (en) * | 2017-07-20 | 2017-11-07 | 云南电网有限责任公司电力科学研究院 | The unstructured distribution data integrated approach of magnanimity of knowledge based graphical spectrum technology |
| CN107609154A (en) * | 2017-09-23 | 2018-01-19 | 浪潮软件集团有限公司 | Method and device for processing multi-source heterogeneous data |
| CN108121828A (en) * | 2018-01-17 | 2018-06-05 | 清华大学 | A kind of multi-source heterogeneous data managing method and system based on key-value pair data storehouse |
| CN108446363A (en) * | 2018-03-13 | 2018-08-24 | 北京奇安信科技有限公司 | A kind of data processing method and device of KV engines |
| CN109165202A (en) * | 2018-07-04 | 2019-01-08 | 华南理工大学 | A kind of preprocess method of multi-source heterogeneous big data |
| CN109063063A (en) * | 2018-07-20 | 2018-12-21 | 泰华智慧产业集团股份有限公司 | Data processing method and device based on multi-source data |
| CN109033387A (en) * | 2018-07-26 | 2018-12-18 | 广州大学 | A kind of Internet of Things search system, method and storage medium merging multi-source data |
| CN109471884A (en) * | 2018-09-12 | 2019-03-15 | 国网浙江省电力有限公司嘉兴供电公司 | The relevant multi-source heterogeneous data processing method of distributed new |
| CN109918472A (en) * | 2019-02-27 | 2019-06-21 | 北京百度网讯科技有限公司 | Method, apparatus, equipment and the medium of storage and inquiry data |
| CN110069495A (en) * | 2019-03-13 | 2019-07-30 | 中科恒运股份有限公司 | Date storage method, device and terminal device |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110990390A (en) * | 2019-12-02 | 2020-04-10 | 东莞中国科学院云计算产业技术创新与育成中心 | Data cooperative processing method and device, computer equipment and storage medium |
| CN110990390B (en) * | 2019-12-02 | 2024-03-08 | 东莞中国科学院云计算产业技术创新与育成中心 | Data cooperative processing method, device, computer equipment and storage medium |
| CN111431967A (en) * | 2020-02-25 | 2020-07-17 | 天宇经纬（北京）科技有限公司 | Multi-source heterogeneous data representation and distribution method and device based on business rules |
| CN112270600A (en) * | 2020-10-29 | 2021-01-26 | 广东通莞科技股份有限公司 | Multi-source data processing method, system and related device |
| CN112883096A (en) * | 2021-03-11 | 2021-06-01 | 广东工业大学 | Data preprocessing method |
| CN112883096B (en) * | 2021-03-11 | 2024-04-30 | 广东工业大学 | Data preprocessing method |
| CN113111503A (en) * | 2021-04-01 | 2021-07-13 | 重庆传晟酷德大数据科技有限公司 | Multi-source heterogeneous data construction method based on CAD |
| CN113111503B (en) * | 2021-04-01 | 2024-03-05 | 重庆传晟酷德大数据科技有限公司 | CAD-based multi-source heterogeneous data construction method |
| CN117195054A (en) * | 2023-09-15 | 2023-12-08 | 苏州优鲜生网络科技有限公司 | Cross-node data identification method and system based on clusters |
| CN117195054B (en) * | 2023-09-15 | 2024-03-26 | 苏州优鲜生网络科技有限公司 | Cross-node data identification method and system based on clusters |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110489475B (en) | 2021-01-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489475A (en) | A kind of multi-source heterogeneous data processing method, system and relevant apparatus | |
| CN108694195B (en) | Management method and system of distributed data warehouse | |
| CN109951323B (en) | Log analysis method and system | |
| CN106909595B (en) | Data migration method and device | |
| CN111475583B (en) | Transaction processing method and device | |
| CN113836925B (en) | Training method and device for pre-training language model, electronic equipment and storage medium | |
| CN106777142A (en) | Service layer's system and method based on mobile Internet mass data | |
| CN111597267A (en) | Data middlebox based on multilayer service engine and construction method | |
| CN111046041A (en) | Data processing method and device, storage medium and processor | |
| CN104267974B (en) | The call method and device of business interface | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN105446981B (en) | Map of website generation method, access method and device | |
| CN107968798B (en) | Network management resource label obtaining method, cache synchronization method, device and system | |
| CN109145092A (en) | A kind of database update, intelligent answer management method, device and its equipment | |
| CN117056565A (en) | Power information processing method, device, equipment and medium based on RPA and AI | |
| CN110895538A (en) | Data retrieval method, device, storage medium and processor | |
| CN115132186A (en) | End-to-end speech recognition model training method, speech decoding method and related device | |
| EP3046307B1 (en) | Processing method, device and system for data of distributed storage system | |
| CN110046132B (en) | Metadata request processing method, device, equipment and readable storage medium | |
| CN109616156B (en) | Gene sequencing data storage method and device | |
| CN113297218A (en) | Multi-system data interaction method, device and system | |
| CN113360558A (en) | Data processing method, data processing device, electronic device, and storage medium | |
| CN107995301B (en) | Rapid data receiving and transmitting method based on Internet | |
| CN106980621A (en) | The method and apparatus of event filing and inquiry based on MongoDB | |
| CN106021307B (en) | A kind of system positioned for electronic document, unit and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |