CN109408598A - A kind of mass data processing system for multimedia research and development training platform - Google Patents
A kind of mass data processing system for multimedia research and development training platform Download PDFInfo
- Publication number
- CN109408598A CN109408598A CN201811073169.4A CN201811073169A CN109408598A CN 109408598 A CN109408598 A CN 109408598A CN 201811073169 A CN201811073169 A CN 201811073169A CN 109408598 A CN109408598 A CN 109408598A
- Authority
- CN
- China
- Prior art keywords
- data
- platform
- data processing
- data warehouse
- processing system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 28
- 238000012827 research and development Methods 0.000 title claims abstract description 17
- 238000012549 training Methods 0.000 title claims abstract description 17
- 230000001747 exhibiting effect Effects 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 2
- 238000000034 method Methods 0.000 description 4
- 230000004075 alteration Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Abstract
The invention discloses a kind of mass data processing systems for multimedia research and development training platform, comprising: scheduler module, judge whether to call data warehouse operations sentence for the current business information and preset scheduling strategy according to acquisition, when the judgment is yes, calling sequence is obtained according to the current business information of acquisition and preset scheduling strategy, data warehouse operations sentence is called to Data Warehouse Platform according to the calling sequence；The Data Warehouse Platform carries out operation to the data that distributed platform stores for the data warehouse operations sentence according to the calling sequence trigger switch, generates result data and store and arrive the distributed platform；The relational database, for storing the corresponding configuration information of the data warehouse operations sentence；Distributed platform, for storing the data and the result data.Invention enhances the flexibilities of mass data processing, have better meet large capacity, high reliability, high stability and have met mass data processing requirement.
Description
Technical field
The present invention relates to multimedia research and development training platform more particularly to a kind of magnanimity for multimedia research and development training platform
Data processing system.
Background technique
Currently, online education industry has become one of the new industry situation NPD projects that Chinese emphasis is made, and online education row
Industry is considered as most potential R&D direction.But existing multimedia platform is difficult to meet large capacity, high reliability, Gao Wen
Qualitative and mass data processing requirement.
Summary of the invention
The technical problem to be solved in the present invention is that in view of the deficiencies of the prior art, a kind of large capacity, highly reliable is provided
Property, high stability and meet mass data processing requirement for multimedia research and development training platform mass data processing system
System.
In order to solve the above technical problems, the present invention adopts the following technical scheme that.
A kind of mass data processing system for multimedia research and development training platform comprising: scheduler module is used for basis
The current business information of acquisition and preset scheduling strategy judge whether calling data warehouse operations sentence, when the judgment is yes,
Calling sequence is obtained according to the current business information of acquisition and preset scheduling strategy, according to the calling sequentially to data warehouse
Platform calls data warehouse operations sentence；The Data Warehouse Platform, for reading the data warehouse from relational database
Corresponding configuration information, the data that distributed platform is stored according to data warehouse operations sentence described in the calling sequence trigger switch
Operation is carried out, result data is generated and stores and arrive the distributed platform；The relational database, for storing the data bins
The corresponding configuration information of library action statement；Distributed platform, for storing the data and the result data.
Preferably, the scheduler module is also used to control the distributed platform and the result data is imported the relationship
Database, control cache module extract common result data according to the preset strategy that shows from the relational database；Institute
State system further include: the cache module: for caching the common result data；Data exhibiting platform is used for from described
It is read in cache module and shows the common result data.
Preferably, the data exhibiting platform is also used to that the number of results is read and showed from the relational database
According to.
Preferably, the system also includes data access platform, for distributed platform transmission data at least once,
When being transmitted every time, data are sent to message interface module and are transmitted message；The message interface module, for receiving
The data are transmitted message；The scheduler module is also used to obtain the number at least once from the message interface module
According to message is transmitted, as the current business information.
Preferably, the data access platform is specifically used for using the communication side Google's message transmission schemes protoBuffer
Formula sends the data to message interface module and is transmitted message.
Mass data processing system disclosed by the invention for multimedia research and development training platform, is researched and developed for multimedia
Scheduler module is added in the mass data processing system of training platform, the module is according to current business information and preset scheduling plan
It slightly determines and calls data warehouse operations sentence and calling sequence, complete data handling procedure under the control of scheduler module, from
And avoid in the mass data processing system of existing platform through console gradually transmitting order to lower levels because by scheduler module into
Row control, can be according to the logic of the business of required realization, the corresponding scheduling strategy of flexible configuration and calling sequence, to increase
The strong flexibility of mass data processing, has better meet large capacity, high reliability, high stability and has met mass data
Processing requirement.
Detailed description of the invention
Fig. 1 is the composition block diagram of mass data processing system of the present invention.
Specific embodiment
The present invention is described in more detail with reference to the accompanying drawings and examples.
The invention discloses a kind of mass data processing systems for multimedia research and development training platform, please refer to Fig. 1,
Include:
Scheduler module judges whether to call data bins for the current business information and preset scheduling strategy according to acquisition
Library action statement obtains calling sequence, root according to the current business information of acquisition and preset scheduling strategy when the judgment is yes
Data warehouse operations sentence is called to Data Warehouse Platform according to the calling sequence；
The Data Warehouse Platform, for reading the corresponding configuration information of the data warehouse, root from relational database
Operation is carried out to the data that distributed platform stores according to data warehouse operations sentence described in the calling sequence trigger switch, generates result
Data are simultaneously stored to the distributed platform；
The relational database, for storing the corresponding configuration information of the data warehouse operations sentence；
Distributed platform, for storing the data and the result data.
Mass data processing system disclosed by the invention for multimedia research and development training platform, is researched and developed for multimedia
Scheduler module is added in the mass data processing system of training platform, the module is according to current business information and preset scheduling plan
It slightly determines and calls data warehouse operations sentence and calling sequence, complete data handling procedure under the control of scheduler module, from
And avoid in the mass data processing system of existing platform through console gradually transmitting order to lower levels because by scheduler module into
Row control, can be according to the logic of the business of required realization, the corresponding scheduling strategy of flexible configuration and calling sequence, to increase
The strong flexibility of mass data processing, has better meet large capacity, high reliability, high stability and has met mass data
Processing requirement.
In the present embodiment, the scheduler module is also used to control the distributed platform will be described in result data importing
Relational database, control cache module extract common number of results according to the preset strategy that shows from the relational database
According to；
The system also includes:
The cache module: for caching the common result data；
Data exhibiting platform, for the common result data to be read and showed from the cache module.
Further, the data exhibiting platform is also used to that the number of results is read and showed from the relational database
According to.
As a preferred method, the system also includes:
Data access platform, for transmitting data at least once to distributed platform, when being transmitted every time, to message
Interface module sends data and is transmitted message；
The message interface module is transmitted message for receiving the data；
The scheduler module, is also used to obtain the data at least once from the message interface module and is transmitted and disappears
Breath, as the current business information.
In the present embodiment, the data access platform is specifically used for logical using Google's message transmission schemes protoBuffer
News mode sends the data to message interface module and is transmitted message.
In a preferred embodiment of the invention, the access of business platform magnanimity includes building mass data differentiated service frame
Structure enables the platform to reach magnanimity traffic load balance state, avoids the service success rate as caused by business network congestion
It is low.Emphasis needs to consider the following aspects in system design:
Firstly, capacity: network layer needs support million grades while online, host, data and storage and request requirement etc.；
Secondly, reliability and stability: need to adapt to any single calculate node may all break down, any research and development
Test in environment is impossible to be sufficient premise, and must adapt to complicated terminal access net and must support frequently
System under alteration is stablized, and user experience is stablized etc.；
In addition, data processing: magnanimity platform needs to face the problem that database is big, database is miscellaneous.
The above is preferred embodiments of the present invention, is not intended to restrict the invention, all in technology model of the invention
Interior done modification, equivalent replacement or improvement etc. are enclosed, should be included in the range of of the invention protect.
Claims (5)
1. a kind of mass data processing system for multimedia research and development training platform characterized by comprising
Scheduler module judges whether to call data warehouse behaviour for the current business information and preset scheduling strategy according to acquisition
Make sentence, when the judgment is yes, calling sequence is obtained according to the current business information of acquisition and preset scheduling strategy, according to institute
It states calling sequence and calls data warehouse operations sentence to Data Warehouse Platform；
The Data Warehouse Platform, for reading the corresponding configuration information of the data warehouse from relational database, according to institute
It states and data warehouse operations sentence described in sequence trigger switch is called to carry out operation to the data that distributed platform stores, generate result data
And it stores and arrives the distributed platform；
The relational database, for storing the corresponding configuration information of the data warehouse operations sentence；
Distributed platform, for storing the data and the result data.
2. the mass data processing system for multimedia research and development training platform as described in claim 1, which is characterized in that institute
It states scheduler module and is also used to control the distributed platform for the result data importing relational database, control caching mould
Root tuber extracts common result data according to the preset strategy that shows from the relational database；
The system also includes:
The cache module: for caching the common result data；
Data exhibiting platform, for the common result data to be read and showed from the cache module.
3. the mass data processing system for multimedia research and development training platform as described in claim 1, which is characterized in that institute
Data exhibiting platform is stated to be also used to that the result data is read and showed from the relational database.
4. the mass data processing system for multimedia research and development training platform as described in claim 1, which is characterized in that institute
State system further include:
Data access platform, for transmitting data at least once to distributed platform, when being transmitted every time, to message interface
Module sends data and is transmitted message；
The message interface module is transmitted message for receiving the data；
The scheduler module, be also used to from the message interface module obtain at least once the data be transmitted message, make
For the current business information.
5. the mass data processing system for multimedia research and development training platform as described in claim 1, which is characterized in that institute
Data access platform is stated to be specifically used for sending out using Google's message transmission schemes protoBuffer communication modes to message interface module
The data are sent to be transmitted message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811073169.4A CN109408598A (en) | 2018-09-14 | 2018-09-14 | A kind of mass data processing system for multimedia research and development training platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811073169.4A CN109408598A (en) | 2018-09-14 | 2018-09-14 | A kind of mass data processing system for multimedia research and development training platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109408598A true CN109408598A (en) | 2019-03-01 |
Family
ID=65464939
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811073169.4A Pending CN109408598A (en) | 2018-09-14 | 2018-09-14 | A kind of mass data processing system for multimedia research and development training platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109408598A (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102214236A (en) * | 2011-06-30 | 2011-10-12 | 北京新媒传信科技有限公司 | Method and system for processing mass data |
-
2018
- 2018-09-14 CN CN201811073169.4A patent/CN109408598A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102214236A (en) * | 2011-06-30 | 2011-10-12 | 北京新媒传信科技有限公司 | Method and system for processing mass data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110769038B (en) | Server scheduling method and device, storage medium and electronic equipment | |
| CN102892089A (en) | Message pushing method, device and system | |
| CN103647726A (en) | Message dispatching method and device thereof | |
| CN107888666A (en) | A kind of cross-region data-storage system and method for data synchronization and device | |
| CN107608627A (en) | A kind of remote data classification storage method, electronic equipment and storage medium | |
| CN103617007A (en) | Method and system for achieving multilevel intelligent storage | |
| CN102982182A (en) | Data storage planning method and device | |
| CN102123511B (en) | Mobile network data transmission method and system as well as mobile terminal | |
| CN110516090A (en) | A kind of object detecting method, device, equipment and computer readable storage medium | |
| CN104135429A (en) | A method and a device for pushing information | |
| CN101827120A (en) | Cluster storage method and system | |
| CN108614847A (en) | A kind of caching method and system of data | |
| CN101562539A (en) | Self-adapting network intrusion detection system | |
| CN103810038A (en) | Method and device for transferring virtual machine storage files in HA cluster | |
| CN107391672A (en) | The reading/writing method of data and the distributed file system of message | |
| CN109189726B (en) | Processing method and device for read-write log | |
| CN102143053B (en) | Method, device and system for transmitting data | |
| CN103077134B (en) | The method and apparatus of multi-pipe data transmission is realized in a kind of embedded system | |
| CN109408598A (en) | A kind of mass data processing system for multimedia research and development training platform | |
| CN102902593B (en) | Agreement distributing and processing system based on caching mechanism | |
| CN103442091A (en) | Data transmission method and device | |
| CN106304228A (en) | A kind of PGW method for handover control, system and server | |
| CN109347746A (en) | A kind of MAC address learning method and device | |
| CN109407988A (en) | Cold data storage system and its storage method | |
| CN105574292B (en) | A method of realizing that any bandwidth of multichannel is given out a contract for a project based on dynamic array |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |