CN105791412A - Big data processing platform network architecture - Google Patents
Big data processing platform network architecture Download PDFInfo
- Publication number
- CN105791412A CN105791412A CN201610199594.2A CN201610199594A CN105791412A CN 105791412 A CN105791412 A CN 105791412A CN 201610199594 A CN201610199594 A CN 201610199594A CN 105791412 A CN105791412 A CN 105791412A
- Authority
- CN
- China
- Prior art keywords
- server
- big data
- data processing
- processing platform
- network architecture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2015—Redundant power supplies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/167—Interprocessor communication using a common memory, e.g. mailbox
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17306—Intercommunication techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- H—ELECTRICITY
- H02—GENERATION; CONVERSION OR DISTRIBUTION OF ELECTRIC POWER
- H02J—CIRCUIT ARRANGEMENTS OR SYSTEMS FOR SUPPLYING OR DISTRIBUTING ELECTRIC POWER; SYSTEMS FOR STORING ELECTRIC ENERGY
- H02J9/00—Circuit arrangements for emergency or stand-by power supply, e.g. for emergency lighting
- H02J9/04—Circuit arrangements for emergency or stand-by power supply, e.g. for emergency lighting in which the distribution system is disconnected from the normal source and connected to a standby source
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Abstract
A big data processing platform network architecture comprises a cloud processing system and multiple terminal electronic devices. The cloud processing system comprises a storage device, a first server, and a second server. The storage device comprises multiple storage disks, and the storage device is connected with the first server and the second server through a storage resource management module. The first server and the second server each comprise a CPU, a process scheduling chip, an FPGA chip, a file processing interface, a fault processing module, a file interaction interface, and a network card, wherein the CPU is connected with the FPGA chip through the process scheduling chip, and the FPGA chip is connected with the storage resource management module through the file processing interface. According to the invention, all the terminal devices can be connected with the servers through a network to realize data uploading and downloading, and the architecture is of higher flexibility and is more convenient to use. Stored data is preprocessed by the FPGA chips, data collecting, data filtering and data processing are completed in one device, and the processing efficiency is greatly improved.
Description
Technical field
The present invention relates to a kind of big data processing structure, specifically a kind of big data processing platform (DPP) network architecture.
Background technology
Computer is one of 20th century greatest invention of the mankind, and along with science and technology and expanding economy, the era of knowledge-driven economy whistles, and as the computer of knowledge and the carrier of the process of information, transmission and storage, have also been obtained development energetically.21 century is the century that the mankind move towards information-intensive society, is the epoch of network, is super-high speed information highway construction age of making substantial progress and entering application.
Along with the high speed development of information technology is with universal, information technology is more and more higher to the dependency of big data, in recent years, have swepts the globe, change the life of people, work and mode of thinking deeply as big data such as tide.Increasing country starts to recognize big data from strategic level in the world, incorporates big data thinking and technology in governance field.The Chinese government also acts in accordance with seasonable for development trend, makes full use of big data and promotes state treatment ability, to agree with the propelling modern requirements of the times of state treatment ability.It may be said that the lifting that big data are government administrative ability brings opportunity to develop.First, big data can promote the change of governance theory and pattern.In governance field, by allowing magnanimity, dynamic, various data effective integration be valuable information resources, promote government to change management philosophy and governance model, and then accelerate improvement system and Governance Ability modernization.Secondly, big data can promote governance decision-making to become more meticulous and scientific.At big data age, the value of data produces qualitative change along with magnanimity accumulation, it is possible to economic society moving law intuitively being presented, thus reducing governance deviation probability, improving becoming more meticulous and scientific of governance.Finally, big data can promote governance improve efficiency and save cost.Utilize big data, it is possible to the data information making governance institute foundation is more comprehensive, and the coordination between different department and agencies is more smooth and easy, and then is effectively improved work efficiency, saves treatment cost.
The big data processing platform (DPP) of government might have stored polytype data such as each district, the population in street, legal person, house, certificate.The data volume in only one district, record sum is it is possible to reach about 100,000,000 at present, and Future Data amount can be continuously increased, and can introduce a large amount of unstructured data.In order to process the data of quickly increase, existing big data processing technique is also required to make corresponding raising.
Summary of the invention
It is desirable to provide a kind of big data processing platform (DPP) network architecture.
For realizing described purpose, the present invention provides following technical scheme:
A kind of big data processing platform (DPP) network architecture, system and multiple terminal electronic device 1 is processed including cloud, 2 ... n, described cloud processes system and includes storage device, first server and second server, described storage device includes multiple storage dish 1, 2 ... n, storage device is connected with first server and second server respectively by SRM module, described first server and second server all include CPU, process scheduling chip, fpga chip, file process interface, fault processing module, file interaction interface and network interface card, described CPU is connected with fpga chip by process scheduling chip, described fpga chip is connected with SRM module by file process interface.
Preferably, the CPU of described first server and second server is connected to fault processing module, network interface card and file interaction interface.
Preferably, the file interaction interface of described first server is connected with the file interaction interface of second server.
Preferably, the network interface card of described first server and second server is all connected with switch, and described switch connects network by router.
Preferably, described file interaction interface adopts two-port RAM.
Preferably, described terminal electronic device 1,2 ... n includes interconnective processing unit and control chip.
Preferably, described storage device, SRM module, first server, second server, switch and router are powered by power supply and are operated.
Preferably, described power supply is dual redundant power supply.
Compared with prior art, the invention has the beneficial effects as follows:
1, the present invention arranges multiple storage dish, and it is uniformly carried out management by SRM module, and be connected with each server and to realize data storage and Treatment Analysis, equipment cost is saved by shared storage device, file interaction can be carried out between server, improve treatment effeciency, also improve processing speed.
2, each server connects network by switch and router, and each terminal unit can pass through the connection of real-time performance and server, realizes uploading and downloading of data further, and motility is higher, and it is more convenient to use.
3, the CPU of each server is connected with SRM module by process scheduling chip and fpga chip, by fpga chip and process scheduling chip, storage data are carried out pretreatment, data acquisition, data filtering and data are processed and concentrates within one device, substantially increase treatment effeciency, also improve processing speed, be particularly suitable for the industry that data processing delay requirement is high.
4, described storage device, SRM module, first server, second server, switch and router unification are powered by power supply and are operated, and power supply is dual redundant power supply, support supplied by AC/DC electricity.
Accompanying drawing explanation
Fig. 1 is the structural representation of the present invention；
Fig. 2 is the structure chart of the terminal electronic device of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole embodiments.Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain under not making creative work premise, broadly fall into the scope of protection of the invention.
nullRefer to Fig. 1-2，In the embodiment of the present invention，A kind of big data processing platform (DPP) network architecture，System and multiple terminal electronic device 1 is processed including cloud、2……n，Described cloud processes system and includes storage device、First server and second server，Described storage device includes multiple storage dish 1、2……n,Storage device is connected with first server and second server respectively by SRM module，Described first server and second server all include CPU、Process scheduling chip、Fpga chip、File process interface、Fault processing module、File interaction interface and network interface card，Described CPU is connected with fpga chip by process scheduling chip，Described fpga chip is connected with SRM module by file process interface，Storage data are carried out pretreatment by fpga chip and process scheduling chip by the present invention，By data acquisition、Data filtering and data process to be concentrated within one device，Substantially increase treatment effeciency，Also improve processing speed，It is particularly suitable for the industry that data processing delay requirement is high.
The CPU of described first server and second server is connected to fault processing module, network interface card and file interaction interface；The file interaction interface of described first server is connected with the file interaction interface of second server, can realize the file interaction between first server and second server by file interaction interface.The network interface card of described first server and second server is all connected with switch, and described switch connects network by router.Multiple terminal electronic device 1,2 ... n passes through network connection route device, it is achieved with files passe and download that cloud processes system.In the present invention, multiple servers shares a set of storage device, and SRM module Main Function is to provide the distribution of virtual storage resource, for the demand according to server, is the corresponding virtual storage resource of server-assignment from storage device；File interaction interface adopts two-port RAM, and server monitors two-port RAM when there being file to write by listening port, reads the file of write from another port, thus realizing the mutual of file.
Described terminal electronic device 1,2 ... n includes interconnective processing unit and control chip, data transmission to the cloud gathered by terminal electronic device is processed system and processes.Each server connects network by switch and router, and each terminal electronic device can pass through the connection of real-time performance and server, realizes uploading and downloading of data further, and motility is higher, and it is more convenient to use.
Described storage device, SRM module, first server, second server, switch and router are powered by power supply and are operated, and described power supply is dual redundant power supply, support supplied by AC/DC electricity.
The above; it is only the present invention preferably detailed description of the invention; but protection scope of the present invention is not limited thereto; any those familiar with the art is in the technical scope that the invention discloses; it is equal to replacement according to technical scheme and inventive concept thereof or is changed, all should be encompassed within protection scope of the present invention.
Claims (8)
1. a big data processing platform (DPP) network architecture, system and multiple terminal electronic device 1 is processed including cloud, 2 ... n, described cloud processes system and includes storage device, first server and second server, described storage device includes multiple storage dish 1, 2 ... n, storage device is connected with first server and second server respectively by SRM module, it is characterized in that, described first server and second server all include CPU, process scheduling chip, fpga chip, file process interface, fault processing module, file interaction interface and network interface card, described CPU is connected with fpga chip by process scheduling chip, described fpga chip is connected with SRM module by file process interface.
2. the big data processing platform (DPP) network architecture of one according to claim 1, it is characterised in that the CPU of described first server and second server is connected to fault processing module, network interface card and file interaction interface.
3. the big data processing platform (DPP) network architecture of one according to claim 1 and 2, it is characterised in that the file interaction interface of described first server is connected with the file interaction interface of second server.
4. the big data processing platform (DPP) network architecture of one according to claim 1 and 2, it is characterised in that the network interface card of described first server and second server is all connected with switch, and described switch connects network by router.
5. the big data processing platform (DPP) network architecture of one according to claim 1 and 2, it is characterised in that described file interaction interface adopts two-port RAM.
6. the big data processing platform (DPP) network architecture of one according to claim 1, it is characterised in that described terminal electronic device 1,2 ... n includes interconnective processing unit and control chip.
7. the big data processing platform (DPP) network architecture of one according to claim 1 and 2, it is characterised in that described storage device, SRM module, first server, second server, switch and router are powered by power supply and be operated.
8. the big data processing platform (DPP) network architecture of one according to claim 7, it is characterised in that described power supply is dual redundant power supply.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610199594.2A CN105791412A (en) | 2016-04-04 | 2016-04-04 | Big data processing platform network architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610199594.2A CN105791412A (en) | 2016-04-04 | 2016-04-04 | Big data processing platform network architecture |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105791412A true CN105791412A (en) | 2016-07-20 |
Family
ID=56394758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610199594.2A Pending CN105791412A (en) | 2016-04-04 | 2016-04-04 | Big data processing platform network architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105791412A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106776024A (en) * | 2016-12-13 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of resource scheduling device, system and method |
| CN107748732A (en) * | 2017-11-09 | 2018-03-02 | 成都才智圣有科技有限责任公司 | A kind of big data processing equipment |
| CN108989811A (en) * | 2017-06-05 | 2018-12-11 | 中兴通讯股份有限公司 | Cloud desktop system and its image sequence compression coding method and medium |
| CN109412940A (en) * | 2018-10-23 | 2019-03-01 | 迈普通信技术股份有限公司 | Router administration method and router management system |
| CN109644157A (en) * | 2016-08-31 | 2019-04-16 | Nicira股份有限公司 | Use the fringe node cluster network redundancy and fast convergence of bottom anycast VTEP IP |
| CN109840844A (en) * | 2017-11-27 | 2019-06-04 | 上海仪电（集团）有限公司中央研究院 | A kind of financial big data acquisition processing device and system based on FPGA |
| CN111131220A (en) * | 2019-12-19 | 2020-05-08 | 广州极尚网络技术有限公司 | Method, device, equipment and storage medium for data transmission among multi-network environments |
| US10938788B2 (en) | 2018-12-12 | 2021-03-02 | Vmware, Inc. | Static routes for policy-based VPN |
| US11252024B2 (en) | 2014-03-21 | 2022-02-15 | Nicira, Inc. | Multiple levels of logical routers |
| US11283731B2 (en) | 2015-01-30 | 2022-03-22 | Nicira, Inc. | Logical router with multiple routing components |
| US11418445B2 (en) | 2016-06-29 | 2022-08-16 | Nicira, Inc. | Installation of routing tables for logical router in route server mode |
| US11425021B2 (en) | 2015-08-31 | 2022-08-23 | Nicira, Inc. | Authorization for advertised routes among logical routers |
| US11533256B2 (en) | 2015-08-11 | 2022-12-20 | Nicira, Inc. | Static route configuration for logical router |
| US11593145B2 (en) | 2015-10-31 | 2023-02-28 | Nicira, Inc. | Static route types for logical routers |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102761521A (en) * | 2011-04-26 | 2012-10-31 | 上海格尔软件股份有限公司 | Cloud security storage and sharing service platform |
| CN204009683U (en) * | 2014-07-29 | 2014-12-10 | 赖兆红 | A kind of large data processing equipment |
| CN204595833U (en) * | 2015-05-25 | 2015-08-26 | 刘佳浩 | A kind of cloud data handling system |
| CN204595849U (en) * | 2015-05-25 | 2015-08-26 | 张聪 | A kind of cloud data handling system |
-
2016
- 2016-04-04 CN CN201610199594.2A patent/CN105791412A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102761521A (en) * | 2011-04-26 | 2012-10-31 | 上海格尔软件股份有限公司 | Cloud security storage and sharing service platform |
| CN204009683U (en) * | 2014-07-29 | 2014-12-10 | 赖兆红 | A kind of large data processing equipment |
| CN204595833U (en) * | 2015-05-25 | 2015-08-26 | 刘佳浩 | A kind of cloud data handling system |
| CN204595849U (en) * | 2015-05-25 | 2015-08-26 | 张聪 | A kind of cloud data handling system |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11252024B2 (en) | 2014-03-21 | 2022-02-15 | Nicira, Inc. | Multiple levels of logical routers |
| US11799800B2 (en) | 2015-01-30 | 2023-10-24 | Nicira, Inc. | Logical router with multiple routing components |
| US11283731B2 (en) | 2015-01-30 | 2022-03-22 | Nicira, Inc. | Logical router with multiple routing components |
| US11533256B2 (en) | 2015-08-11 | 2022-12-20 | Nicira, Inc. | Static route configuration for logical router |
| US11425021B2 (en) | 2015-08-31 | 2022-08-23 | Nicira, Inc. | Authorization for advertised routes among logical routers |
| US11593145B2 (en) | 2015-10-31 | 2023-02-28 | Nicira, Inc. | Static route types for logical routers |
| US11418445B2 (en) | 2016-06-29 | 2022-08-16 | Nicira, Inc. | Installation of routing tables for logical router in route server mode |
| CN109644157A (en) * | 2016-08-31 | 2019-04-16 | Nicira股份有限公司 | Use the fringe node cluster network redundancy and fast convergence of bottom anycast VTEP IP |
| US11539574B2 (en) | 2016-08-31 | 2022-12-27 | Nicira, Inc. | Edge node cluster network redundancy and fast convergence using an underlay anycast VTEP IP |
| CN106776024A (en) * | 2016-12-13 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of resource scheduling device, system and method |
| CN106776024B (en) * | 2016-12-13 | 2020-07-21 | 苏州浪潮智能科技有限公司 | Resource scheduling device, system and method |
| WO2018223786A1 (en) * | 2017-06-05 | 2018-12-13 | 中兴通讯股份有限公司 | Cloud desktop system, and image sequence compression and encoding method, and medium therefor |
| CN108989811B (en) * | 2017-06-05 | 2022-06-03 | 中兴通讯股份有限公司 | Cloud desktop system, image sequence compression encoding method and medium thereof |
| CN108989811A (en) * | 2017-06-05 | 2018-12-11 | 中兴通讯股份有限公司 | Cloud desktop system and its image sequence compression coding method and medium |
| CN107748732A (en) * | 2017-11-09 | 2018-03-02 | 成都才智圣有科技有限责任公司 | A kind of big data processing equipment |
| CN109840844A (en) * | 2017-11-27 | 2019-06-04 | 上海仪电（集团）有限公司中央研究院 | A kind of financial big data acquisition processing device and system based on FPGA |
| CN109840844B (en) * | 2017-11-27 | 2023-12-22 | 上海仪电（集团）有限公司中央研究院 | Financial big data acquisition processing device and system based on FPGA |
| CN109412940B (en) * | 2018-10-23 | 2022-02-01 | 迈普通信技术股份有限公司 | Router management method and router management system |
| CN109412940A (en) * | 2018-10-23 | 2019-03-01 | 迈普通信技术股份有限公司 | Router administration method and router management system |
| US10938788B2 (en) | 2018-12-12 | 2021-03-02 | Vmware, Inc. | Static routes for policy-based VPN |
| CN111131220A (en) * | 2019-12-19 | 2020-05-08 | 广州极尚网络技术有限公司 | Method, device, equipment and storage medium for data transmission among multi-network environments |
| CN111131220B (en) * | 2019-12-19 | 2022-01-04 | 广州极尚网络技术有限公司 | Method, device, equipment and storage medium for data transmission among multi-network environments |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105791412A (en) | Big data processing platform network architecture | |
| CN105681628B (en) | A kind of convolutional network arithmetic element and restructural convolutional neural networks processor and the method for realizing image denoising processing | |
| CN106529670B (en) | It is a kind of based on weight compression neural network processor, design method, chip | |
| CN106878395A (en) | A kind of charging electric vehicle monitoring system based on mobile terminal | |
| CN106651144B (en) | Tower and line relation modeling method with same-tower parallel line information | |
| CN105468756A (en) | Design and realization method for mass data processing system | |
| CN102625144A (en) | Parallel video processing method based on Cloud Network of local area network | |
| CN106505575A (en) | A kind of Line Flow economic load dispatching method based on Granule Computing | |
| CN113177088B (en) | Multi-scale simulation big data management system for material irradiation damage | |
| CN109767002A (en) | A kind of neural network accelerated method based on muti-piece FPGA collaboration processing | |
| CN103020008B (en) | The reconfigurable micro server that computing power strengthens | |
| CN107729151A (en) | A kind of method of cluster management FPGA resource | |
| CN204887003U (en) | Big data processing platform network architecture | |
| CN108629448A (en) | A kind of data center's heat abstractor optimal control method and system based on Lyapunov optimizations | |
| CN204887018U (en) | Big data processing platform network architecture | |
| CN104866611B (en) | A kind of practical method for fast searching of power system Active Splitting section | |
| CN104090813A (en) | Analysis modeling method for CPU (central processing unit) usage of virtual machines in cloud data center | |
| CN104657797A (en) | Mobile Internet E-government platform system | |
| CN105630896A (en) | Method for quickly importing mass data | |
| Zhang et al. | Research on power system dispatching automation technology based on big data | |
| CN103631659A (en) | Schedule optimization method for communication energy consumption in on-chip network | |
| Nazaré et al. | Green Computing for Energy Transition: A Survey | |
| Liu et al. | Software-Hardware Co-Optimization for CNNs Based on Reconfigurable Devices | |
| WO2024082452A1 (en) | Transformer architecture-based urban integrated energy internet of things and intelligent management and control method and system | |
| CN102624688A (en) | Complex audio processing method adopting server capable of conducting dynamic allocation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160720 |