CN102821042B - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN102821042B CN102821042B CN201210263936.4A CN201210263936A CN102821042B CN 102821042 B CN102821042 B CN 102821042B CN 201210263936 A CN201210263936 A CN 201210263936A CN 102821042 B CN102821042 B CN 102821042B
- Authority
- CN
- China
- Prior art keywords
- data
- message
- computing node
- processing equipment
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L67/125—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks involving control of end-device applications over a network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
The embodiment of the invention provides a data processing method, device and system. The method comprises the following steps of: receiving a first data report message sent from a data center, wherein the first data report message carries data sent to computing nodes; acquiring the data from the first data report message and pre-processing the data; and sending the pre-processed data to the computing nodes. According to the embodiment of the invention, the data sent from the data center can be pre-processed, the processing pressures of the computing nodes can be shared, and the computing nodes can be effectively prevented from the data congestion.
Description
Technical field
The embodiment of the present invention relates to technical field of data processing, particularly relates to a kind of data processing method, device and system.
Background technology
At existing Data processing, usually need the data of dispersion to converge, converge to unified computing node and process, according to the difference of the data type of each computing node process, multiple computing node can be set.
Prior art, in concrete implementation process, needs various types of data to converge, and the convergence device completing above-mentioned convergence function is called as service channel and converges frame, i.e. IO frame, and this IO frame plays convergence effect.IO frame is from different channel reception different pieces of information and be transmitted to computing node, and computing node needs to consume a large amount of resources and carries out data processing, and this can have a strong impact on the performance of each computing node.
Summary of the invention
The embodiment of the present invention provides a kind of data processing method, device and system, realizes carrying out preliminary treatment to data, shares the processing pressure of computing node, effectively prevents the congested of the data of computing node.
The embodiment of the present invention provides a kind of data processing method, comprising:
The first data that data processing equipment receives data center's transmission report message, and described first data report in message the data of carrying and sending to computing node;
Described data processing equipment reports message from described first data and obtains described data, carries out preliminary treatment to described data;
Pretreated data are sent to described computing node by described data processing equipment.
The embodiment of the present invention also provides a kind of data processing equipment, comprising:
First receiver module, the first data sent for receiving data center report message, and described first data report in message the data of carrying and sending to computing node;
Pretreatment module, obtains described data for reporting in message from described first data, and carries out preliminary treatment to described data;
First sending module, for sending to described computing node by pretreated data
The embodiment of the present invention also provides a kind of data handling system, comprising: above-mentioned data processing equipment and computing node, and computing node reports message for the second data receiving the transmission of described data processing equipment, reports message to process to described second data.
The data processing method that the embodiment of the present invention provides, device and system, carry out preliminary treatment by the data sent data center, shared the processing pressure of computing node, can effectively prevent at computing node generation data congestion.
Accompanying drawing explanation
Fig. 1 is the schematic flow sheet of the data processing method of the embodiment of the present invention one;
Fig. 2 is the schematic flow sheet of the data processing method of the embodiment of the present invention two;
Fig. 3 is the schematic flow sheet of the data processing method of the embodiment of the present invention three;
Fig. 4 is the structural representation of the data processing equipment of the embodiment of the present invention one;
The structural representation of the data processing equipment of Fig. 5 position embodiment of the present invention two;
Fig. 6 is the structural representation of data handling system embodiment of the present invention.
Embodiment
Fig. 1 is the schematic flow sheet of the data processing method of the embodiment of the present invention one.As shown in Figure 1, this data processing method comprises:
Step S101: the first data that data processing equipment receives data center's transmission report message, and the first data report in message the data of carrying and sending to computing node.
Wherein, data center such as: the computing node group etc. of server cluster, data acquisition system.Convergence device, as IO frame, converged by the data message that data center sends, data processing equipment receives the data message through converging sent from data center, namely the first data report message, and these first data report message to carry needs to issue the data of computing node.Computing node for receiving pretreated data, and through inner master data process operation, obtains result data.The implication of the data center used in each embodiment is below identical with implication used herein, repeats no more.
Step S102: data processing equipment reports message from the first data and obtains data, carries out preliminary treatment to these data.
Data processing equipment reports from the first data the data obtaining message and need to send to computing node, and carries out data scrubbing, data integration, data transformation, hough transformation etc. to these data, namely carries out preliminary treatment to data.
Step S103: pretreated data are sent to computing node by data processing equipment.
Pretreated data are sent to computing node, by the CPU in computing node, pretreated data is processed.
In the embodiment of the present invention, convergence device, as IO frame, can become one with data processing equipment, namely data processing equipment has data prediction function while carrying out data message convergence, and data processing equipment can as shown in this embodiment, be also self-contained unit.No matter which kind of implementation, just pro forma difference, can't cause the substantial variation of the present invention program.
The data processing method that the present embodiment provides, has shared the processing pressure of computing node, can play the effect effectively prevented at computing node data congestion.
Fig. 2 is the schematic flow sheet of the data processing method of the embodiment of the present invention two.As shown in Figure 2, the step of this data processing comprises:
S201: the first data receiving data center's transmission from the first physical interface of data processing equipment report message, and this first physical interface reports the protocol type of message corresponding with described first data, reports message for transmitting the first data.
Convergence device, as IO frame, converged by the data message that data center sends, data processing equipment receives the data message through converging sent from data center, namely the first data report message, and these first data report message to carry needs to issue the data of computing node.After converging, the first data that data center sends report message, by the first physical interface of data processing equipment, arrive data processing equipment.First data report message to comprise the data message of following arbitrary protocol type, such as: bus and interface standard (Peripheral Component Interface Express, hereinafter referred to as: PCIE) data message, it transfers to data processing equipment by the interface in corresponding PCIE interface group; Proprietary protocol data message, it, by the interface in corresponding proprietary protocol interface group, transfers to data processing equipment; Infiniband (InfiniBand, hereinafter referred to as IB) data message, it transfers to data processing equipment by the interface in corresponding IB interface group; 100000000 nets (Gigabit Ethernet, hereinafter referred to as GE) data message, Ethernet (Fast Ethernet, hereinafter referred to as: FE) data message (FE/GE data message), it transfers to data processing equipment by the interface in corresponding FE/GE interface group; Ten thousand megabits (TenGigabit Ethernet, hereinafter referred to as: 10GE) data message, it transfers to data processing equipment by the interface in corresponding 10GE interface group.
First physical interface is such as above-mentioned PCIE interface, proprietary protocol interface, IB interface, FE/GE interface, 10GE interface etc.
Step S202: the protocol type reporting message according to the first data, reports from the first data the data obtaining message and send to computing node.
Wherein, above-mentioned from the first data report message obtain send to the data of computing node to comprise:
Enterprise Resources Plan class data are such as the supply consumption of material and lastest imformations in producing, the production capacity of product and storing and change information etc. that shipment managing detailed catalogue, the technological process in production management, the calendar information that works, staff resources change and produce;
Audio frequency and video class data are such as the video monitoring datas that video monitoring system gathers, or the voice data etc. gathered in similar videoconference process;
Production process monitor data is such as to the temperature in product line, the isoparametric collection of humidity and the working condition that obtains controls class data.
Except above-mentioned send to the data of computing node except, also just like the data to delay requirement higher-class such as urgent fill order, abnormal alarm information, then do not need to carry out preliminary treatment, those data are directly sent to computing node to process by data center by direct connected link.
Step S203: preliminary treatment is carried out to the data of computing node that send to got.
In this step, data processing equipment carries out preliminary treatment to data, comprising: data scrubbing, data integration, data transformation, the one in the preliminary treatment such as hough transformation or its combination.
Wherein data scrubbing comprises: for incomplete in data, have vacant position, carry out arranging and supplementing containing noise, inconsistent data, vacancy value generally fills in the mean value of of a sort all samples with given tuple, or uses most probable value to fill.Through convergence device, after IO frame carries out convergence, a large amount of invalid false datas and repeating data can be formed, if such data are directly sent to computing node, huge processing pressure can be caused to computing node, therefore need to filter and Flow Control data, such as to the temperature gathered in production process, humidity parameter, if the sampled point arranged is overstocked, the a large amount of data collected, but the mass data gathered is substantially identical, therefore can carries out necessary Flow Control, namely by data processing equipment, data scrubbing preliminary treatment be carried out to data.
Data integration comprises: the first data from different data centers are reported the data in message, is integrated into consistent storage data, by data word for word section be mapped to data warehouse.Such as, in enterprise resource planning, the information that two relevant with personnel is had in data, when the type of the personnel of definition sex attribute, one may be char (2), the data value stored is " man " and " female ", and another attribute type is char (1), data value is " F " and " M ", the information of these two personnel is all correct, but for computing node, a kind of single method must be processed as to show this property value, the quality of guarantee data, by the preprocessed data integrating process of data processing equipment, finally realize the real unification of data.
Data transformation comprises: data are carried out conceptization, standardization, will comprise unstructured data and non-standard structure data transaction and become to be suitable for the form of data mining, to meet different requirements while acquisition.Such as, in the audio, video data gathered, different computing nodes requires different to the definition of data, then can carry out secondary structure by data processing equipment to data, converts necessary form to.
Hough transformation comprises: data carried out gathering, assemble, compress, and making the excavation of data more effective, as arranged authority to data staging, and carrying out weight sequencing to data.Such as, there is trans-regional branch company in enterprise, and different branch companies there will be business overlap for individual event project, the independent settlement data that branch company produces separately, so just easily duplicates clearing.By stipulations preliminary treatment, the part repeated in settlement data can be carried out normalizing, unduplicated part be carried out gathering and assembling, generates the Term Weight of each branch company simultaneously, the result of hough transformation directly generates individual event object totally to settle accounts, thus compression integral data volume.
Step S204: the protocol type used according to computing node, generate second data corresponding with the protocol type of this computing node and report message, these second data report message to comprise pretreated data, and send to this computing node by the second physical interface corresponding with the protocol type of computing node.Second physical interface reports the protocol type of message corresponding with described second data, and reporting message for transmitting described second data, such as, is PCIE interface, proprietary protocol interface, IB interface, FE/GE interface, 10GE interface etc.
In the embodiment of the present invention, convergence device, as IO frame, can become one with data processing equipment, namely data processing equipment has data prediction function while carrying out data message convergence, and data processing equipment can as shown in this embodiment, be also self-contained unit.No matter which kind of implementation, just pro forma difference, can't cause the substantial variation of the present invention program.
The data processing method that the embodiment of the present invention provides, adopts Ethernet by data center, and the first data that the plurality of access modes such as IB, PCIE send report message to carry out preliminary treatment, and meet the demand at multi-stage data center, long-range access capability is powerful.By data preprocessing method, avoid computing node in running and complex process is carried out to mass data make computing node data processing pressure excessive, cause the problem wasting computing node resource.
Fig. 3 is the schematic flow sheet of the data processing method of the embodiment of the present invention three.In the embodiment of the present invention, convergence device, as the data message that data center sends converges by IO frame, data processing equipment receives the data message through converging sent from data center, namely the first data report message, and these first data report message to carry needs to issue the data of computing node.Described first data through converging that the data center that data processing equipment receives sends report the data of carrying in message to be specially the data lower to delay requirement, any one or its combination in the data that occupied bandwidth is higher or the lower data of processing priority; Any one data in the data higher to the higher data of delay requirement, occupied bandwidth or the higher data of processing priority or its combination directly send to computing node by data center by direct connected link.
Concrete, please refer to Fig. 3, in the present embodiment, the data that data center sends are via after the conventional IO frame convergence of data center, first data report the data of carrying in message to be the data lower to delay requirement, any one or its combination in the data that occupied bandwidth is higher or the lower data of processing priority, such as to the memory access class instruction in the insensitive data of delay requirement, often because data volume is larger, preliminary treatment is carried out by data processing equipment, by corsspoint switch matrix channel transfer to computing node, as shown in the solid arrow flow direction in figure; And to any one data in the higher data of the higher data of delay requirement, occupied bandwidth or the higher data of processing priority or its combination, such as to the non-memory access class instruction in the higher data of delay requirement, the direct access path of computing node and data center can be taken, the equipment be as far as possible reduced by, the time delay of reduction equipment, gives stable bandwidth simultaneously, sends to computing node in time, thus reduce due to bandwidth occupied after the data congestion that causes, as indicated by a dashed arrow in the figure.Above-mentioned is only a kind of mode classification, can also according to different data class, and data mode decides different data paths.
In the above embodiment of the present invention, data processing equipment sends to computing node after data are carried out preliminary treatment, be specially: according to the plural message queue of pretreated data genaration, data in plural message queue are sent to corresponding computing node by each message queue corresponding computing node respectively.Traditional convergence device, i.e. IO frame, as IO frame service channel (Input/Output, hereinafter referred to as IO) virtual address (Virtual Address, hereinafter referred to as VA) can be mapped on data processing equipment, the protocol type used according to computing node is again mapped on this computing node, namely completes the address maps for real IO by Quadratic Map.For data processing equipment, itself play the effect of a virtual bridge of IO.Relatively traditional convergence device, as IO frame, CPU in data processing equipment or Reduced Instruction Set Computer (Reduced Instruction-Set Computer, hereinafter referred to as RISC) be a single IO, relative to computing node group, affiliated IO is invented the IO that can support many by data processing equipment, data processing equipment is divided by timeslice or the division of communication queue, reach the support to many with software, meet computing node to many virtualized requirements.
In the above embodiment of the present invention, not only can realize single virtual (Single-Root I/OVirtualization, hereinafter referred to as SR-IOV), after data processing equipment preliminary treatment, the Data Integration that IO frame forwards is issued, also can realize many virtual (Multi-Root I/O Virtualization, hereinafter referred to as MR-IOV), data packet level carries out resource-sharing and isolation to conventional IO frame, realizes high-level virtual.
In the embodiment of the present invention, convergence device, as IO frame, can become one with data processing equipment, even if data processing equipment has data prediction function while carrying out data message convergence, data processing equipment can as shown in this embodiment, be also self-contained unit.No matter which kind of implementation, just pro forma difference, can't cause the substantial variation of the present invention program.
Fig. 4 is the structural representation of the data processing equipment of the embodiment of the present invention one.As shown in Figure 4, this data processing equipment comprises: the first receiver module 1, pretreatment module 2, first sending module 3.
Wherein, the first receiver module 1 reports message for the first data receiving data center's transmission, and described first data report in message the data of carrying and sending to computing node.
Convergence device, as the data message that data center sends converges by IO frame, data processing equipment receives the data message through converging sent from data center, and namely the first data report message, and these first data report message to carry needs to issue the data of computing node.In embodiments of the present invention, the first receiver module 1 of data processing equipment receive data center send the first data report message.Wherein, data center is such as: the computing node group etc. of server cluster, data acquisition system, first receiver module receives the data message from each server and computing node, and namely the first data report message, and these first data report message to carry needs to issue the data of computing node.
Pretreatment module 2 for reporting the data obtaining in message and send to computing node from the first data, and carries out preliminary treatment to described data.Wherein, above-mentioned data comprise:
Enterprise Resources Plan class data are such as the supply consumption of material and lastest imformations in producing, the production capacity of product and storing and change information etc. that shipment managing detailed catalogue, the technological process in production management, the calendar information that works, staff resources change and produce;
Audio frequency and video class data are such as the video monitoring datas that video monitoring system gathers, or the voice data etc. gathered in similar videoconference process;
Production process monitor data is such as to the temperature in product line, the isoparametric collection of humidity and the working condition that obtains controls class data etc.
Except above-mentioned data, also just like the data of urgent fill order, the contour delay requirement class of abnormal alarm information, those data then do not need to carry out preliminary treatment, are directly sent to computing node to process by data center by direct connected link.
The pretreatment module 2 of data processing equipment reports from the first data that the first receiver module 1 receives the data obtaining message and send to computing node, and data scrubbing, data integration, data transformation, hough transformation etc. are carried out to these data, namely preliminary treatment is carried out to data.First sending module 3, for sending to computing node by pretreated data.
The data that pretreatment module 2 processed are sent to computing node by the first sending module 3, are processed it by the CPU in this computing node.
In the embodiment of the present invention, data processing equipment can with convergence device, as IO frame, become one, namely IO frame has data prediction function while carrying out data message convergence, and data processing equipment can as shown in this embodiment, be also self-contained unit.No matter which kind of implementation, just pro forma difference, can't cause the substantial variation of the present invention program.
The structural representation of the data processing equipment of Fig. 5 position embodiment of the present invention two.As shown in Figure 5, this data processing equipment comprises: the first physical interface 4, first receiver module 11, pretreatment module 22, first sending module 23, second physical interface 5.
Convergence device, as IO frame, converged by the data message that data center sends, data processing equipment receives the data message through converging sent from data center, namely the first data report message, and these first data report message to carry needs to issue the data of computing node.In embodiments of the present invention, the first physical interface 4, reports message for the first data transmitting specific protocol type, such as, is: PCIE interface group, proprietary protocol interface group, IB interface group, FE/GE interface group, 10GE interface group etc.
First receiver module 11, the first data sent for receiving data center from the first physical interface 4 of data processing equipment report message.First receiver module 11, such as be integrated with field programmable gate array (Field Programmable Gata Array, hereinafter referred to as FPGA) or Reduced Instruction Set Computer (Reduced Instruction-Set Computer, hereinafter referred to as RISC) the interface protocol modular converter of chip, it is by the first physical interface 4 of data processing equipment, and the first data receiving data center's transmission report message.First data report message to comprise the data message of following arbitrary protocol type, and such as: PCIE data message, it transfers to data processing equipment by the interface in corresponding PCIE interface group; Private protocol data message, it, by the interface in corresponding proprietary protocol interface group, transfers to data processing equipment; IB data message, it, by the interface in corresponding IB interface group, transfers to data processing equipment; GE/FE data message, it, by the interface in corresponding FE/GE interface group, transfers to data processing equipment; 10GE data message, it, by the interface in corresponding 10GE interface group, transfers to data processing equipment.
Pretreatment module 22, data for receiving the first receiver module 11 carry out preliminary treatment, as carried out data scrubbing to data, data being carried out to data integration, data are carried out to data transformation, carrying out stipulations to data, pretreatment module 22 according to circumstances can select process action data being carried out to above-mentioned a kind of or combination in any.In the present embodiment, pretreatment module 22 such as can make to be integrated with the preliminary treatment of the chips such as FPGA/RISC and virtualizes logical controls core, notebook data processing unit is made to become powerful CPU or Reduced Instruction Set Computer (Reduced Instruction-Set Computer, hereinafter referred to as RISC), the throughput performance of I/O gets a promotion.
First sending module 23, for the protocol type used according to computing node, generate second data corresponding with the protocol type of this computing node and report message, these second data report message to comprise pretreated data, and send to this computing node by the second physical interface corresponding with the protocol type of this computing node.In the present embodiment, pretreatment module 22 such as can make the interface protocol modular converter being integrated with the chips such as FPGA/RISC.Its by for the second physical interface, pretreated data are sent to this computing node.
Pretreated data, for according to specific protocol type, are sent to this computing node by the second physical interface 5.Concrete, the second physical interface is such as: PCIE interface group, proprietary protocol interface group, IB interface group, FE/GE interface group, 10GE interface group etc.
In another embodiment that data processing of the present invention is put, the first data that the data center that first receiver module 11 receives sends report the data of carrying in message to be specially the data lower to delay requirement, any one or its combination in the data that occupied bandwidth is higher or the lower data of processing priority; Any one data in the data higher to the higher data of delay requirement, occupied bandwidth or the higher data of processing priority or its combination directly send to computing node by data center by direct connected link.
To the insensitive data of delay requirement in above-described embodiment, be suitable for the data processing equipment adopting powerful disposal ability and access capability.Concrete, first receiver module 11 receives the first data that convergence device (as IO frame) sends and reports message to be the data lower to delay requirement, any one or its combination in the data that occupied bandwidth is higher or the lower data of processing priority, the instruction of such as memory access class, often because data volume is larger, preliminary treatment is carried out, by corsspoint switch matrix channel transfer to computing node by data processing equipment.
Such as, and to any one data in the higher data of the higher data of delay requirement, occupied bandwidth or the higher data of processing priority or its combination, non-memory access class instruction, can take the direct access path of computing node and data center.Data processing equipment needs the equipment be as far as possible reduced by for the I/O data that delay requirement is higher, reduce equipment time delay, give the stable network bandwidth simultaneously, with reduce due to bandwidth occupied after the data congestion that causes.
In the above embodiment of the present invention, pretreated data are sent to computing node, be specially: by plural for pretreated data genaration message queue, data in plural message queue are sent to corresponding computing node by each message queue corresponding computing node respectively.First sending module is by the plural message queue of pretreated for pretreatment module 22 data genaration, and the data in plural message queue are sent to corresponding computing node by each message queue corresponding computing node respectively.Above-mentioned is only a kind of mode classification, can also according to different data class, and data mode decides different data paths.
In the embodiment of the present invention, convergence device, such as IO frame and data processing equipment can become one, and namely IO frame has data prediction function while carrying out data message convergence, data processing equipment can as shown in this embodiment, be also self-contained unit.No matter which kind of implementation, just pro forma difference, can't cause the substantial variation of the present invention program.
Fig. 6 is the structural representation of data handling system of the present invention.As shown in Figure 6, this data handling system comprises: at least one data processing equipment 6, at least one computing node 7, at least one corsspoint switch matrix 8, at least one conventional IO frame 9 and at least one data center 10.
The data that the delay requirement that data center 10 sends is high, such as, in high delay requirement class data, low strap width variety data or high priority class data any one or its combination etc., computing node 7 is transmitted to after can being converged by conventional IO frame 9, data prediction is done by computing node 7, the action such as to divide into groups, unpack, namely directly send to conventional IO frame to be forwarded to computing node again by data center.For the insensitive data of delay requirement, such as low delay class data, the data that any one or its combination in high-band width variety data or low priority class data etc. is sent by data center 10, data processing equipment 6 can be transferred to by the first physical interface (not shown) of correspondence through the convergence of conventional IO frame 9, data are obtained by data processing equipment 6, and data scrubbing is carried out to data, data integration, data transformation and hough transformation etc. at least generate the second data after one of them data prediction and report message, computing node 7 is sent to via the second physical interface (not shown).The data sent by data center 10, also directly can transfer to after data processing equipment 6 processes accordingly and send to computing node 7.Can be connected by corsspoint switch matrix 8 between above-mentioned computing node 7, between data processing equipment 6 and computing node 7.
The methods, devices and systems of the data processing that the embodiment of the present invention provides, by carrying out preliminary treatment to data, share the disposal ability of computing node, and according to dissimilar, the height of priority, the size of data traffic of data, I/O data is flowed away different topological paths, prevent critical data congested, improve whole efficiency.
Last it is noted that above each embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to foregoing embodiments to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein some or all of technical characteristic; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the scope of various embodiments of the present invention technical scheme.
Claims (10)
1. a data processing method, is characterized in that, comprising:
The first data that data processing equipment receives data center's transmission report message, and described first data report in message the data of carrying and sending to computing node;
Described data processing equipment reports message from described first data and obtains described data, carries out preliminary treatment to described data;
Pretreated data are sent to described computing node by described data processing equipment;
Described data processing equipment reports message from described first data and obtains described data and be specially:
Described data processing equipment reports the protocol type of message according to described first data, reports message obtain described data from described first data;
Pretreated data send to computing node to comprise by described data processing equipment:
The protocol type that described data processing equipment uses according to described computing node, generate second data corresponding with the protocol type of described computing node and report message, described second data report message to comprise described pretreated data, and send to described computing node by the second physical interface corresponding with the protocol type of described computing node;
Described second physical interface reports the protocol type of message corresponding with described second data, reports message for transmitting described second data.
2. data processing method according to claim 1, it is characterized in that, described first data report message to comprise the data message of following arbitrary protocol type: PCIE data message, proprietary protocol data message, IB data message, FE/GE data message or 10GE data message;
The first data that described reception data center sends report message to be specially:
Described first data receiving the transmission of described data center from the first physical interface of described data processing equipment report message, and described first physical interface reports the protocol type of message corresponding with described first data, reports message for transmitting described first data.
3. data processing method according to claim 1, is characterized in that, described data comprise any one data following or its combination:
Enterprise Resources Plan class data, audio frequency and video class data or production process monitoring class data.
4. data processing method according to claim 1, is characterized in that, described to described data carry out preliminary treatment comprise as follows arbitrary or its combination:
Data scrubbing is carried out to described data;
Data integration is carried out to described data;
Data transformation is carried out to described data;
Data regularization is carried out to described data.
5. data processing method according to claim 1, is characterized in that, described first data report in message also carries low delay requirement class data, any one or its combination in high-band width variety data or low priority class data.
6. a data processing equipment, is characterized in that, comprising:
First receiver module, the first data sent for receiving data center report message, and described first data report in message the data of carrying and sending to computing node;
Pretreatment module, obtains described data for reporting in message from described first data, and carries out preliminary treatment to described data;
First sending module, for sending to described computing node by pretreated data;
Described pretreatment module, specifically for reporting the protocol type of message according to described first data, reports message from described first data and obtains described data;
The protocol type of described first sending module specifically for using according to described computing node, generate second data corresponding with the protocol type that described computing node uses and report message, described second data report message to comprise described pretreated data, and send to described computing node by the second physical interface corresponding to the protocol type of described computing node;
Described second physical interface, reports the protocol type of message corresponding with described second data, reports message for transmitting described second data.
7. data processing equipment according to claim 6, it is characterized in that, described first data report message to comprise the data message of following arbitrary protocol type: PCIE data message, proprietary protocol data message, IB data message, FE/GE data message or 10GE data message;
Described first receiver module reports message specifically for described first data receiving the transmission of described data center from the first physical interface;
Described first physical interface, reports the protocol type of message corresponding with described first data, reports message for transmitting described first data.
8. data processing equipment according to claim 6, is characterized in that, described pretreatment module is specifically for carrying out arbitrary or its combined treatment as follows:
Data scrubbing is carried out to described data;
Data integration is carried out to described data;
Data transformation is carried out to described data;
Data regularization is carried out to described data.
9. data processing equipment according to claim 6, it is characterized in that, the first data that the data center that described first receiver module receives sends report in message specifically carries low delay and requires class data, any one or its combination in high-band width variety data or low priority class data.
10. a data handling system, it is characterized in that, comprising: the arbitrary described data processing equipment of claim 6-9, also comprises computing node, described computing node reports message for the second data receiving the transmission of described data processing equipment, reports message to process to described second data.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210263936.4A CN102821042B (en) | 2012-07-27 | 2012-07-27 | Data processing method, device and system |
| PCT/CN2013/070509 WO2014015665A1 (en) | 2012-07-27 | 2013-01-16 | Data processing method, device, and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210263936.4A CN102821042B (en) | 2012-07-27 | 2012-07-27 | Data processing method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102821042A CN102821042A (en) | 2012-12-12 |
| CN102821042B true CN102821042B (en) | 2015-05-06 |
Family
ID=47304903
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210263936.4A Active CN102821042B (en) | 2012-07-27 | 2012-07-27 | Data processing method, device and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN102821042B (en) |
| WO (1) | WO2014015665A1 (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102821042B (en) * | 2012-07-27 | 2015-05-06 | 华为技术有限公司 | Data processing method, device and system |
| EP3460016A4 (en) * | 2016-05-18 | 2020-01-15 | Kuraray Co., Ltd. | Layered product |
| CN108171596A (en) * | 2017-12-28 | 2018-06-15 | 广州华夏职业学院 | A kind of multi task process analysis system and method for finance data |
| CN111125208A (en) * | 2018-10-30 | 2020-05-08 | 华为技术有限公司 | Data acquisition processing method, device and system |
| CN112613007B (en) * | 2020-12-22 | 2024-02-09 | 北京八分量信息科技有限公司 | Data admission method and device based on trusted authentication and related products |
| CN114267289A (en) * | 2021-12-31 | 2022-04-01 | 深圳市中科创激光技术有限公司 | Lamp temperature monitoring method and related equipment thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101027889A (en) * | 2004-08-06 | 2007-08-29 | 高通股份有限公司 | Wireless network supporting failover between network protocols |
| CN102118448A (en) * | 2011-03-17 | 2011-07-06 | 中兴通讯股份有限公司 | Method, system and terminal for realizing cloud computing |
| CN102576349A (en) * | 2009-08-18 | 2012-07-11 | 弗森-艾奥公司 | Communicating between host computers and peripheral resources in an input/output (I/O) virtualization system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006285811A (en) * | 2005-04-04 | 2006-10-19 | Hitachi Ltd | Storage system and data processing method |
| CN102821042B (en) * | 2012-07-27 | 2015-05-06 | 华为技术有限公司 | Data processing method, device and system |
-
2012
- 2012-07-27 CN CN201210263936.4A patent/CN102821042B/en active Active
-
2013
- 2013-01-16 WO PCT/CN2013/070509 patent/WO2014015665A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101027889A (en) * | 2004-08-06 | 2007-08-29 | 高通股份有限公司 | Wireless network supporting failover between network protocols |
| CN102576349A (en) * | 2009-08-18 | 2012-07-11 | 弗森-艾奥公司 | Communicating between host computers and peripheral resources in an input/output (I/O) virtualization system |
| CN102118448A (en) * | 2011-03-17 | 2011-07-06 | 中兴通讯股份有限公司 | Method, system and terminal for realizing cloud computing |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2014015665A1 (en) | 2014-01-30 |
| CN102821042A (en) | 2012-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102821042B (en) | Data processing method, device and system | |
| CN107590182B (en) | Distributed log collection method | |
| CN108335075B (en) | Logistics big data oriented processing system and method | |
| US20170131757A1 (en) | Real-time data management for a power grid | |
| CN102790771B (en) | A kind of document transmission method and system | |
| CN105468626B (en) | data processing method and system | |
| WO2018072687A1 (en) | Resource scheduling method and apparatus, and filtered scheduler | |
| CN107563947B (en) | Traffic and transportation integrated comprehensive service method and system | |
| CN101652750B (en) | Data processing device, distributed processing system and data processing method | |
| CN102546742B (en) | The method and apparatus of the down hop identifier in managing distributed switch structure system | |
| CN103761309A (en) | Operation data processing method and system | |
| CN109240820B (en) | Image processing task processing method and device, electronic equipment and storage medium | |
| CN103777950B (en) | Gridding method for resolving AOS (Advanced Orbiting System) telemetering data | |
| CN104038957A (en) | 4G base station operation maintenance information analysis process method based on integration structure | |
| CN113485792B (en) | Pod scheduling method in kubernetes cluster, terminal equipment and storage medium | |
| CN105760459A (en) | Distributed data processing system and method | |
| CN102480335A (en) | Method and system for transmitting business data | |
| CN102752219A (en) | Method for implementing virtual device (VD) interconnection and switching equipment | |
| CN100488114C (en) | Network element management method and system | |
| CN102510403B (en) | Receive and the cluster distributed system and method for real-time analysis for vehicle data | |
| US10534737B2 (en) | Accelerating distributed stream processing | |
| CN103269431B (en) | A kind of cloud transcoding implementation method and device | |
| US20180069941A1 (en) | Hierarchical caching and analytics | |
| CN103716183A (en) | Management method and management device of optical transport network | |
| CN109358820B (en) | Data access method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |