CN116389485A

CN116389485A - Data center system and method, map node, reduce node, equipment and chip

Info

Publication number: CN116389485A
Application number: CN202310651182.8A
Authority: CN
Inventors: 徐方鑫; 冉建军; 胡林平; 杨瑾
Original assignee: Shanghai Langli Semiconductor Co ltd
Current assignee: Shanghai Langli Semiconductor Co ltd
Priority date: 2023-06-05
Filing date: 2023-06-05
Publication date: 2023-07-04
Anticipated expiration: 2043-06-05
Also published as: CN116389485B

Abstract

The application provides a Map-Reduce-based transmission and calculation integrated data processing method, a data center system, map nodes, reduce nodes, electronic equipment and chips, and is applied to the fields of data processing technology and wireless communication technology. Wherein the data center system comprises: the system comprises a plurality of Map nodes and a Reduce node, wherein the Map nodes are in wireless connection with the Reduce node; the Map nodes and the Reduce nodes are deployed in equal distance, so that a plurality of Map nodes can transmit wireless data to the Reduce nodes simultaneously on a network structure. Key is mapped onto an orthogonal sequence at a Map node to be sent as a wireless signal, value can be counted at a Reduce node through orthogonal correlation demodulation, a data processing architecture and a data center system architecture integrated with transmission and calculation are realized, and data transmission efficiency is improved.

Description

Data center system and method, map node, reduce node, equipment and chip

Technical Field

The application relates to the technical field of data processing technology and wireless communication technology, in particular to a Map-Reduce based transmission and calculation integrated data processing method, a data center system, map nodes, reduce nodes, electronic equipment and chips.

Background

In a data center system for performing big data calculation processing, a core component of the data center system generally comprises a Map-Reduce (Map-Reduce) calculation model for performing big data distributed statistical calculation, wherein Map-Reduce is a distributed calculation model based on an index Key-Value (Key-Value) structure (i.e. Key Value pair), and a mapping Node (Map Node) and a Reduce Node (Reduce Node) in the model are connected together based on a communication network (such as a wired communication network, a wireless communication network, etc.), and perform data transmission based on the communication network.

The overall performance of the traditional data center system is easy to be limited by application deployment of a communication network in an actual scene, for example, when a wired communication network is adopted, the practical application limitations such as wired network topology, wired network card deployment cost, transmission conflict detection mechanism and transmission efficiency limitation are required to be considered, for example, when a wireless communication network is adopted, the application limitations such as setting of a wireless transmission mechanism and improvement of transmission efficiency are required to be faced, and the overall flexibility and scene adaptability are not high; in addition, the transmission and the calculation belong to two independent and separate links, namely, the physical layer in the OSI (Open System Interconnection) model performs data communication transmission of big data and the Map-Reduce calculation of the big data on the application layer, so that the overall performance of the big data calculation process needs to be improved.

Disclosure of Invention

In view of this, the embodiments of the present disclosure provide a Map-Reduce-based data processing method and a data center system, map nodes, reduce nodes, electronic devices, and chips, so as to implement a Map-Reduce overall data processing architecture with integrated transmission and calculation (i.e., integrated transmission and calculation) in the data center system based on a Map-Reduce architecture, which not only can save resource overhead, but also can improve processing efficiency, and can provide a brand new calculation processing architecture for the data center system with large data calculation processing.

The embodiment of the specification provides the following technical scheme:

the embodiment of the specification provides a Map-Reduce-based integrated data center system, which comprises: the system comprises a plurality of Map nodes and a Reduce node, wherein the Map nodes are in wireless communication connection with the Reduce node; the distances between each Map node and the Reduce node are equal;

the Map node comprises a mapping device and a wireless transmitting device, wherein the mapping device is used for carrying out mapping processing on input data to be processed, and the mapping processing comprises mapping Key values corresponding to all data in the data to be processed respectively with orthogonal sequences for signature; the wireless transmitting device is used for transmitting the mapped orthogonal sequence to the air in a wireless signal under the time synchronization;

The Reduce node comprises a wireless receiving device, a correlator unit and a key value pair statistics unit, wherein the wireless receiving device is used for wirelessly receiving wireless signals sent by the Map nodes under the time synchronization; the correlator unit is used for carrying out orthogonal sequence correlation demodulation on the wireless signal so as to obtain a Value counting result corresponding to the orthogonal sequence after demodulation; and the Key Value pair counting unit is used for counting Key-Value Key Value pair results corresponding to the data to be processed, which are input to the Map nodes, according to the Value counting result output by the correlator unit.

Compared with the prior art, the beneficial effects that above-mentioned at least one technical scheme that this description embodiment adopted can reach include at least:

in the invention, the plurality of Map nodes and the Reduce node are in wireless connection, and a wireless communication topological structure is formed between the plurality of Map nodes and the Reduce node, so that the plurality of Map nodes can simultaneously perform wireless data transmission to the Reduce node on a network structure, the data transmission efficiency is improved, the data transmission is not needed to be realized in a form of a wired network packet, and the related expenditure of the network packet is reduced;

In addition, the Map node and the Reduce node perform data transmission and data processing through the orthogonal sequences, and the Key in the Map-Reduce and the orthogonal sequences are associated, so that data transmission and calculation are realized on an air interface, value corresponding to the Key is obtained, the calculation efficiency of the Map-Reduce is improved, and a data center system and a data processing architecture integrating transmission and calculation are realized.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a data center system constructed of a conventional wired network;

FIG. 2 is a schematic diagram of a switch-based implementation of a multi-point access wired communication network between a Map node and a Reduce node;

FIG. 3 is a schematic diagram of a point-to-point wired communication network between a Map node and a Reduce node;

FIG. 4 is a schematic diagram of a conventional wireless network configured data center system;

FIG. 5 is a schematic diagram of a conventional data center system for Map-Reduce calculation;

FIG. 6 is a schematic diagram of the structure of an input text being processed by Map-Reduce calculation;

FIG. 7 is a schematic diagram of a Map-Reduce-based integrated data center system in the present application;

FIG. 8 is a schematic structural diagram of an air interface architecture for implementing a unified transmission and calculation in the present application;

FIG. 9 is a schematic diagram of mapping and forming data packets by Map nodes in the present application;

FIG. 10 is a schematic diagram of a structure in which multiple Map nodes together transmit a hybrid wireless signal to a Reduce node under time synchronization;

fig. 11 is a schematic structural diagram of a hybrid wireless signal being correlated in the present application;

fig. 12 is a schematic diagram of another structure of a hybrid wireless signal being correlated in the present application;

FIG. 13 is a schematic diagram of a result of a Value obtained by a Reduce node through correlation demodulation;

FIG. 14 is a schematic diagram of a data center system including 4 Map nodes and a Reduce node;

FIG. 15 is a schematic diagram of a Map node in the present application;

FIG. 16 is a schematic diagram of a Reduce node of the present application;

FIG. 17 is a schematic diagram of a hardware configuration of Map nodes for mapping and wireless transmission in the present application;

FIG. 18 is a schematic diagram of a hardware configuration of a Reduce node for wireless reception and associated demodulation in the present application;

fig. 19 is a schematic diagram of a communication-integrated data center system having a wireless communication network and a wired communication network.

Detailed Description

Embodiments of the present application are described in detail below with reference to the accompanying drawings.

Other advantages and effects of the present application will become apparent to those skilled in the art from the present disclosure, when the following description of the embodiments is taken in conjunction with the accompanying drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. The present application may be embodied or carried out in other specific embodiments, and the details of the present application may be modified or changed from various points of view and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

It is noted that various aspects of the embodiments are described below within the scope of the following claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present application, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, apparatus may be implemented and/or methods practiced using any number and aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.

It should also be noted that the illustrations provided in the following embodiments merely illustrate the basic concepts of the application by way of illustration, and only the components related to the application are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.

In addition, in the following description, specific details are provided in order to provide a thorough understanding of the examples. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details.

On the one hand, the existing data center system has respective application limitations whether the large data mapping-reduction (Map-Reduce) calculation is realized based on wired communication of a wired network architecture or the large data mapping-reduction (Map-Reduce) calculation is realized based on wireless communication of a wireless network architecture.

As shown in fig. 1, a data center system based on a wired network architecture based on a big data mapping-reduction (Map-Reduce) calculation model, wherein mapping nodes (Map nodes) and reduction nodes (Reduce nodes) are used for data transmission based on a communication network of wired connection (wired link).

In a traditional wired network, based on Map-Reduce calculation model characteristics, a connection topology for performing wired data communication between a plurality of Map nodes and one Reduce node can have two connection modes:

as illustrated in fig. 2, the first method is to implement a Multi-access (Multi-access) wired communication topology based on a Switch (Switch), that is, all nodes (Map nodes or Reduce nodes) transit through the Switch, for example, multiple Map nodes (such as M1 to M8) all make wired communication connection with the Reduce nodes through the Switch.

Disadvantages of Multi-access: all nodes need to pass through the switch, if only one wired network card is assumed to be arranged on the Reduce node (hereinafter, may be abbreviated as R), only one Map node (hereinafter, may be abbreviated as M) can receive data transmitted by the Map node at the same time. Therefore, if M1-M8 simultaneously transmit data to R, i.e. a many-to-one (MANUY-to-one) transmission scenario, since R only has one network card to receive one data at a time, M1-M8 simultaneously transmit data to R inevitably generates data transmission collision. In addition, even if it is possible to set up a collision-free data transmission, for example, M1 to M8 are transmitted through multiple rounds (for example, at least 8 times are required) at different times.

In other words, although collision-free data transmission is possible, the architecture is based on CSMA/CD (Carrier Sense Multiple Access with Collision Detection ), so when there are k M nodes in the system, at least k SLOTs (SLOTs) are needed to complete the data transmission.

As illustrated in fig. 3, the second is that the Map node and the Reduce node implement a point-to-point (point-to-point) wired communication topology directly through wired connection, that is, all Map nodes (e.g., M1 to M8) are directly wired to the Reduce node.

Point-to-Point disadvantage: all M nodes are directly connected to R node, which means that there are multiple network cards (8 network cards in FIG. 3) needed on R, so that data transmitted by all nodes can be received at the same time, namely if M1-M8 simultaneously transmit data to R (i.e. a source-to-one scene), no conflict exists (because there are multiple network cards), and only one round of transmission is needed.

In other words, in the point-to-line wired communication topology, although data transmission between k M nodes and R nodes can be completed only by 1 time SLOT (SLOT), at least k network cards need to be set in the R nodes, so that the hardware cost is very expensive, and the R nodes are provided with a plurality of network cards, so that not only the hardware architecture of the R nodes is complex, but also the software architecture of the R nodes is complex.

As shown in fig. 4, a data center system based on a wireless network architecture and based on a big data mapping-reduction (Map-Reduce) computing model, that is, on the basis of the Point-to-Point topology illustrated in fig. 3, mapping nodes (Map nodes) and reduction nodes (Reduce nodes) are communication networks based on wireless connection (wireless link) for data transmission, where a plurality of M nodes are stations (stations) respectively, and an R node is an AP (Access Point), so that mapping work of the Map nodes can be deployed on corresponding stations, and work of the Reduce nodes can be deployed on the Access points, so that all M nodes can be directly connected to the R node through the wireless network without an additional network card (such as a plurality of wired network cards illustrated in fig. 3).

Disadvantages of wireless network connections: since the connection of the wireless network is based on CSMA/CA (Carrier Sense Multiple Access with Collision Avoid, i.e. carrier sense multiple access with collision avoidance), i.e. the AP can only accept data of one station at a time, at least N rounds of transmission are also required to be transmitted with the wireless network (e.g. at least 8 rounds of transmission if n=8 in fig. 4, at least 16 rounds of transmission if other transmission links such as Acknowledgements (ACKs) etc. are also required to be included).

In two aspects, as in the foregoing conventional data center system, whether the large data mapping-reduction (Map-Reduce) calculation is implemented based on wired communication of the wired network architecture or the large data mapping-reduction (Map-Reduce) calculation is implemented based on wireless communication of the wireless network architecture, the data transmission and the data calculation in the large data processing are independent and belong to different separate links, and the overall performance of the large data computing processing based on Map-Reduce needs to be improved.

As shown in fig. 5, in the seven-layer OSI model of a computer network, a transmission link between a Map node and a Reduce node in a data center system only performs data transmission, such as data transmission through a physical layer, while a Map-Reduce calculation link (i.e., a Map-Reduce link) only performs Map-Reduce (Reduce) calculation of data, such as data mapping (Map) at an application layer, data reduction (Reduce) at an application layer, and the like, and other layers (such as a presentation layer, a session layer, a transmission layer, a network layer, and the like) complete data encapsulation and unpacking, so as to perform data adaptation between the application layer and the physical layer.

Specifically, the Map-Reduce calculation process flow performed by the data center system is approximately as follows:

firstly, the mapping node splits input data (such as file data) on an application layer, such as splitting the input data into single words line by line, wherein each word is used as a Value and corresponds to a Key Value, so that a Key can be used as an index to form a mapping processing result of a Key-Value pair (Key-Value); then, the mapping node transmits the key value pairs to a receiving unit (such as a wired network card, a wireless network card, an antenna and the like) in a reduction node at a physical layer by a transmitting unit (such as the wired network card, the wireless network card, the antenna and the like) in the physical layer in the mapping node in a network packet form required by a transmission network; finally, after receiving the packet data, the reduction node performs reduction processing of Key-Value statistical calculation on an application layer in the reduction node, namely a server where the reduction node is located reduces the Value number corresponding to each Key, integrates all the values, and finally obtains public statistical data.

Therefore, the transmission and the calculation are two separate links, so that the overall performance of Map-Reduce for performing large data processing in the conventional data center system is not high, for example, the transmission efficiency is low, for example, a large amount of resource overhead (such as computing resources for packet processing, transmission resources, storage resources, etc.) corresponding to network packet data is required.

As illustrated in fig. 6, assume that there is one Input (Input) document to be processed Input into a data center, the Input document including: "Deer Bear River Car Car River Deer Car Bear".

Firstly, splitting an input document row by row through a Mapping task (Mapping Tasks) of Map nodes to obtain three rows, wherein the three rows are respectively: the method comprises the steps of decomposing each line of text into each independent word after splitting line by line, wherein each word corresponds to a Key.

Then, using keys as indexes, sending different keys to different Reduce servers to carry out induction arrangement, wherein keys corresponding to each row of text are respectively: first row: deer,1; bear,1; river,1; second row: car,1; car,1; river,1; third row: deer,1; car,1; bear,1.

Then, a reduction task (reduction Tasks) in the Reduce node (such as a server) reduces the Value number corresponding to each Key, and reduces the number of times of keys appearing in the first row, the second row and the third row. Specifically, the reduction is carried out on the Bear, (1, 1) to obtain that the Bear appears twice for 1 time; reduce "Car, (1, 1)" until three 1 times Car appears; reducing Deer, (1, 1) to obtain 3 Deers appearing 1 time; reducing the 'River, (1, 1)' to obtain that the River appears twice for 1 time;

Finally, all values are integrated to obtain "Bear,2", "Car,3", "Deer,2" and "River,2", so that a common statistical data, i.e. Output, can be obtained according to the reduction result.

In the above process, the data transmission of each key-value pair is completed by the data communication between physical layers in the form of network packets, and the mapping (Map) of the data is performed at the application layer of the Map node, and there are other layers for completing the data packets required by the physical layer transmission, and after the physical layer of the Reduce node completes the receipt of the packets, the other layers of the Reduce node complete the unpacking and then provide the data required by the statistical computation of the reduction (Reduce) to the application layer, and finally the Reduce node completes the statistical computation of the reduction at the application layer. The idea of packaging and unpacking is to package keys and values into data (data).

Therefore, not only are the transmission and the calculation separated, but also a large amount of network packets, unpacking and other related resource overheads (such as calculation resources, storage resources, transmission resources and the like) exist, and the overall performance of the data center system for carrying out the big data Map-Reduce is not high.

Based on this, the embodiment of the specification provides a Map-Reduce-based transmission and calculation integrated data processing scheme:

as shown in fig. 7, the Key and the value are directly mapped to the wireless signal (i.e. the orthogonal sequence signal) in the Map node, wherein the orthogonal sequence can be used for signing the Key, so that the wireless signal with the same orthogonal sequence can be overlapped by utilizing the characteristic of wireless transmission, namely the signal superposition principle, and further the Reduce calculation processing of the corresponding Key can be completed in the Reduce node by demodulating the overlapped wireless signal, thereby realizing the data processing flow of the transmission and calculation integration.

In fig. 7, the upper layers of Map nodes (e.g., seven layers in the OSI model, i.e., all layers including the physical layer are collectively referred to as upper layers) Map the input pending data, wherein the mapping is mapping keys in Key value pairs to signatures (signatures) in corresponding codebooks (codebooks), wherein each signature is an orthogonal sequence.

Therefore, the core idea of the invention is that: the key in the Map-Reduce is corresponding to an orthogonal sequence, based on the characteristic of the orthogonal sequence, a Map-Reduce computing architecture integrating key-value transmission and reception can be realized through a corresponding orthogonal sequence transmitting and receiving circuit structure, so that a basic element of transmission and reception is not Data any more, but an orthogonal sequence (Orthogonal Sequence) is replaced, the orthogonal sequence itself has orthogonality, the length of the orthogonal sequence determines the number of sequences in the space, and if the length of the sequence is longer, the number of orthogonal sequences in the space is more.

In the present invention, the nodes can transmit orthogonal sequences simultaneously in the case of time synchronization. The orthogonal sequences transmitted by different nodes may be different and only time synchronization is required. The receiving node can receive and detect the orthogonal sequences in parallel, finally carries out threshold judgment and maps the sequence detection result to different value values, and finally completes key-value calculation required by Map-reduce.

As illustrated in fig. 8, a data center system of wireless network Topology (Topology) makes the following assumptions:

in the whole architecture, an access point AP (namely R in the figure, representing a Reduce node) is arranged, N nodes (namely M in the figure, representing Map nodes, namely N=8 in the figure) are arranged, the M and R work in an Air Merge mode, the signals of all M nodes are assumed to keep the same attenuation (namely equal distance and no multipath) to the R nodes, and only Gaussian white noise exists in a channel;

m node side: each M node works in a CSMA/CN (Carrier Sense Multiple Access with Collision Notification, carrier sense multiple access with conflict notification) mode and inherits all designs of the CN (Collision Notification, conflict notification), and all M nodes are jointly responsible for processing the work corresponding to the Map;

R node side: assuming that R works the same as the conventional AP design, i.e., one antenna is used, and the AP is responsible for handling the Reduce work;

such as processing the following text: "all the peer, all the bird, and all the xxx".

Thus, the Map-Reduce for the data center system is schematically illustrated as follows:

as shown in fig. 9, the M1 node picks up the first character "all", and the M1 node may map to form a data packet, where the data packet includes the "all" character and its corresponding signature, where the signature is a randomly selected one from the codebook, and it is assumed that all the signatures in the codebook are orthogonal;

accordingly, other Mi's also complete mapping and packet preparation accordingly (since multiple nodes in the system parse the text exemplified above at the same time, if a node takes a new character, mapping and packet preparation is also completed);

as shown in fig. 10 and 11, under the same time synchronization, all M nodes send the signature processed at this time to the R node, that is, identify a character and send a signature corresponding to the character when the Map works; after the R-side receives the mixed signal, the R-side correlates with K correlators (wherein K is equal to the number of correlated signal+1, uncorrelated is not correlated, and added 1 is CN) respectively, so as to obtain each value, for example, the signal sign_1 is correlated to obtain Val_1;

Specifically, if there are two nodes transmitting the same signature to R, the signal becomes 2 times due to the superposition principle of electromagnetic waves, and the result by the correlator is also increased by the same factor (as illustrated in fig. 12). The correlation value for a single signature is assumed to be 1 (by the assumption of topology, so the correlation value can be fixed). Thus, a table (e.g., the table illustrated in fig. 13) may be constructed at the Reduce node, where the statistics represent the number of local corresponding statistics, and the correlation value represents the correlation value obtained for the slot, and the statistics=statistics+correlation value. The data superposition of each Map is realized through electromagnetic wave superposition at the air interface, so that the transmission and calculation are completed synchronously.

In summary, based on Air Merge, N nodes need 1 SLOT to complete transmission, and meanwhile, the superposition operation is completed. Meanwhile, based on the relevant characteristics, the method is more robust under the condition of a good channel. Therefore, the multiple Map nodes and the Reduce node can be connected through wireless communication, and are arranged in an equidistant mode, for example, M is arranged on a round topology with the center of R, so that the multiple Map nodes can transmit data to the Reduce node at the same time, and after the orthogonal sequence is associated with the Key by utilizing the signal superposition and cancellation characteristics of the radio, only the value of signal processing is required to be associated with the value of Map-Reduce, so that Map-Reduce calculation can be completed when data are transmitted, and integral data processing of Map-Reduce is realized.

The following describes the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

As shown in fig. 14, an embodiment of the present disclosure provides a Map-Reduce-based data center system, including: the system comprises a plurality of Map nodes and a Reduce node, wherein the Map nodes are in wireless connection with the Reduce node, and the distances between each Map node and the Reduce node are equal.

In the embodiment of the present application, the number of Map nodes is not limited, and may be set according to actual needs, and examples may be 5, 6, and so on.

Fig. 14 is a schematic structural diagram including 4 Map nodes provided in the embodiment of the present application, where, as shown in fig. 14, M represents a Map Node (Map Node), R represents a Reduce Node (Reduce Node), and the Map Node includes: m1, M2, M3 and M4, wherein M1, M2, M3 and M4 are in Wireless connection (Wireless Link) with R.

In implementation, each Map node may include a mapping device and a wireless transmitting device, where the mapping device is configured to perform mapping processing on input data to be processed, where the mapping processing includes mapping Key values corresponding to respective data in the data to be processed respectively with orthogonal sequences for signature; the wireless transmitting device is used for transmitting the mapped orthogonal sequence to the air in a wireless signal under the time synchronization.

In implementation, the Reduce node includes a wireless receiving device, a correlator unit and a key value pair statistics unit, where the wireless receiving device is configured to wirelessly receive wireless signals sent by the Map nodes under the time synchronization; the correlator unit is used for carrying out orthogonal sequence correlation demodulation on the wireless signal so as to obtain a Value counting result corresponding to the orthogonal sequence after demodulation; the Key Value pair statistics unit is configured to count Key-Value Key values corresponding to the data to be processed input to the Map nodes according to the Value count result output by the correlator unit, and obtain the result

In other words, the mapping device maps the Key corresponding to the data with the orthogonal sequence for signature, and since the orthogonal sequence can be used as the baseband signal of the wireless signal, when the orthogonal sequence forms the wireless signal for wireless transmission, electromagnetic waves based on the wireless signal can be spatially superimposed, and thus the superimposed signal can be used for reflecting the Value count superposition of the Key after demodulation. Therefore, in the Map node, there is no longer a mapping between data and data, but a mapping between data and wireless signals. In addition, in the Reduce node, key-Value statistics is not performed again based on the packet and the unpack of the data, but Value statistics can be completed based on orthogonal correlation demodulation of the wireless signal containing the orthogonal sequence.

In some embodiments, a total of 5 nodes are included on the network architecture, and the nodes all have a wired network. The distance between any one node (e.g., M1, M2, M3, M4) and the central node R must be equal. Typically, all nodes need to physically construct an equally spaced circular topology around the central node R.

In some embodiments, in the embodiments of the present disclosure, the Map node and the Reduce node perform wireless communication through a wireless module. Wherein, can dispose wireless module respectively in Map node and Reduce node, fig. 15 is a Map node that this application embodiment provided, fig. 16 is a schematic diagram of wireless module in Reduce node, map node includes wireless transmitting device (wireless Transmitter) and mapping device in fig. 15, reduce node includes wireless receiving device (Wireless Correlator) in fig. 16, wireless transmitting device is connected with the transmitting antenna, carry out data transmission, wireless receiving device is connected with receiving antenna, wireless signal data that receiving Map node's wireless transmitting device sent, transmitting antenna and receiving antenna work on same channel.

The Map node includes a wireless transmitting device, and the Reduce node includes a wireless receiving device. The invention takes the characteristics of Key and Value in the wireless data center into consideration, replaces a method for packaging the Key and the Value into data (data), directly maps the Key and the Value into wireless signals, and optimizes related processing flows by utilizing the characteristics of wireless transmission.

Specifically, the wireless transmission apparatus is provided with a single antenna, i.e., a transmission antenna, to be dedicated for processing data transmission. The wireless receiving module is also provided with a single antenna which is used to specifically handle data reception. The transmit antenna and the receive antenna operate on the same channel.

Further, the method and the device associate the orthogonal sequence with Key indexes in Map-Reduce by utilizing the orthogonal characteristic of the orthogonal sequence, and associate the Value of signal processing with the Value of Map-Reduce by utilizing the signal superposition and cancellation characteristic of radio, so that wireless transmission and calculation integration is realized on an air interface, namely, map-Reduce calculation operation is completed while transmission is carried out, and the Map-Reduce efficiency is improved.

Specifically, the Map node transmits orthogonal sequences through a transmitting antenna connected with the first wireless transmitting module, each orthogonal sequence corresponds to one index keyword (Key), data (data) transmitted by the Map node is composed of a plurality of index keywords, and the Reduce node receives the orthogonal sequences transmitted by the Map nodes through a receiving antenna connected with the first wireless correlation module and processes the orthogonal sequences.

Wherein the length of the orthogonal sequence is determined according to the number of the orthogonal sequences.

The application designs a circuit structure for transmitting and receiving orthogonal sequences based on mathematical characteristics of the orthogonal sequences. In this application, a basic element of transmission and reception is not data, but an orthogonal sequence (Orthogonal Sequence) which itself has orthogonality. Here, the Key in Map-Reduce is actually corresponding to one orthogonal sequence, and the length of the orthogonal sequence determines the number of orthogonal sequences in space, and if the length of the orthogonal sequence is longer, the number of orthogonal sequences in space is greater. In the invention, the Map node can transmit orthogonal sequences simultaneously under the condition of time synchronization. Orthogonal sequences transmitted by different Map nodes can be different, and only time synchronization is needed. The Reduce node can receive and detect the orthogonal sequences in parallel, finally carries out threshold judgment and maps the orthogonal sequence detection result to different Value values, and finally completes Key-Value calculation required by Map-Reduce.

The generation of the orthogonal sequence by the Map node in the present application will be described in detail below.

Fig. 17 is a schematic hardware structure diagram of performing orthogonal sequence mapping, generating and transmitting by a Map node provided in an embodiment of the present application, as shown in fig. 17, where the mapping device includes: a plurality of orthogonal sequence generators and a mapping switch, each of said orthogonal sequence generators generating one of said orthogonal sequences; the mapping switch maps Key data generated by an upper layer to the corresponding orthogonal sequence generator; the orthogonal sequence generator generates an orthogonal sequence corresponding to a Key according to a mapping result of the mapping switch, and transmits the orthogonal sequence as a first baseband signal to the wireless transmitting device.

In implementation, each orthogonal sequence generator generates an orthogonal sequence, each orthogonal sequence corresponds to a specific Key, and the specific Key can be mapped to the corresponding orthogonal sequence generator through the mapping switch to generate an orthogonal sequence corresponding to the Key.

Specifically, the mapping switch maps data generated by an upper layer to a corresponding orthogonal sequence generator, index keywords (keys) in the data are in one-to-one correspondence with the orthogonal sequences, the orthogonal sequence generator generates a corresponding orthogonal sequence according to a mapping result of the mapping switch, the orthogonal sequence is transmitted to the first radio frequency unit as a first baseband signal to be processed, a first radio frequency signal is obtained, and the first radio frequency signal is transmitted to the first radio correlation module through a transmitting antenna.

In practical cases, N orthogonal sequence generators (i.e., orthogonal Sequence to Orthogonal Sequence N, which respectively correspond to Key 1 to Key N), the mapping relationship between the orthogonal sequences and the keys is preset through negotiation between nodes.

In Map nodes, the upper layer does not generate data, but directly maps the data to the corresponding Key in Map-Reduce, e.g., the upper layer directly generates a Key x. That is, the mapping device may determine which orthogonal sequence corresponds to the Key x, generate the corresponding orthogonal sequence by the orthogonal sequence generator, and then directly transmit the orthogonal sequence as the first Baseband Signal (Baseband Signal) to the radio transmitting device at the transmitting frequency end for processing, for example, up-converting the Signal, thereby generating the transmitted first radio frequency Signal (RF Signal). And finally, transmitting the signal through a transmitting antenna, thereby completing the whole transmitting process.

The following describes in detail the radio reception, correlation processing, and orthogonal sequence processing performed by the Reduce node in the present application.

Fig. 18 is a schematic diagram of a hardware structure of a Reduce node for performing wireless reception and orthogonal sequence correlation processing, where, as shown in fig. 18, a correlator unit includes a plurality of orthogonal sequence correlators; the key value pair statistical unit comprises a plurality of correlation peak level deciders; wherein, each orthogonal sequence correlator corresponds to the orthogonal sequence one by one, one of the orthogonal sequence correlators is connected with a corresponding correlation peak level decision device; the orthogonal sequence correlator is used for performing orthogonal sequence correlation demodulation on the input wireless signal and outputting a level value corresponding to the wireless signal; the correlation peak level decision device is used for performing decision counting on the level Value output by the orthogonal sequence correlator connected with the correlation peak level decision device so as to obtain the Value counting result of the orthogonal sequence corresponding to the orthogonal sequence correlator.

Specifically, after receiving the second radio frequency signal, the second radio frequency unit processes the second radio frequency signal to obtain a second baseband signal, and the second baseband signal is transmitted to the orthogonal sequence correlator, wherein the second radio frequency signal comprises first radio frequency signals sent by a plurality of first radio sending modules; after receiving the second baseband signal, the plurality of orthogonal sequence correlators perform correlation calculation on the second baseband signal to obtain correlation results, wherein the correlation results of each orthogonal sequence correlator represent the number of Map nodes for transmitting orthogonal sequences related to the orthogonal sequence correlators; the multiple correlation peak level deciders acquire correlation results transmitted by the orthogonal sequence correlators, obtain values corresponding to each orthogonal sequence, transmit the values to an upper layer, and represent the number of Map nodes for transmitting the orthogonal sequences.

In the embodiment of the present application, the Reduce node sends the index key word and the corresponding value corresponding to each orthogonal sequence correlator to the upper layer.

In an alternative embodiment, multiple Map nodes send orthogonal sequences to the Reduce node simultaneously.

In practical situations, the first wireless correlation module includes 1 receiving antenna, and the receiving antenna is connected with the first wireless receiving module. The first wireless receiving module includes n orthogonal sequence correlators (i.e., coreactors 1 through coreactor n correspond to keys 1 through key n, respectively), and each of the orthogonal sequence correlators is responsible for performing a correlation calculation of one of the orthogonal sequences.

In the first wireless receiving Module, after the receiving antenna receives the second radio frequency Signal (RF Signal), the second radio frequency Signal is transferred to a second radio frequency unit (Wi-Fi RF Module) for processing, such as performing down-conversion of the Signal, so as to generate a received second Baseband Signal (Baseband Signal). The second baseband signal is then passed in parallel to all of the orthogonal sequence correlators and correlation calculated. Since the correlation object of the orthogonal sequence Correlator is an orthogonal sequence, the correlation result is 0 as long as the correlation sequence and the correlated sequence are different (i.e. orthogonal), otherwise, if the correlation result is the same (i.e. the value of the coreactor Pulse), the correlation result represents the correlation amplitude, and represents the number of Map nodes for transmitting the orthogonal sequence at the moment. And sending the correlation result to a corresponding correlation peak level decision device (Pulse Level Detector) for level judgment, so as to obtain the Value corresponding to the orthogonal sequence. For example, when only one Map node transmits the orthogonal sequence, its coreformer Pulse should be 1, and if two Map nodes transmit the orthogonal sequence, its coreformer Pulse should be 2, in other words, the more Map nodes transmit the orthogonal sequence, the higher the Value corresponding to the coreformer Pulse and thus the higher the Value mapped.

Since the orthogonal sequence correlator has been associated with a different Key by the orthogonal sequence and also obtains the Value by threshold judgment after correlation, the corresponding Key and Value pair is actually generated on the different orthogonal sequence correlator. After that, the Reduce node may transfer both the Key and Value values mapped by the different orthogonal sequence correlators to the upper layer, and illustratively, as shown in fig. 18, key1, value1, key2, value2, and Key n, value n are sent to the upper layer.

According to the Map-Reduce large data calculation scene, the orthogonal characteristic of the orthogonal sequence is utilized to correlate the orthogonal sequence with the Key index in the Map-Reduce, the signal superposition and cancellation characteristic of the radio is utilized to correlate the Value of the signal processing with the Value of the Map-Reduce, so that wireless transmission and calculation integration is realized at the air interface, namely, map-Reduce calculation operation is completed while transmission is carried out, and the calculation efficiency of the Map-Reduce is improved.

In some embodiments, the plurality of Map nodes are disposed on a circumference centered on the Reduce node.

Preferably, the Map nodes are equally spaced on a circumference with the Reduce node as a center.

In some embodiments, as shown in fig. 15 and 16, the Map node further includes a first wired network card, and the Reduce node further includes a second wired network card; and the Map node and the Reduce node form a wired communication network based on the first wired network card and the second wired network card.

As shown in fig. 19, the wired communication network includes a wired communication network based on a switch, and a data center system integrating transmission and calculation can be realized in the original data center system by disposing a plurality of M nodes into wireless connection equidistant to R nodes.

Based on the same inventive concept, the embodiment of the present disclosure further provides a Map-Reduce-based data processing method, which is applicable to the Map-Reduce-based data center system described in any one of the above examples.

The embodiment of the application also provides a Map-Reduce-based transmission and calculation integrated data processing method, which comprises the following steps:

step S1: in each Map node, mapping Key values corresponding to all data in split data input to the Map node with orthogonal sequences for signature, wherein the split data is split from data to be processed and input to the Map node;

Step S2: under the time synchronization, the Map nodes transmit the mapped orthogonal sequences to the air through wireless signals;

step S3: in the Reduce node, wireless receiving is carried out on the wireless signals sent by the Map nodes under the time synchronization, a Value counting result corresponding to the orthogonal sequence is obtained after the orthogonal sequence correlation demodulation is carried out on the wireless signals, and Key-Value Key Value pair results corresponding to the data to be processed and input to the Map nodes are counted according to the Value counting result.

Specifically, multiple Map nodes send orthogonal sequences to the Reduce node at the same time, each orthogonal sequence corresponds to one index keyword, and data sent by the Map nodes consist of multiple index keywords.

And then, the Reduce node receives a plurality of orthogonal sequences sent by a plurality of Map nodes, carries out correlation calculation on the orthogonal sequences, and obtains the numerical value corresponding to each index keyword according to the result of the correlation calculation.

Based on the same inventive concept, the embodiment of the present disclosure further provides a Map-Reduce-based integrated transmission and calculation Map node, which is applied to the Map-Reduce-based integrated transmission and calculation data center system described in any one embodiment of the present disclosure.

Referring to the foregoing illustration of fig. 15, the transmission-calculation integrated Map Node (i.e., map Node) includes: mapping means and wireless transmitting means; the mapping device is used for carrying out mapping processing on input data to be processed, and the mapping processing comprises mapping Key values corresponding to all data in the data to be processed with orthogonal sequences for signature respectively; the wireless transmitting device is used for transmitting the mapped orthogonal sequences to the air in a wireless signal mode under time synchronization, so that after receiving the wireless signals transmitted by the Map nodes, a Reduce node in the integrated data center system calculates Key-Value Key Value pair results corresponding to the data to be processed and input to the Map nodes through orthogonal sequence correlation demodulation.

Based on the same inventive concept, the embodiment of the present disclosure further provides a Map-Reduce-based all-in-one reduction node, which is characterized in that the Map-Reduce-based all-in-one reduction data center system described in any one embodiment of the present disclosure is applied.

Referring to the foregoing illustration of fig. 16, the Reduce Node (i.e., reduction Node) includes a wireless receiving device, a correlator unit, and a key-value pair statistics unit; the wireless receiving device is used for wirelessly receiving wireless signals transmitted by a plurality of Map nodes in the integrated data center system under time synchronization; the correlator unit is used for carrying out orthogonal sequence correlation demodulation on the wireless signal so as to obtain a Value counting result corresponding to the orthogonal sequence after demodulation; and the Key Value pair counting unit is used for counting Key-Value Key Value pair results corresponding to the data to be processed, which are input to the Map nodes, according to the Value counting result output by the correlator unit.

Based on the same inventive concept, the embodiments of the present specification also provide an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform one or more steps of the Map-Reduce based data processing method described above.

Based on the same inventive concept, the embodiments of the present specification also provide a chip including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform one or more steps of the Map-Reduce based data processing method described above.

In this specification, identical and similar parts of the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the product embodiments described later, since they correspond to the methods, the description is relatively simple, and reference is made to the description of parts of the system embodiments.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions easily conceivable by those skilled in the art within the technical scope of the present application should be covered in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A Map-Reduce based data center system, comprising: the system comprises a plurality of Map nodes and a Reduce node, and is characterized in that the Map nodes are in wireless communication connection with the Reduce node; the distances between each Map node and the Reduce node are equal;

2. The Map-Reduce based data center system of claim 1, wherein the mapping means comprises: a plurality of orthogonal sequence generators and a mapping switch, each of said orthogonal sequence generators generating one of said orthogonal sequences;

the mapping switch maps Key data generated by an upper layer to the corresponding orthogonal sequence generator;

the orthogonal sequence generator generates an orthogonal sequence corresponding to a Key according to a mapping result of the mapping switch, and transmits the orthogonal sequence as a first baseband signal to the wireless transmitting device.

3. The Map-Reduce based data center system of claim 1, wherein the correlator unit comprises a plurality of orthogonal sequence correlators; the key value pair statistical unit comprises a plurality of correlation peak level deciders; wherein, each orthogonal sequence correlator corresponds to the orthogonal sequence one by one, one of the orthogonal sequence correlators is connected with a corresponding correlation peak level decision device;

the orthogonal sequence correlator is used for performing orthogonal sequence correlation demodulation on the input wireless signal and outputting a level value corresponding to the wireless signal;

The correlation peak level decision device is used for performing decision counting on the level Value output by the orthogonal sequence correlator connected with the correlation peak level decision device so as to obtain the Value counting result of the orthogonal sequence corresponding to the orthogonal sequence correlator.

4. The Map-Reduce based data center system of claim 1, wherein the plurality of Map nodes are disposed on a circumference centered on the Reduce node.

5. The Map-Reduce based data center system of claim 4, wherein the plurality of Map nodes are equally spaced apart on a circumference centered on the Reduce node.

6. The Map-Reduce based all-in-one data center system of any of claims 1-5, wherein the Map node further comprises a first wired network card and the Reduce node further comprises a second wired network card; and the Map node and the Reduce node form a wired communication network based on the first wired network card and the second wired network card.

7. The Map-Reduce based data center system of claim 6, wherein the wired communication network comprises a switch-based wired communication network.

8. The Map-Reduce-based integrated data processing method is characterized by being applicable to the Map-Reduce-based integrated data center system according to any one of claims 1-7, and comprises the following steps:

in each Map node, mapping Key values corresponding to all data in split data input to the Map node with orthogonal sequences for signature, wherein the split data is split from data to be processed and input to the Map node;

under the time synchronization, the Map nodes transmit the mapped orthogonal sequences to the air through wireless signals;

in the Reduce node, wireless receiving is carried out on the wireless signals sent by the Map nodes under the time synchronization, a Value counting result corresponding to the orthogonal sequence is obtained after the orthogonal sequence correlation demodulation is carried out on the wireless signals, and Key-Value Key Value pair results corresponding to the data to be processed and input to the Map nodes are counted according to the Value counting result.

9. A Map-Reduce based integrative Map node, which is applied to the Map-Reduce based integrative data center system as defined in any one of claims 1 to 7; the transmission and calculation integrated Map node comprises: mapping means and wireless transmitting means; the mapping device is used for carrying out mapping processing on input data to be processed, and the mapping processing comprises mapping Key values corresponding to all data in the data to be processed with orthogonal sequences for signature respectively; the wireless transmitting device is used for transmitting the mapped orthogonal sequences to the air in a wireless signal mode under time synchronization, so that after receiving the wireless signals transmitted by the Map nodes, a Reduce node in the integrated data center system calculates Key-Value Key Value pair results corresponding to the data to be processed and input to the Map nodes through orthogonal sequence correlation demodulation.

10. A Map-Reduce based all-in-one Reduce node, applied to the Map-Reduce based all-in-one data center system of any one of claims 1-7; the Reduce node comprises a wireless receiving device, a correlator unit and a key value pair statistics unit; the wireless receiving device is used for wirelessly receiving wireless signals transmitted by a plurality of Map nodes in the integrated data center system under time synchronization; the correlator unit is used for carrying out orthogonal sequence correlation demodulation on the wireless signal so as to obtain a Value counting result corresponding to the orthogonal sequence after demodulation; and the Key Value pair counting unit is used for counting Key-Value Key Value pair results corresponding to the data to be processed, which are input to the Map nodes, according to the Value counting result output by the correlator unit.

11. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform one or more steps of the Map-Reduce based data processing method of claim 8.

12. A chip, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform one or more steps of the Map-Reduce based data processing method of claim 8.