CN111092943B - Multi-cluster remote sensing method and system of tree structure and electronic equipment - Google Patents
Multi-cluster remote sensing method and system of tree structure and electronic equipment Download PDFInfo
- Publication number
- CN111092943B CN111092943B CN201911281722.8A CN201911281722A CN111092943B CN 111092943 B CN111092943 B CN 111092943B CN 201911281722 A CN201911281722 A CN 201911281722A CN 111092943 B CN111092943 B CN 111092943B
- Authority
- CN
- China
- Prior art keywords
- data center
- node
- tree
- data
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000004364 calculation method Methods 0.000 claims abstract description 32
- 238000009826 distribution Methods 0.000 claims description 23
- 230000015654 memory Effects 0.000 claims description 15
- 238000010276 construction Methods 0.000 claims description 9
- 238000003780 insertion Methods 0.000 claims description 8
- 230000037431 insertion Effects 0.000 claims description 8
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/44—Star or tree networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
- H04L41/0836—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability to enhance reliability, e.g. reduce downtime
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of information, and discloses a multi-cluster remote sensing method, a system and electronic equipment with a tree structure, wherein the method comprises the steps of firstly constructing a data center tree which comprises data center nodes for storing a data center; constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing; receiving tasks and distinguishing according to data information contained in the tasks; inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree; and the data center calculates according to the received tasks to obtain and display a calculation result. The invention can effectively utilize the storage and calculation resources of a plurality of data centers, namely, the invention has expansibility and maximization of calculation resources, can improve the processing speed, reduces unnecessary repeated calculation and improves the efficiency.
Description
Technical Field
The present invention relates to the field of information technology, and in particular, to a method, a system, and an electronic device for multi-cluster remote sensing with a tree structure.
Background
The big data remote sensing system disclosed at present has the following two types: one is a G-Hadoop framework structure, the whole Hadoop cluster is arranged into a large cluster, a slave node is controlled by a G-Hadoop master node, then one slave node is a cluster, and then a computing node is arranged inside each cluster. The whole system is constructed by adding a plugin on a main node to control the matedata of the whole cluster and then generating a Gfast file system, the system mainly stores data to be calculated by the whole system and plays a role of a data center, then a task is submitted to a master node every time, then the master schedules a file system to transmit files to each computing node through scheduling, and then the computing nodes receive the tasks to perform calculation, so that the construction work of the Hadoop cluster with multiple computing nodes is realized.
The other is Hadoop federation issued by Hadoop official website in 2017, which is a router-based federation system. The system writes another plug-in for scheduling and managing clusters, and each sub-cluster is a high-availability Hadoop cluster. Each Hadoop cluster then provides services to the outside through the router. There is then a state store on top of the sub-cluster, which is used to store the relationships of each node and the data storage structures in the node. The interaction of each sub-cluster is through a router to provide network services. When the user submits the task, the information is fed back to the user through the dispatching of the router to obtain the information at the node, and then the service is provided for the user.
The remote sensing big data system does not consider the data distribution problem of different data centers, and when a plurality of remote sensing data processing centers with different data distributions are processed, only the broadcasting distribution work of tasks is carried out, the distribution mode can cause the waste of resources, and the speed of data processing can be greatly influenced, namely, the existing system has the following defects:
1. the existing method is insensitive to the distribution of resources, and leads to excessive waste of communication resources.
2. Existing systems are mostly designed around a single data computing center, with certain obstacles to scalability and maximum utilization of computing resources.
Disclosure of Invention
The invention aims to provide a tree-structured multi-cluster remote sensing method, a tree-structured multi-cluster remote sensing system and electronic equipment aiming at the technical problems in the prior art.
In order to solve the problems proposed above, the technical scheme adopted by the invention is as follows:
a multi-cluster remote sensing method of a tree structure comprises the following specific steps:
step a: constructing a data center tree which comprises data center nodes for storing the data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: and the data center calculates according to the received tasks to obtain and display a calculation result.
Further, in the step a, the specific steps of constructing the data center tree are as follows:
collecting data distribution conditions and corresponding geographic information in each data center;
and constructing a data center tree of a top-down tree structure according to the distribution condition and the geographic information, wherein each leaf node of the tree corresponds to one data center respectively, namely being used as a data center node.
Further, the data center tree further comprises a main node, a process node and a null node, wherein the main node is connected with the process node, and the process node is connected with the data center node or the null node; the main node stores the full-range data latitude and longitude as an interface of the data center tree and external communication; the process nodes are divided according to the latitude and longitude range of the data, and two or more process nodes are set; and the data center node is connected with the corresponding process node according to the latitude and longitude range.
Furthermore, the data center node also stores a longitude and latitude range corresponding to the data information, and adopts a pointer string corresponding to the longitude and latitude range to guide the position of the data center node.
Further, the step a further comprises:
and querying the data center, namely querying the data center node in the data center tree corresponding to the data center node to find the data center to be queried.
Further, the step a further comprises: the method for inserting the data center comprises the following specific steps:
sending a message to be inserted into the data center to the main node;
after the host node receives the message, inquiring the data center to be inserted to obtain the latitude and longitude range contained in the data center to be inserted;
generating a node as a node to be inserted according to the latitude and longitude range, adjusting the data center tree, and inserting the node to be inserted into the data center tree;
and establishing the connection between the process node and other data center nodes in the data center tree and the node to be inserted to complete the insertion operation of the data center.
Further, the step a further comprises: deleting the data center, which comprises the following specific steps:
sending a message of a data center to be deleted to a main node;
after the master node A receives the message, inquiring a data center to be deleted to obtain a data center node corresponding to the data center to be deleted;
and adjusting the data center tree, deleting the data center nodes, disconnecting the data center nodes from the corresponding process nodes and other data center nodes, and finishing the deletion operation of the data center.
A tree-structured multi-cluster remote sensing system, the system comprising:
the data center tree building module: the data center tree is used for constructing a data center tree and comprises data center nodes for storing data centers;
a standby node construction module: the data center tree is used for constructing standby nodes, and the standby nodes are used for storing the data center tree and switching and periodically performing storage work;
the task receiving and distinguishing module: the task receiving module is used for receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
the inquiry distribution module: the data center node is used for inquiring a data center table and distributing the task to the corresponding data center node in the data center tree;
the data calculation display module: and the data center calculates according to the received tasks to obtain and display a calculation result.
Further, the data center tree building module comprises:
a data information collection module: for collecting data distribution and corresponding geographical information in each data center.
A central tree construction module: and constructing a data center tree of a top-down tree structure according to the distribution condition and the geographic information, wherein each leaf node of the tree corresponds to one data center respectively, namely being used as a data center node.
An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the following operations of the tree structured multi-cluster remote sensing method:
step a: constructing a data center tree, including data center nodes of a storage data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving tasks and distinguishing according to data information contained in the tasks;
step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: and the data center performs data calculation according to the received tasks to obtain and display a calculation result.
Compared with the prior art, the invention has the beneficial effects that:
according to the multi-cluster remote sensing method, the system and the electronic equipment with the tree structure, the data center is stored by constructing the data center tree, and the data center nodes are generated, so that the storage and calculation resources of a plurality of data centers can be effectively utilized; the standby nodes storing the data center tree are arranged and used for scheduling and coordinating work, so that fault tolerance can be effectively carried out; in addition, task distribution is carried out according to the data center nodes, processing speed can be improved, unnecessary repeated calculation is reduced, and efficiency is improved.
Drawings
FIG. 1 is a flow chart of a multi-cluster remote sensing method of a tree structure according to the present invention.
FIG. 2 is a schematic diagram of a data center tree structure according to the present invention.
FIG. 3 is a flow chart of data center insertion according to the present invention.
FIG. 4 is a flow chart of data center deletion according to the present invention.
FIG. 5 is a diagram illustrating a data center insert request according to the present invention.
FIG. 6 is a schematic diagram of the completion of the insertion of a data center according to the present invention.
FIG. 7 is a schematic diagram of the tree structured multi-cluster remote sensing system of the present invention.
FIG. 8 is a schematic diagram of a hardware device structure of a tree-structured multi-cluster remote sensing method provided by the present invention.
Detailed Description
To facilitate an understanding of the invention, the invention will now be described more fully with reference to the accompanying drawings. Preferred embodiments of the present invention are shown in the drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Referring to fig. 1, the present invention provides a tree-structured multi-cluster remote sensing method, which includes the following steps:
step a: and constructing a data center tree, wherein the data center tree comprises data center nodes for storing the data centers, namely leaf nodes of the tree correspond to the data centers as the data center nodes.
The method comprises the following specific steps of constructing the data center tree:
data distribution and corresponding geographic information in each data center is collected.
And constructing a data center tree of a top-down tree structure according to the distribution condition and the geographic information, wherein each leaf node of the tree corresponds to one data center respectively, namely being used as a data center node.
Step b: and constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing.
In the step b, since the data center contains a large amount of data and the data center tree contains a plurality of data centers, a standby node is constructed in the external system to store the data center tree, that is, to store the data of the data center, so as to perform switching when the data center tree fails, thereby ensuring the working reliability of the whole remote sensing system.
Step c: and receiving the tasks and distinguishing the tasks according to the data information contained in the tasks.
Step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree; wherein the data center table is generated by each data center.
Step e: and the data center performs data calculation according to the received task to obtain a calculation result and display the calculation result.
According to the multi-cluster remote sensing method with the tree structure, each data center or computing center is abstracted into one leaf node in the tree to construct the data center tree, and tasks can be distributed according to the corresponding data center nodes in the data center tree, so that the whole task distribution and task coordination process is more reasonable, resources are saved, and the working efficiency is improved.
In the present invention, the data center tree has a structure as shown in fig. 2, and the data center tree includes four nodes, which are a master node a, a process node B, a data center node C, and a null node D, respectively, where the master node a is connected to the process node B, and the process node B is connected to the data center node C or the null node. The master node A stores the full-range data latitude and longitude and serves as an interface of the data center tree and external communication. The process node B is divided according to the latitude and longitude range of the data, two or more than two process nodes are arranged, and the number of the process nodes is increased or decreased according to actual needs. And the data center node C stores the data information and the corresponding latitude and longitude range, and is connected with the corresponding process node B according to the latitude and longitude range. Since the number of nodes in the data center tree is greater than the number of data center nodes in practical use, there must be empty nodes, i.e., the process node B will also be connected to empty nodes.
The whole data center tree is a multi-branch tree, leaf nodes of each tree, namely data center nodes, are constructed by a data structure, each data center node comprises a section of latitude and longitude range to represent contained data information, a pointer string is adopted to guide the position of the data center node, and the corresponding data center node can be quickly found through the latitude and longitude range so as to find the required data information, so that the data searching time can be reduced, and the working efficiency is improved.
According to the data center tree constructed in the invention, because the number of data centers is huge, a main node and process nodes are set to connect the data center nodes, the main node is connected with a plurality of process nodes, each process node is connected with a plurality of data center nodes, namely, the process nodes are sequentially layered and connected according to the latitude and longitude ranges corresponding to the data, so that the whole data center tree has high expansibility and is adaptive to geographic data, the problem of data distribution of different clusters is effectively solved, the division of data contained in different data centers is also realized, the data center tree can be adaptive to a plurality of different data centers, the resource integration is facilitated, and the efficiency is improved.
In the invention, because the data in the data center nodes of the data center tree can be updated in real time, the query, insertion and deletion of the data center can be carried out after the data center tree is constructed. Further, the step a further comprises:
and querying the data center, namely querying the data center node corresponding to the data center tree to find the data center to be queried.
Further, the step a further comprises: the data center is inserted, and the flow is shown in fig. 3, which specifically includes the following steps:
and sending a message to be inserted into the data center to the main node A.
And after the master node A receives the message, inquiring the data center to be inserted to obtain the latitude and longitude range contained in the data center to be inserted.
And generating a node as a node to be inserted according to the latitude and longitude range, adjusting the data center tree, and inserting the node to be inserted into the data center tree.
And establishing the connection between other nodes (namely the process node B and other data center nodes C) in the data center tree and the node to be inserted, and completing the insertion operation of the data center.
Further, the step a further comprises: deleting the data center, wherein the flow is shown in fig. 4, and specifically includes the following steps:
and sending a message of the data center to be deleted to the main node A.
And after the master node A receives the message, inquiring the data center to be deleted to obtain the data center node corresponding to the data center to be deleted.
And adjusting the data center tree, deleting the data center node, disconnecting the data center node from other nodes (namely the corresponding process node B and other data center nodes C), and finishing the deletion operation of the data center.
The insertion process of the data center is further described by using specific examples, and referring to fig. 5, the latitude and longitude ranges (Lon: 0-45, Lat: 0-90) included in the data center to be inserted are included.
And sending a message to be inserted into the data center to the main node A.
After the master node A receives the message, inquiring the data center to be inserted to obtain the latitude and longitude range contained in the data center to be inserted, namely (Lon: 0-45, Lat: 0-90).
And generating a node as a node F to be inserted according to the latitude and longitude range, adjusting the data center tree, and inserting the node F to be inserted into the data center tree. Specifically, a process node B and a data center node C which need to be connected are searched according to the latitude and longitude range, namely the process node B corresponding to the latitude and longitude range (Lon: 0-180, Lat: 0-90) and the data center node C corresponding to the latitude and longitude range (Lon: 0-90, Lat: 0-90).
Establishing the connection between other nodes (i.e. the process node B and other data center nodes C that need to be connected) in the data center tree and the node F to be inserted, and completing the insertion operation of the data center F to be inserted, as shown in fig. 6.
Referring to fig. 7, the present invention further provides a tree-structured multi-cluster remote sensing system, which includes the following components:
the data center tree construction module: the method is used for constructing the data center tree and comprises the steps of storing data center nodes of the data centers, namely leaf nodes of the tree of the data center nodes correspond to the data centers.
A standby node construction module: the method is used for constructing a standby node, and the standby node stores the data center tree and is used for switching and periodically carrying out storage work.
And a task receiving and distinguishing module: the task receiving module is used for receiving the tasks and distinguishing the tasks according to data information contained in the tasks.
The inquiry distribution module: and the data center node is used for inquiring a data center table and distributing the task to the corresponding data center node in the data center tree.
The data calculation display module: and the data center calculates according to the received tasks to obtain and display a calculation result.
Further, the data center tree building module comprises:
a data information collection module: for collecting data distribution and corresponding geographical information in each data center.
A central tree construction module: and constructing a data center tree of a top-down tree structure according to the distribution condition and the geographic information, wherein each leaf node of the tree corresponds to one data center respectively, namely being used as a data center node.
FIG. 8 is a schematic diagram of a hardware device structure of a tree-structured multi-cluster remote sensing method provided by the present invention. As shown in fig. 8, the device includes one or more processors and memory. Taking a processor as an example, the apparatus may further include: an input system and an output system.
The processor, memory, input system, and output system may be connected by a bus or other means, as exemplified by the bus connection in fig. 8.
The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules. The processor executes various functional applications and data processing of the electronic device, i.e., implements the processing method of the above-described method embodiment, by executing the non-transitory software program, instructions and modules stored in the memory.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the processing system over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input system may receive input numeric or character information and generate a signal input. The output system may include a display device such as a display screen.
The one or more modules are stored in the memory and, when executed by the one or more processors, perform the following for any of the above method embodiments:
step a: constructing a data center tree which comprises data center nodes for storing the data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: and the data center performs data calculation according to the received tasks to obtain and display a calculation result.
The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the methods provided in the embodiments of the present application.
Embodiments of the present application provide a non-transitory (non-volatile) computer storage medium having stored thereon computer-executable instructions that may perform the following operations:
step a: constructing a data center tree which comprises data center nodes for storing the data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: and the data center performs data calculation according to the received tasks to obtain and display a calculation result.
Embodiments of the present application provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the following:
step a: constructing a data center tree which comprises data center nodes for storing the data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: the data center performs data calculation according to the received tasks to obtain calculation results and display the calculation results
According to the multi-cluster remote sensing method, the multi-cluster remote sensing system and the electronic equipment with the tree structure, the data center is stored by constructing the data center tree, and the data center nodes are generated, namely the data center tree can comprise a plurality of data centers, so that the storage and calculation resources of the plurality of data centers can be effectively utilized, and the expansibility and the maximization of the calculation resources are realized; the standby nodes which store the data center tree are arranged and used for scheduling and coordinating work, and fault tolerance can be effectively carried out. In addition, the latitude and longitude range is stored in the data center node, and a guide pointer is arranged, namely the whole data center tree is constructed according to geographic information, and tasks can be distributed according to data contained in different data centers in task scheduling, so that the processing speed is increased. Meanwhile, the data center tree is divided according to the geographic information, namely a plurality of process nodes are set, unnecessary repeated calculation operation is effectively solved, data of each data center can be more effectively concentrated, and redundancy is reduced.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (9)
1. A multi-cluster remote sensing method of a tree structure is characterized in that: the method comprises the following specific steps:
a, step a: constructing a data center tree which comprises data center nodes for storing the data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
step d: inquiring a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: the data center calculates according to the received task to obtain and display a calculation result;
the data center tree further comprises a main node, a process node and a null node, wherein the main node is connected with the process node, and the process node is connected with the data center node or the null node; the host node stores the full-range data latitude and longitude and serves as an interface of the data center tree and external communication; the process nodes are divided according to the latitude and longitude range of the data, and two or more process nodes are set; and the data center node is connected with the corresponding process node according to the latitude and longitude range.
2. The tree-structured multi-cluster remote sensing method according to claim 1, characterized in that: in the step a, the specific steps of constructing the data center tree are as follows:
collecting data distribution conditions and corresponding geographic information in each data center;
and constructing a data center tree of a top-down tree structure according to the distribution condition and the geographic information, wherein each leaf node of the tree corresponds to one data center respectively, namely being used as a data center node.
3. The tree-structured multi-cluster remote sensing method according to claim 2, characterized in that: the data center node also stores longitude and latitude ranges corresponding to the data information, and adopts pointer strings corresponding to the longitude and latitude ranges to guide the position of the data center node.
4. The tree-structured multi-cluster remote sensing method according to claim 3, wherein: the step a further comprises the following steps:
and querying the data center, namely querying the data center node in the data center tree corresponding to the data center node to find the data center to be queried.
5. The tree-structured multi-cluster remote sensing method according to claim 4, wherein: the step a further comprises the following steps: the method for inserting the data center comprises the following specific steps:
sending a message to be inserted into the data center to the main node;
after the host node receives the message, inquiring the data center to be inserted to obtain the latitude and longitude range contained in the data center to be inserted;
generating a node as a node to be inserted according to the latitude and longitude range, adjusting the data center tree, and inserting the node to be inserted into the data center tree;
and establishing the connection between the process node and other data center nodes in the data center tree and the node to be inserted to complete the insertion operation of the data center.
6. The tree-structured multi-cluster remote sensing method according to claim 5, wherein: the step a further comprises the following steps: deleting the data center, which comprises the following specific steps:
sending a message of a data center to be deleted to a main node;
after the master node A receives the message, inquiring a data center to be deleted to obtain a data center node corresponding to the data center to be deleted;
and adjusting the data center tree, deleting the data center nodes, disconnecting the data center nodes from the corresponding process nodes and other data center nodes, and finishing the deletion operation of the data center.
7. A system for implementing the multi-cluster remote sensing method of the tree structure of any one of claims 1 to 6, characterized in that: the system comprises:
the data center tree construction module: the data center tree is used for constructing a data center tree and comprises data center nodes for storing data centers;
a standby node construction module: the data center tree is used for building a standby node, and the standby node stores the data center tree and is used for switching and periodically storing;
and a task receiving and distinguishing module: the task receiving module is used for receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
the inquiry distribution module: the data center node is used for inquiring a data center table and distributing the task to a corresponding data center node in the data center tree;
the data calculation display module: and the data center calculates according to the received tasks to obtain and display a calculation result.
8. The tree structured multi-cluster remote sensing system according to claim 7, wherein: the data center tree building module comprises:
a data information collection module: the data center is used for collecting data distribution conditions and corresponding geographic information in each data center;
a central tree construction module: and constructing a data center tree of a top-down tree structure according to the distribution condition and the geographic information, wherein each leaf node of the tree corresponds to one data center respectively, namely being used as a data center node.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the following operations of the tree structured multi-cluster remote sensing method of any one of 1 to 6 above:
step a: constructing a data center tree which comprises data center nodes for storing the data center;
step b: constructing a standby node, wherein the standby node stores the data center tree and is used for switching and periodically storing;
step c: receiving the tasks and distinguishing the tasks according to data information contained in the tasks;
step d: querying a data center table, and distributing the task to a corresponding data center node in the data center tree;
step e: and the data center performs data calculation according to the received tasks to obtain and display a calculation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911281722.8A CN111092943B (en) | 2019-12-13 | 2019-12-13 | Multi-cluster remote sensing method and system of tree structure and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911281722.8A CN111092943B (en) | 2019-12-13 | 2019-12-13 | Multi-cluster remote sensing method and system of tree structure and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111092943A CN111092943A (en) | 2020-05-01 |
| CN111092943B true CN111092943B (en) | 2022-09-20 |
Family
ID=70396019
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911281722.8A Active CN111092943B (en) | 2019-12-13 | 2019-12-13 | Multi-cluster remote sensing method and system of tree structure and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111092943B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508639A (en) * | 2011-10-10 | 2012-06-20 | 北京邮电大学 | Distributed parallel processing method based on satellite remote sensing data characteristics |
| US8655070B1 (en) * | 2009-11-04 | 2014-02-18 | Google Inc. | Tree detection form aerial imagery |
| CN103793442A (en) * | 2012-11-05 | 2014-05-14 | 北京超图软件股份有限公司 | Spatial data processing method and system |
| CN104036286A (en) * | 2014-05-14 | 2014-09-10 | 武汉图歌信息技术有限责任公司 | Multispectral remote-sensing image classification processing method based on Hadoop |
| CA2846717A1 (en) * | 2013-03-15 | 2014-09-15 | Idan Shatz | Systems and methods for managing large volumes of data in a digital earth environment |
| CN108920542A (en) * | 2018-06-13 | 2018-11-30 | 苏州涅瓦信息科技有限公司 | A kind of distributed memory big data processing system and its data processing method |
-
2019
- 2019-12-13 CN CN201911281722.8A patent/CN111092943B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8655070B1 (en) * | 2009-11-04 | 2014-02-18 | Google Inc. | Tree detection form aerial imagery |
| CN102508639A (en) * | 2011-10-10 | 2012-06-20 | 北京邮电大学 | Distributed parallel processing method based on satellite remote sensing data characteristics |
| CN103793442A (en) * | 2012-11-05 | 2014-05-14 | 北京超图软件股份有限公司 | Spatial data processing method and system |
| CA2846717A1 (en) * | 2013-03-15 | 2014-09-15 | Idan Shatz | Systems and methods for managing large volumes of data in a digital earth environment |
| CN104036286A (en) * | 2014-05-14 | 2014-09-10 | 武汉图歌信息技术有限责任公司 | Multispectral remote-sensing image classification processing method based on Hadoop |
| CN108920542A (en) * | 2018-06-13 | 2018-11-30 | 苏州涅瓦信息科技有限公司 | A kind of distributed memory big data processing system and its data processing method |
Non-Patent Citations (5)
| Title |
|---|
| "Hierarchical Metric Learning for Optical Remote Sensing Scene Categorization";Akashdeep Goel;《IEEE》;20181219;全文 * |
| "云计算: 体系架构与关键技术";罗军舟;《通信学报》;20110111;全文 * |
| "大数据环境下卫星对地观测数据集成系统的关键技术";谢榕;《地球科学进展》;20150830;正文第856-860页，第1.3，2.2，3.3小节，第4，6节 * |
| "面向遥感应用的云服务关键技术研究";刘国良;《中国优秀硕士学位论文全文数据库》;20190915;全文 * |
| Massive remote sensing image data management based on HBase and GeoSOT;Lin Wang等;《IEEE》;20151112;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111092943A (en) | 2020-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258978B (en) | Data storage method | |
| CN105117171A (en) | Energy SCADA massive data distributed processing system and method thereof | |
| CN112236760B (en) | Graph data updating method, system, computer readable storage medium and equipment | |
| CN113900810A (en) | Distributed graph processing method, system and storage medium | |
| CN109639773B (en) | Dynamically constructed distributed data cluster control system and method thereof | |
| CN103399894A (en) | Distributed transaction processing method on basis of shared storage pool | |
| CN103544285A (en) | Data loading method and device | |
| CN110413845B (en) | Resource storage method and device based on Internet of things operating system | |
| CN105405070A (en) | Distributed memory power grid system construction method | |
| CN110727738B (en) | Global routing system based on data fragmentation, electronic equipment and storage medium | |
| CN113590576A (en) | Database parameter adjusting method and device, storage medium and electronic equipment | |
| CN105138679A (en) | Data processing system and method based on distributed caching | |
| CN103942330A (en) | Method and system for processing big data | |
| CN115221186A (en) | Data query method, system and device and electronic equipment | |
| CN102724290B (en) | Method, device and system for getting target customer group | |
| CN117056303B (en) | Data storage method and device suitable for military operation big data | |
| CN107908713B (en) | Distributed dynamic rhododendron filtering system based on Redis cluster and filtering method thereof | |
| CN111092943B (en) | Multi-cluster remote sensing method and system of tree structure and electronic equipment | |
| CN108829735B (en) | Synchronization method, device, server and storage medium for parallel execution plan | |
| Peng et al. | Message oriented middleware data processing model in Internet of things | |
| CN111881086B (en) | Big data storage method, query method, electronic device and storage medium | |
| CN104898608A (en) | Hadoop-based crop growth monitoring cloud platform and realization method thereof | |
| CN113051323B (en) | Water environment big data exchange method | |
| CN110109919A (en) | The method and apparatus for determining logical message | |
| CN113835728A (en) | Data updating method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |