CN109634739A - A kind of data transmission method, device - Google Patents
A kind of data transmission method, device Download PDFInfo
- Publication number
- CN109634739A CN109634739A CN201811291689.2A CN201811291689A CN109634739A CN 109634739 A CN109634739 A CN 109634739A CN 201811291689 A CN201811291689 A CN 201811291689A CN 109634739 A CN109634739 A CN 109634739A
- Authority
- CN
- China
- Prior art keywords
- data
- transmission
- unit
- tree
- consumption side
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Abstract
The embodiment of the present application provides a kind of data transmission method, device, this method comprises:, according to data tree corresponding in data structure storage unit, generating first data with data tree when data producer generates data；When data transmission unit carries out first data transmission, according to data tree and preset data transmission structure, the first data are handled, to be converted to the second data；The second data are sent to data consumption side by data transmission unit；When data consumption side receives transmission data, summed data transmission structure is needed according to the data acquisition of data consumption side, obtains target data needed for transmission data.Therefore different data handling procedures is corresponded to using different data structure definition specifications, eliminate the coupled relation of the data structure between data consumption side and data producer, it realizes the decoupling between service, to simplify the data handling procedure of distributed system, saves development and maintenance cost.
Description
Technical field
This application involves information technology fields, more particularly to a kind of data transmission method, device.
Background technique
In the large-scale distributed system in internet, each event handler procedure has been coordinated by different application services
At a data may be transmitted between multiple application services, be handled.Therefore guarantee application service to the correct of data information
Interpretation is the key link of co-ordination between application service.In the prior art, unified data structure definition is generallyd use, also
It is that can be defined that data are produced and interpreted according to unified structure when the producer of data and the consumer of data handle data.But
With the continuous development of internet application field and technology, the data transmitted between distributed system are also increasingly complicated various, on
Processing scheme is stated to be difficult to meet the processing requirement of current large-scale distributed system.
Meanwhile application program processing data are depended on unduly when unifying a data structure definition, when the data structure of producer
It changes, the solution reader of consumer has to be followed by variation.It is very using data volume in large-scale distributed system
Huge, a data may circulate in many consumer applications, therefore can also bring huge development and maintenance cost.
Summary of the invention
In view of the above problems, the embodiment of the present application provides a kind of data transmission method, adopts when generating, sending and receiving
With different data structure specifications, the data structure that data consumption side's processing routine can be produced with noninductive primary data producer becomes
Change, and then solves the problems, such as that the data processing complex of distributed system, cost are huge.
Correspondingly, the embodiment of the present application also provides a kind of data transmission devices.
To solve the above-mentioned problems, the embodiment of the present application discloses a kind of data transmission method, is applied to data processing system
System, the data processing system includes data producer, Date Conversion Unit, data transmission unit, data consumption side, data knot
Structure definition unit, data structure storage unit and data decryptor unit, which comprises
When the data producer generates data, according to the tree-shaped knot of data corresponding in the data structure storage unit
Structure generates first data with the data tree；
When the data transmission unit carries out first data transmission, according to the data tree and preset data
Transmission structure handles first data, to be converted to the second data；
Second data are sent to the data consumption side by the data transmission unit；
When the data consumption side receives the transmission data, according to the data acquisition demand of the data consumption side
With the data transmission structure, target data needed for obtaining the data consumption side.
Correspondingly, the embodiment of the present application also discloses a kind of data transmission device, it is applied to data processing system, the number
It include data producer, Date Conversion Unit, data transmission unit, data consumption side, data structure definition list according to processing system
Member, data structure storage unit and data decryptor unit, comprising:
Data Integration module is used for when the data producer generates data, according to the data structure storage unit
In corresponding data tree, generate have the data tree the first data；
Data processing module is used for when the data transmission unit carries out first data transmission, according to the data tree
Shape structure and preset data transmission structure handle first data, to be converted to the second data；
Data transmission blocks are sent to for the data transmission unit using second data as transmission data described
Data consumption side；
Data acquisition module, for being disappeared according to the data when the data consumption side receives the transmission data
The data acquisition demand and the data transmission structure of Fei Fang, target data needed for obtaining the data consumption side.
The embodiment of the present application includes the following advantages:
The embodiment of the present application by the data producer generate data when, according in the data structure storage unit
Corresponding data tree generates first data with the data tree；It is carried out in the data transmission unit
When first data transmission, according to the data tree and preset data transmission structure, at first data
Reason, to be converted to the second data；Second data are sent to the data by the data transmission unit
Consumer；When the data consumption side receives the transmission data, according to the data acquisition demand of the data consumption side
With the data transmission structure, target data needed for the transmission data is obtained.
Therefore different data handling procedures correspond to using different data structure definition specifications, eliminate data consumption side with
The coupled relation of data structure between data producer realizes the decoupling between service, to simplify the data of distributed system
Treatment process saves development and maintenance cost.
Detailed description of the invention
Fig. 1 is a kind of step flow chart of data transmission method embodiment of the application；
Fig. 2 is a kind of step flow chart of data transmission method alternative embodiment of the application；
Fig. 3 is a kind of step flow chart of data transmission method alternative embodiment of the application；
Fig. 4 is a kind of step flow chart of data transmission method alternative embodiment of the application；
Fig. 5 is the step flow chart of another data transmission method embodiment of the application；
Fig. 6 is a kind of structural block diagram of data transmission device embodiment of the application；
Fig. 7 is a kind of structural schematic diagram for data processing system that one embodiment of the application provides.
Specific embodiment
In order to make the above objects, features, and advantages of the present application more apparent, with reference to the accompanying drawing and it is specific real
Applying mode, the present application will be further described in detail.
Before introducing the specific embodiment of the application, the application background of the disclosure is introduced first, is applied at present
Mainly there is XML (Chinese: extensible markup language in the data structure definition of the mainstream of the data processing of distributed system；English:
Extensible Markup Language), Json (Chinese: object representation；English: JavaScript Object
Notation), Protobuf (English: Google Protocol Buffers) etc., these data structure definitions are produced in data
Side and data consumer handle data using unified structural information.
It must include root element when wherein XML document defines, which is father's element of every other element, XML document
In element form a document tree.This tree expands to the lowermost end of tree since root.All elements can possess
Daughter element, while all elements can possess attribute, the element of leaf node can possess text.And Json is as a kind of light
The data interchange format of magnitude, every message are all wrapped around within braces, before the key name in key-value pair combination writes on
Face is simultaneously wrapped up with double quotation marks, and key and value are separated using colon, and colon is followed by value.In addition Protobuf is
The a set of of google exploitation stores for data, tool storage room when network communication for protocol encoding and decoding.It is fixed using Protobuf
When adopted data format, it is necessary to define corresponding * .proto file, then run the step that Protobuf is provided and produce corresponding processing
Program.
Inventor has found that above scheme is due to using unified structure in data producer and data consumer in use
Definition, so if data change, each side requires that variation is followed to change, and can not be suitable for large-scale distributed system pair
In the processing requirement of data, simultaneously because being increased using cumbersome data structure definition and data consumption pattern in large size point
The cost of data processing in cloth system increases burden for the exploitation and maintenance of system, therefore present applicant proposes one kind to be based on
The data transmission method and device of distributed system can adapt at the analysis that complicated and diversified data structure realizes data
Reason simplifies the analysis treatment process of data, reduces exploitation and use cost.
Referring to Fig.1, a kind of step flow chart of data transmission method embodiment of the application is shown, is applied at data
Reason system, the data processing system include data producer, Date Conversion Unit, data transmission unit, data consumption side, data
Structure definition unit, data structure storage unit and data decryptor unit, can specifically include following steps:
Step 101, when data producer generates data, according to the tree-shaped knot of data corresponding in data structure storage unit
Structure generates first data with data tree.
Illustratively, the Date Conversion Unit in the data processing system in the application, data transmission unit, data storage knot
Structure unit, data structure definition unit and data decryptor unit individually can be integrated in data processing system by processing unit
In, it is also possible to an achievable function in data processing system, is integrated with other function/unit or module,
The application and with no restrictions.The data processing system can also including journal processing unit etc. functions/module, to preferably it is complete
The realization of kind data processing system, the application is also with no restrictions.
Wherein, data producer is various business processing service, in the business processing service process of each application
Produced processing result is to be used as the source data of data producer, these processing results are varied, and different business generates knot
The structure of fruit is different.Such as in the anti-fraud of company and the decision in the face of risk business of credit, the information that client provides is carried out
Risk identification processing.In this process, it can be defined according to the index and rule of risk, calculate relevant historical data, obtain
The hit results of index.And these hit results can be processed by other operation systems and be used, so this treatment process is exactly to count
According to the source data generating process of producer.Since the generation of this source data is the processing result of risk indicator, so different indexs
Different data results will necessarily be generated with rule definition.As corporate business constantly develops, customer group constantly expands, and refers to
Mark and the type of rule definition are also more and more, therefore have also resulted in the complexity multiplicity of source data, and then directly affect
The data consumption side in downstream, service application processing load aggravate.For distributed system, data producer is as total
Database and/or processor, to a large number of services according to performed by system processing caused by mass data carry out store and/or
Processing, and data consumption side is other subdata base/processors, each subdata base is determined according to the business of its processing
It is required to obtain data from total Database, its business operation is executed so that processor is corresponding.
By introducing the technical solution of the application, source data is subjected to classification processing, before being transferred to data consumption side
It is processed as data transmission structure, so that different consumption services defines the data acquisition demand of data consumption side as needed, from
And make data consumption side only need from data transmission structure extract required for data, that is to say, that no matter data producer
Source data how to change, have the processing routine that will not influence data consumption side.
In the concrete realization, different data trees can be used according to generated different data class, it is first
The data of generation are generated as first data with corresponding data tree by first data producer, to be based on being somebody's turn to do later
The processing, encapsulation and transmission that data tree carries out data according to data transmission structure operate.
Step 102, when data transmission unit carries out first data transmission, according to data tree and preset data
Transmission structure handles the first data, to be converted to the second data.
Illustratively, after data producer is based on the first data of the tree-shaped structural generation of data by step 101, data are passed
Defeated unit is converted to the second data by the packing processing to the first data first, then will be with preset data transmission structure
Second data are sent to corresponding data consumption side, thus, it is possible to guarantee the validity of data transmission procedure and accuracy, and
When the structure of data producer changes, the data structure of recipient i.e. data consumption side is had no effect on.
Step 103, the second data are sent to data consumption side by data transmission unit.
Step 104, when data consumption side receives transmission data, according to the data acquisition demand sum number of data consumption side
According to transmission structure, target data needed for obtaining data consumption side.
That is, data consumption side is receiving the transmission with data transmission structure sent from data producer
Data need to carry out the transmission data received according to data transmission structure to unpack decapsulation processing, and combined data obtains demand
Target data is extracted, is further used convenient for data of the data consumption side for data producer.Such as based on data
First Data Integration of tree is mapped to corresponding data content by node serial number at transmission data.
In conclusion data transmission method provided by the present application, when data producer generates data, according to data structure
Corresponding data tree in storage unit generates first data with data tree；Data transmission unit into
When row first data transmission, according to data tree and preset data transmission structure, the first data are handled, to turn
It is changed to the second data；The second data are sent to data consumption side by data transmission unit；In data consumption side
When receiving transmission data, summed data transmission structure is needed according to the data acquisition of data consumption side, obtains institute in transmission data
The target data needed.Therefore different data handling procedures is corresponded to using different data structure definition specifications, eliminates data and disappears
The coupled relation of data structure between Fei Fang and data producer realizes the decoupling between service, to simplify distributed system
Data handling procedure, save development and maintenance cost.
Referring to Fig. 2, a kind of step flow chart of data transmission method alternative embodiment of the application, step 101 are shown
It is described when data producer generates the first data, will according to data tree preset in data structure storage unit
The classification of first data is integrated into data tree, including following sub-step:
Step 1011, the data according to caused by data producer determine corresponding in data structure storage unit
Data tree.
Step 1012, first data with data tree are generated.
Wherein, data tree includes intermediate node and leaf node, and intermediate node includes common intermediate node sum number
Group formula intermediate node；Leaf node be the bottom child node of data tree and leaf node include common leaf node, it is outer
Portion's field leaf node and reference leaf node.
Illustratively, in order to more be bonded the description of actual data structure, the data producer data knot that the application proposes
The data tree that structure defines has multiple nodes in the data tree without root node, for data producer
Data carry out exhaustive division definition, intermediate node and leaf node in data tree all have in the tree-shaped knot of data
Unique node serial number in structure, all nodes in data tree are distinguished and sorted out with management, array type middle node
Point is multiple types and the set for naming identical child node.All data informations caused by data producer pass through leaf
Node is described, wherein common leaf node is the child node of the intermediate node defined in a normal way, is defining one
When common leaf node other than the unique node serial number of needs and to the explanation of node type, it is also necessary to specify its node name
And data type, data type are divided into monodrome type, set Value Types and on-fixed Value Types.
And external field leaf node is the leaf node for quoting the field information defined in other libraries or file,
Already existing a large amount of field informations can be directly introduced into the data knot of data producer using external field leaf node
In structure, to avoid re-defining, therefore in external field leaf node other than unique node serial number and node type
It further include external reference field information, to illustrate external field information cited in the outside field leaf node.
And reference leaf node is to carry out node reference to the data structure of the data producer defined, is fitted
With there are in the data structure of associated two data producers.After introducing reference leaf node definition, the side of being cited can be changed
Data structure, the data structure of the side of reference will change correspondingly.It quotes in leaf node in addition to unique node serial number and section
It further include reference node information other than vertex type, to illustrate nodal information cited in the reference leaf node.
Several data trees are stored in advance in data structure storage unit, in order to which producer is generating number
According to when, carry out the use of corresponding data tree.
It should be noted that after handling through the above steps the first data, and by the number of all data producers
It is, for example, when data consumption bearing data producer requires to obtain data, will have in spatial cache according to storage region is stored in
There are the first data of data tree to be sent to corresponding data transmission unit, so as to data transmission unit to the first data into
Row is packaged encapsulation process.Therefore it can be more clear using such data tree and the first data are sorted out, and
Be conducive to establish the clearly mutual mapping relations between data producer, data transmission procedure and data consumption side, be convenient for
Data search and acquisition operation.
Referring to Fig. 3, a kind of step flow chart of data transmission method alternative embodiment of the application, step 102 are shown
It is described when data transmission unit carries out first data transmission, according to data tree and preset data transmission structure,
First data are handled, to be converted to the second data, including following sub-step:
Step 1021, data transmission unit receives the first data that data producer is sent.
Step 1022, data transmission unit calls Date Conversion Unit, to handle the first data.
Step 1023, Date Conversion Unit obtains data tree from data structure storage unit.
Step 1024, according to data tree and data transmission structure, Date Conversion Unit is by the first data processing
At the binary array being made of Head Section, structural area, undefined field area and data field.
Step 1025, binary array is sent to data transmission unit by Date Conversion Unit.
Be then for data transmission structure it is preset, it is fixed for data to be transferred to data consumption from data producer
The data structure of side, Head Section are used for the position in definition structure area, undefined field area and data field, and structural area includes transmission
Data structure information, undefined field area include the field information undefined in data consumption side, and data field includes the number of transmission
According to content information.
Illustratively, Head Section may include structural area index, undefined field area so and data field index；In structural area
Including quoting external field identification, node serial number, dynamic field number, value set length and value index etc.；Undefined field
It include higher level's type, higher level's number, field number, the value of field name index, Value Data type, set length, value index in area
Deng；Data field includes value length, value etc..When the first data are transferred to data consumption side from data producer, due to first
Data are that have the data of corresponding data tree, and data transmission unit transmits combined data data tree and data
Structure calls Date Conversion Unit to be converted to the second data to the packing multidomain treat-ment of the first data, and then is conducive to transmission process
While acquisition convenient for data consumption side to required data.And undefined field area, refer to the data of data producer production
The middle data that can have some definition corresponding not in data tree, also can not just generate the id information of correspondence mappings, be
Guarantee that the data of data producer will not lose, and then records this kind of data by undefined field area.
The first data processing is become into the second data suitable for transmission by the packing processing to the first data, data pass
Second data are sent to corresponding data consumption side by defeated unit, and thereby guarantee that data transmission procedure has
Effect property and accuracy.
Referring to Fig. 4, a kind of step flow chart of data transmission method alternative embodiment of the application, step 104 are shown
It is described when data consumption side receives transmission data, summed data transmission knot is needed according to the data acquisition of data consumption side
Structure, target data needed for obtaining data consumption side, comprising the following steps:
Step 1041, data consumption side receives the transmission data that data transmission unit is sent.
Step 1042, Date Conversion Unit calls in data consumption side, to handle transmission data.
Illustratively, the data transmission method that the application is proposed is that the conversion operation of data is utilized a data conversion list
What member was completed, which can be an independent processing unit in data processing system, be also possible to collect
At the function in some application service, e.g. in the integrated data conversion for being converted to data in data consumption side
Unit, when needing to carry out data conversion, data transmission unit or data producer can send data to data and turn
Unit is changed, with the conversion that defines of structure needed for carrying out to data, e.g. by the transmission data conversion received at meeting data
The target data of consumer's demand.
Step 1043, Date Conversion Unit obtains data acquisition demand from data structure storage unit.
Step 1044, Date Conversion Unit is extracted from transmission data according to data transmission structure and data acquisition demand
Meet the target data of data acquisition demand out.
Step 1045, target data is sent to data consumption side by Date Conversion Unit.
Wherein, data acquisition demand is used to illustrate the data requirements of data consumption side, it may be said that is to define data consumption
The data structure of side；That is, Date Conversion Unit is obtained according to the data of fixed data transmission structure and data consumer
Demand is taken, the transmission data received are converted, to obtain target data, and the target data is retransmited and is back to data
Consumer carries out the recycling of target data convenient for it.
Optionally, referring to Fig. 5, the step process of another data transmission method alternative embodiment of the application is shown
Figure, this method further include:
Step 105, when data decryptor unit monitors that data structure definition unit updates data tree and/or data
When acquisition demand, the updated data tree of correspondence and/or data acquisition demand are stored in data surplus storage unit.
In specific implementation, data processing rule may according to practical application scene change, data requirements change or
External field information variation etc. is corresponding to generate variation, such as data content needed for data consumption side changes, it is therefore desirable to logarithm
It is changed according to the data structure of consumer, modification data tree and/or data acquisition demand should be corresponded to, to adapt to data
The variation of data content needed for consumer.The monitoring of data processing rule can be in real time, data structure definition unit
The corresponding modification for carrying out data tree and/or data acquisition demand, later by modified data tree and/or number
According to obtain demand buffering into data structure storage unit, convenient for after carry out data processing when, Date Conversion Unit again into
Row calls.
Referring to Fig. 6, a kind of structural block diagram of data transmission device embodiment of the application is shown, is applied to data processing
System, the data processing system include data producer, Date Conversion Unit, data transmission unit, data consumption side, data knot
Structure definition unit, data structure storage unit and data decryptor unit, can specifically include following module:
Data Integration module 610, for being corresponded to according in data structure storage unit when data producer generates data
Data tree, generate have data tree the first data.
Data processing module 620 is used for when data transmission unit carries out first data transmission, according to data tree
With preset data transmission structure, the first data are handled, to be converted to the second data.
Data transmission blocks 630, for data transmission unit, by treated, the first data are used as transmission data to be sent to
Data consumption side.
Data acquisition module 640, for data consumption side receive transmission data when, according to the data of data consumption side
Acquisition needs summed data transmission structure, target data needed for obtaining data consumption side.
In the alternative embodiment of the application, Data Integration module 610, including following submodule:
Structure determination submodule is used for the data according to caused by data producer, determines in data structure storage unit
In corresponding data tree.
Data generate submodule, for generating the first data with data tree.
Wherein, data tree includes intermediate node and leaf node, and intermediate node includes common intermediate node sum number
Group formula intermediate node；Leaf node be the bottom child node of data tree and leaf node include common leaf node, it is outer
Portion's field leaf node and reference leaf node.
In the alternative embodiment of the application, data processing module 620, including following submodule:
Data receiver submodule receives the first data that data producer is sent for data transmission unit.
Data processing submodule calls Date Conversion Unit for data transmission unit, to handle the first data.
Structure acquisition submodule obtains data tree for Date Conversion Unit in data structure storage unit.
Data conversion submodule, for according to data tree and data transmission structure, Date Conversion Unit is by the
One data are processed into the binary array being made of Head Section, structural area, undefined field area and data field.
Binary array is sent to data transmission unit for Date Conversion Unit by data sending submodule.
Wherein, data acquisition demand is used to illustrate the data requirements of data consumption side, and Head Section is for definition structure area, uncertain
The position of adopted field area and data field, structural area include the data structure information of transmission, and undefined field area is included in data
The undefined field information of consumer, data field include the content information of the data of transmission.
In the alternative embodiment of the application, data acquisition module 640, including following submodule:
Data receiver submodule receives the transmission data that data transmission unit is sent for data consumption side.
Submodule is called, Date Conversion Unit is called for data consumption side, to handle transmission data.
Structure acquisition submodule obtains data acquisition demand for Date Conversion Unit from data structure storage unit.
Data acquisition submodule, for Date Conversion Unit according to data transmission structure and data acquisition demand, from transmission
The target data for meeting data acquisition demand is extracted in data.
Target data is sent to data consumption side for Date Conversion Unit by data sending submodule.
Optionally, the device further include: update module, for monitoring data structure definition unit when data decryptor unit
When updating data tree and/or the modification of data acquisition demand, the updated data tree of correspondence and/or data are obtained
Demand is taken to be stored in data structure storage unit.
For device embodiment, since it is basically similar to the method embodiment, related so being described relatively simple
Place illustrates referring to the part of embodiment of the method.
The embodiment of the present application also provides a kind of non-volatile readable storage medium, be stored in the storage medium one or
Multiple modules (programs) when the one or more module is used in terminal device, can make the terminal device execute
The instruction (instructions) of various method steps in the embodiment of the present application.
Fig. 7 is a kind of structural schematic diagram for data processing system that one embodiment of the application provides.As shown in fig. 7, the number
It may include data producer 70, Date Conversion Unit 71, data transmission unit 72, data consumption side 73, number according to processing system
According to structure definition unit 76, data decryptor unit 74 and data storage cell 75, for executing data described in above-described embodiment
Step in transmission method.
Wherein, data transmission unit 72 is for realizing the data communication between data producer and data consumer.Data
Producer 70 handles the request of data consumer, can obtain detailed geographic location information or facility information, such as
Geographical location information service, device-fingerprint service are provided.Date Conversion Unit 71 can be built-in data producer and data
Processing unit in consumer's service enables data producer and data consumer to call directly the unit and carries out to data
Processing.Data transmission unit 72 is responsible for carrying out data transmission in a distributed system, and the mode of data transmission is general to be divided into two
Kind: one is by message-oriented middleware transmission, such as there is kafka, ActiveMQ, RabbitMQ etc.；Another kind is that RPC is remotely adjusted
With representative has Dubbo, Thrift, RESTful etc..Data consumption side 73 is by obtaining address information service and facility information
Data, to judge that request data with the presence or absence of risk, such as provides decision in the face of risk service.Data decryptor unit 74 for example passes through
Zookeeper monitors the variation of the information such as the structure configuration information in the monitored datas Transmission systems such as the binlog variation of mysql.
Storage is divided into caching and two kinds of persistent storage in data storage cell 75.Caching can be outside JVM heap memory or JVM heap
Memory or Redis, the remote caches such as Memcache are used to producer and consumer's quick obtaining data structure information.Persistently
Changing storage can be mysql, and the databases such as oracle can be with persistence data structure information.Data structure definition unit (or
Be data structure configuration unit) 76 be taken through interaction platform (such as webpage) creation, modification, delete data tree and
The mode of data acquisition demand, and the tree-shaped structure of data structure and data acquisition demand are first persisted in database (ratio
Such as mysql database), then by data decryptor unit from the database by information exploitation into data storage cell 75.
All the embodiments in this specification are described in a progressive manner, the highlights of each of the examples are with
The difference of other embodiments, the same or similar parts between the embodiments can be referred to each other.
Although preferred embodiments of the embodiments of the present application have been described, once a person skilled in the art knows bases
This creative concept, then additional changes and modifications can be made to these embodiments.So the following claims are intended to be interpreted as
Including preferred embodiment and all change and modification within the scope of the embodiments of the present application.
Finally, it is to be noted that, herein, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that process, method, article or terminal device including a series of elements not only wrap
Those elements are included, but also including other elements that are not explicitly listed, or further includes for this process, method, article
Or the element that terminal device is intrinsic.In the absence of more restrictions, being wanted by what sentence "including a ..." limited
Element, it is not excluded that there is also other identical elements in process, method, article or the terminal device for including the element.
Specific examples are used herein to illustrate the principle and implementation manner of the present application, and above embodiments are said
It is bright to be merely used to help understand the present processes and its core concept；At the same time, for those skilled in the art, foundation
The thought of the application, there will be changes in the specific implementation manner and application range, in conclusion the content of the present specification is not
It is interpreted as the limitation to the application.
Claims (10)
1. a kind of data transmission method, which is characterized in that be applied to data processing system, the data processing system includes data
Producer, Date Conversion Unit, data transmission unit, data consumption side, data structure definition unit, data structure storage unit
And data decryptor unit, which comprises
When the data producer generates data, according to data tree corresponding in the data structure storage unit,
Generate first data with the data tree；
When the data transmission unit carries out the first data transmission, according to the data tree and preset data
Transmission structure handles first data, to be converted to the second data；
Second data are sent to the data consumption side by the data transmission unit；
When the data consumption side receives the transmission data, according to the data acquisition demand of the data consumption side and institute
Data transmission structure is stated, target data needed for obtaining the data consumption side.
2. the method according to claim 1, wherein it is described the data producer generate data when, according to
Corresponding data tree in the data structure storage unit generates first data with the data tree,
Include:
According to the data caused by the data producer, corresponding number in the data structure storage unit is determined
According to tree；
Generate first data with the data tree；
Wherein, the data tree includes intermediate node and leaf node, and the intermediate node includes common intermediate node
With array type intermediate node；The leaf node is the bottom child node of the data tree and the leaf node includes
Common leaf node, external field leaf node and reference leaf node.
3. the method according to claim 1, wherein described carry out first number in the data transmission unit
When according to transmission, according to the data tree and preset data transmission structure, first data are handled, to turn
It is changed to the second data, comprising:
The data transmission unit receives first data that the data producer is sent；
The data transmission unit calls the Date Conversion Unit, to handle first data；
The Date Conversion Unit obtains the data tree from the data structure storage unit；
According to the data tree and the data transmission structure, the Date Conversion Unit will be at first data
Manage into the binary array being made of Head Section, structural area, undefined field area and data field；
The binary array is sent to the data transmission unit by the Date Conversion Unit；
Wherein, the Head Section is used to define the position of the structural area, the undefined field area and the data field, described
Structural area includes the data structure information of the transmission, and the undefined field area includes undefined in the data consumption side
Field information, the data field include the content information of the data of the transmission.
4. the method according to claim 1, wherein receiving the transmission data in the data consumption side
When, according to the data acquisition demand and the data transmission structure of the data consumption side, obtain needed for the data consumption side
Target data, comprising:
The data consumption side receives the transmission data that the data transmission unit is sent；
The Date Conversion Unit calls in the data consumption side, to handle the transmission data；
The Date Conversion Unit obtains the data acquisition demand from the data structure storage unit；
The Date Conversion Unit is mentioned from the transmission data according to the data transmission structure and the data acquisition demand
Take out the target data for meeting the data acquisition demand；
The target data is sent to the data consumption side by the Date Conversion Unit；
Wherein, the data acquisition demand is used to illustrate the data requirements of the data consumption side.
5. the method according to claim 1, wherein the method also includes:
When the data decryptor unit monitors that the data structure definition unit updates the data tree and/or described
When data acquisition demand, the updated data tree of correspondence and/or the data acquisition demand are stored in described
In data surplus storage unit.
6. a kind of data transmission device, which is characterized in that be applied to data processing system, the data processing system includes data
Producer, Date Conversion Unit, data transmission unit, data consumption side, data structure definition unit, data structure storage unit
And data decryptor unit, described device include:
Data Integration module is used for when the data producer generates data, according to right in the data structure storage unit
The data tree answered generates first data with the data tree；
Data processing module is used for when the data transmission unit carries out the first data transmission, according to the data tree
Shape structure and preset data transmission structure handle first data, to be converted to the second data；
Data transmission blocks are sent to the data for second data as transmission data for the data transmission unit
Consumer；
Data acquisition module, for when the data consumption side receives the transmission data, according to the data consumption side
Data acquisition demand and the data transmission structure, target data needed for obtaining the data consumption side.
7. device according to claim 6, which is characterized in that the Data Integration module, comprising:
Structure determination submodule is used for the data according to caused by the data producer, determines in the data structure
Corresponding data tree in storage unit；
Data generate submodule, for generating first data with the data tree；
Wherein, the data tree includes intermediate node and leaf node, and the intermediate node includes common intermediate node
With array type intermediate node；The leaf node is the bottom child node of the data tree and the leaf node includes
Common leaf node, external field leaf node and reference leaf node.
8. device according to claim 6, which is characterized in that the data processing module, comprising:
Data receiver submodule receives first data that the data producer is sent for the data transmission unit；
Data processing submodule calls the Date Conversion Unit for the data transmission unit, to handle first number
According to；
Structure acquisition submodule obtains the data tree from the data structure storage unit for the Date Conversion Unit
Shape structure；
Data conversion submodule, for according to the data tree and the data transmission structure, the data conversion
First data are processed into the binary array being made of Head Section, structural area, undefined field area and data field by unit；
Data sending submodule is sent to for the Date Conversion Unit using the binary array as second data
The data transmission unit；
Wherein, the Head Section is used to define the position of the structural area, the undefined field area and the data field, described
Structural area includes the data structure information of the transmission, and the undefined field area includes undefined in the data consumption side
Field information, the data field include the content information of the data of the transmission.
9. device according to claim 6, which is characterized in that the data acquisition module, comprising:
Data receiver submodule receives the transmission data that the data transmission unit is sent for the data consumption side；
Submodule is called, the Date Conversion Unit is called for the data consumption side, to handle the transmission data；
Structure acquisition submodule obtains the data from the data structure storage unit for the Date Conversion Unit and obtains
Take demand；
Data acquisition submodule, for the Date Conversion Unit according to the data transmission structure and the data acquisition need
It asks, the target data for meeting the data acquisition demand is extracted from the transmission data；
The data sending submodule is also used to the Date Conversion Unit for the target data and is sent to the data consumption
Side；
Wherein, the data acquisition demand is used to illustrate the data requirements of the data consumption side.
10. device according to claim 6, which is characterized in that described device further include:
Update module, for monitoring that the data structure definition unit update data are tree-shaped when the data decryptor unit
When structure and/or the data acquisition demand, the updated data tree of correspondence and/or the data acquisition are needed
It asks and is stored in the data surplus storage unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811291689.2A CN109634739B (en) | 2018-10-31 | 2018-10-31 | Data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811291689.2A CN109634739B (en) | 2018-10-31 | 2018-10-31 | Data transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109634739A true CN109634739A (en) | 2019-04-16 |
| CN109634739B CN109634739B (en) | 2021-03-02 |
Family
ID=66066956
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811291689.2A Active CN109634739B (en) | 2018-10-31 | 2018-10-31 | Data transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109634739B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112162868A (en) * | 2020-08-28 | 2021-01-01 | 广州南方卫星导航仪器有限公司 | Data communication method, device and storage medium applied to cluster processing |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399904A (en) * | 2013-07-26 | 2013-11-20 | 浙江吉利汽车研究院有限公司 | Data processing method and data processing system |
| CN104869172A (en) * | 2015-06-01 | 2015-08-26 | 北京圆通慧达管理软件开发有限公司 | Data interaction method and data interaction system for plurality of applications under heterogeneous system |
| CN107818120A (en) * | 2016-09-14 | 2018-03-20 | 博雅网络游戏开发（深圳）有限公司 | Data processing method and device based on big data |
-
2018
- 2018-10-31 CN CN201811291689.2A patent/CN109634739B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399904A (en) * | 2013-07-26 | 2013-11-20 | 浙江吉利汽车研究院有限公司 | Data processing method and data processing system |
| CN104869172A (en) * | 2015-06-01 | 2015-08-26 | 北京圆通慧达管理软件开发有限公司 | Data interaction method and data interaction system for plurality of applications under heterogeneous system |
| CN107818120A (en) * | 2016-09-14 | 2018-03-20 | 博雅网络游戏开发（深圳）有限公司 | Data processing method and device based on big data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112162868A (en) * | 2020-08-28 | 2021-01-01 | 广州南方卫星导航仪器有限公司 | Data communication method, device and storage medium applied to cluster processing |
| CN112162868B (en) * | 2020-08-28 | 2022-05-13 | 广州南方卫星导航仪器有限公司 | Data communication method, device and storage medium applied to cluster processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109634739B (en) | 2021-03-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106909644B (en) | A kind of multistage tissue and indexing means towards mass remote sensing image | |
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| CN112115198B (en) | Urban remote sensing intelligent service platform | |
| US11277498B2 (en) | Method, apparatus and system for processing data | |
| CN108959352A (en) | Time-space data analysis platform and processing method based on time and Spatial Data Model | |
| US20160253364A1 (en) | System for linking diverse data systems | |
| CN109189835A (en) | The method and apparatus of the wide table of data are generated in real time | |
| Ye et al. | Development of a highly flexible mobile GIS-based system for collecting arable land quality data | |
| US11423358B2 (en) | Electric power software development platform | |
| CN111090657B (en) | Automatic updating and instant service system for vector tiles | |
| US10296499B2 (en) | Dynamic database mapping | |
| CN103095823A (en) | Object description method and object information interaction system in Internet of Things system | |
| CN105117938A (en) | E-commerce framework request data analysis method based on model view controller | |
| Ramanathan et al. | Software service architecture to access weather data using RESTful web services | |
| CN113094601A (en) | GIS cloud management method and system | |
| US10432702B1 (en) | Separated device detection architecture | |
| CN113918669A (en) | Device and method for realizing natural resource homeland space planning one-map system | |
| CN109634739A (en) | A kind of data transmission method, device | |
| Nativi et al. | Unidata’s common data model mapping to the ISO 19123 data model | |
| US11403313B2 (en) | Dynamic visualization of application and infrastructure components with layers | |
| CN109213940A (en) | Method, storage medium, equipment and system that user location calculates are realized under big data | |
| Shi et al. | Automated geoprocessing mechanism, processes and workflow for seamless online integration of geodata services and creating geoprocessing services | |
| US20220335044A1 (en) | Database query execution on multiple databases | |
| CN115795187A (en) | Resource access method, device and equipment | |
| Marsh-Hunn et al. | A Comparative Study in the Standardization of IoT Devices Using Geospatial Web Standards |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210909 Address after: 311121 room 210, building 18, No. 998, Wenyi West Road, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province Patentee after: Hangzhou Bodun Xiyan Technology Co.,Ltd. Address before: Room 704, building 18, No. 998, Wenyi West Road, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province Patentee before: TONGDUN HOLDINGS Co.,Ltd. |
|
| TR01 | Transfer of patent right |