CN107885881A - Business datum real-time report, acquisition methods, device, equipment and its storage medium - Google Patents
Business datum real-time report, acquisition methods, device, equipment and its storage medium Download PDFInfo
- Publication number
- CN107885881A CN107885881A CN201711235076.2A CN201711235076A CN107885881A CN 107885881 A CN107885881 A CN 107885881A CN 201711235076 A CN201711235076 A CN 201711235076A CN 107885881 A CN107885881 A CN 107885881A
- Authority
- CN
- China
- Prior art keywords
- data
- business datum
- service
- database
- binary log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
This application discloses business datum real-time report, acquisition methods, device, equipment and its storage medium.This method is performed by Stream Processing module, including：The binary log data of Service Database are obtained in real time；The field attribute related to business datum in real time parsing binary log data；The class file of acquisition is instantiated using field attribute, generates service data object；Data/address bus is write after service data object is serialized.The characteristics such as the scalability of streaming framework and component, disaster tolerance are utilized according to the technical scheme of the embodiment of the present application, in the case of guarantee mass data high concurrent, binary log also can be accurately resolved in real time, so as to meet a variety of demands of backend services, while reduced business datum and reported cost.
Description
Technical field
Present application relates generally to network data processing technique, more particularly to business datum real-time report, acquisition methods,
Device, equipment and its storage medium.
Background technology
With the fast development and popularization and application of computer and information technology, the scale of sector application system expands rapidly,
Data caused by sector application are in explosive increase.Easily hundreds of TB even industry/enterprise of tens of to hundreds of PB scales are reached
Sparetime university's data are far beyond the disposal ability of traditional computing technique and information system.
Big data process field, the front end of big data platform must complete Data Collection, and common business datum, which is collected, to be had
The following two kinds method：
First, there is provided the SDK (Software Development Kit, SDK) that business datum reports,
SDK is called in operation system by business side, embedding data reports API to carry out real time data active push and reported；
Second, there is provided metadata acquisition tool, such as：Logstash, flume etc., passively capturing service system business data
Report.
For first method, although real time data acquisition is high, the embedding data in operation system is needed to report
Correlative code logic, cause business invasive relatively strong, and also promote constant.
For second method, although business is invasive smaller, real time data acquisition is poor, and for gathering
Business data object is the situation of journal file, and the possibility of loss of data is larger.
Urgently propose a kind of to solve the above problems in the scheme of big data field real-time report business datum.
The content of the invention
In view of drawbacks described above of the prior art or deficiency, it is expected that provide one kind performs business datum by Stream Processing module
The scheme of real-time report.
In a first aspect, the embodiment of the present application provides a kind of method of business datum real-time report, this method is by streaming
Module is managed to perform, including：
The binary log data of Service Database are obtained in real time；
The field attribute related to business datum in real time parsing binary log data；
The class file of acquisition is instantiated using field attribute, generates service data object；
Data/address bus is write after service data object is serialized ...
Second aspect, the embodiment of the present application provide a kind of device of business datum real-time report, and the device includes：
Log acquisition unit, for obtaining the binary log data of Service Database in real time；
Daily record resolution unit, for field attribute related to business datum in real time parsing binary log data；
Class instantiating unit, for the class file of acquisition being instantiated using field attribute, generates service data object
Data cell is write, data/address bus is write after being serialized for service data object
The third aspect, the embodiment of the present application provide a kind of method that business datum obtains in real time, and this method includes：
The service data object obtained such as the method for the embodiment of the present application offer is subscribed to from data/address bus；
Service data object progress unserializing is handled to obtain business datum.
Fourth aspect, the embodiment of the present application provide the device that a kind of business datum obtains in real time, and the device includes：
Subscriber units, for subscribing to the business datum pair obtained such as the method for the embodiment of the present application offer from data/address bus
As,
Acquiring unit, for handling to obtain business datum service data object progress unserializing.
5th aspect, the embodiment of the present application provide a kind of computer equipment, and the equipment includes：
One or more processors；
Storage device, for storing one or more programs, when one or more programs are by foregoing one or more
Manage device to perform so that foregoing one or more processors realize the method provided such as the embodiment of the present application.
6th aspect, the embodiment of the present application provide a kind of computer-readable recording medium, are stored thereon with computer journey
Sequence, the method provided such as the embodiment of the present application is realized when the computer program is executed by processor.
The technical scheme of the embodiment of the present application, obtain the binary system day of Service Database in real time by Stream Processing module
Will, the problem of big data business system mass data reports is can solve the problem that, and by parsing binary log file association attributes,
Then association attributes is assigned to pre-defined template file, realize the instantiation of class file, so as to generate business datum pair
As providing abundant service resources for rear end, meeting a variety of demands of backend services, while reduce business datum and report into
This.
Brief description of the drawings
By reading the detailed description made to non-limiting example made with reference to the following drawings, the application's is other
Feature, objects and advantages will become more apparent upon：
Fig. 1 shows the schematic flow sheet of the business report method of the embodiment of the present application.
Fig. 2 shows the structural representation of the business reporting device of the embodiment of the present application.
Fig. 3 shows the schematic flow sheet of the business obtaining method of the embodiment of the present application.
Fig. 4 shows the structural representation of the business acquisition device of the embodiment of the present application.
Fig. 5 shows the structural representation of the computer system suitable for being used for the terminal device for realizing the embodiment of the present application.
Embodiment
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to
It is easy to describe, the part related to invention is illustrate only in accompanying drawing.
It should be noted that in the case where not conflicting, the feature in embodiment and embodiment in the application can phase
Mutually combination.Describe the application in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 is refer to, Fig. 1 shows the schematic flow sheet of the business report method of the embodiment of the present application.
As shown in figure 1, this method is performed by Stream Processing module, it includes：
Step S1, the binary log data of Service Database are obtained in real time.
The embodiment of the present application, Stream Processing framework (i.e. Stream Processing module), it belongs to the stream processing system that main source is increased income,
Such as Storm frameworks.The framework generally includes a host node (Nimbus background programs), and a group working node
(Supervisors), these nodes are coordinated by zookeeper.Storm frameworks are true using upstream data backup and message
The mechanism recognized ensures that message can be handled again after failure.Wherein, message authentication principle：Each operation can it is preceding once
Operation processing message confirmation return.When the processing confirmation of all data records receives, backup i.e. can be by safety
Remove.After failure, received if not all Message Processing confirmations, that data record can be replaced by data source data.
The integrality of data is ensured by this message acknowledgement mechanism, so as to realize that no data is lost.
It can be used for consumption and the business datum resource handled, it is necessary to big data platform to be provided to rear end big data platform
Front end can real-time and accurately obtain the relevant information of business datum.For example, obtain the binarylog (two of Service Database
System daily record) data.Can by checking daily record data, acquisition with operation behavior relevant field attribute, such as the shape of operation
The record of state, operation behavior, and the information of database that operation behavior is directed to, pass through the acquisition of these information so that rear end
Big data platform realizes synchronization, statistics and databases comparison monitoring of business datum etc. according to different business demands.
The embodiment of the present application, it can simulate subordinate Service Database by Stream Processing module, establish subordinate business datum
Communication connection between storehouse and Service Database, backup request is sent to Service Database, and receiving Service Database,
After receiving backup request, the binary log data of push.
The embodiment of the present application, it is not necessary to which embedded any data report logic in operation system, and data backup is not yet
The running status of Service Database is relied on, and accomplishes that noninvasive data report, obtaining, which reduces business datum, reports cost
Effect.
Step S2, the field attribute related to business datum in real time parsing binary log data.
During binary log data are obtained, it is also necessary to the extraction field attribute related to business datum, with full
The miscellaneous service demand of sufficient rear end big data platform.The field attribute includes one or more in herein below：Various data
Storehouse operation title, action type, data field before changing after value, database and its databases storage tables of data,
The information such as SQL statement.Wherein, the action type of data includes data manipulation language DML, the behaviour such as it includes insertion, renewal, deleted
Make；DDL DDL, it is included for creating the various objects in database, for example, insertion, renewal, delete tables of data,
View, index, synonym, cluster.DCL DCL, for authorizing or reclaiming certain privilege for accessing database,
And time and the effect of stack room operation affairs generation are controlled, and realized by the language to database monitoring etc..
Pass through the parsing of binary log file, it is possible to achieve Disaster Data is recovered, and can also realize data syn-chronization.And
Data of the binary log comprising all renewals or all sentences of the data of potential renewal, the sentence is with " event "
Form preserves, and it describes data change.
Therefore, parsing binary log file can obtain with business datum relevant information, it has taken into full account consumption number
According to multiple business demand, by many kinds of parameters limit in a template so that backend services system can obtain more fully business
Data message.
Step S3, the class file of acquisition is instantiated using field attribute, generates service data object.
In the embodiment of the present application, by parsing obtain operational attribute related to business datum in binary log data it
Afterwards, operational attribute is assigned to class file, so as to create service data object, to be supplied to data/address bus to be consumed to rear end.
In the embodiment of the present application, the acquisition of class file can automatically generate according to template, for example, by across speech communication platform
According to the class file of pre-defined template generation.Wherein, across speech communication platform, i.e., across speech communication framework, such as
Protocol buffer, Apache Avro etc..Protocol buffer are a kind of mechanism of serializing and structural data,
With cross-platform, resolution speed is fast, serialized data small volume, autgmentability are high, using it is simple the characteristics of.Apache Avro are
One binary Data Serialization system.Avro is the sub-project for belonging to Hadoop, designed for supporting high-volume number
According to the application of exchange, rely on template (Schema) to realize data structure definition, pattern is represented by JSON objects, Avro also by
Used as a kind of RPC frameworks.Because Avro frameworks can quickly realize RPC, synchronization and asynchronous communication are supported, supports dynamic
The advantages that message, can be as a preferred embodiment of the embodiment of the present application.
For example, in Avro frameworks, the data structure template of binary log data is first defined, then Avro frameworks are according to this
Data structure template can automatically generate class file, and then, stream process framework is mainly responsible for the operation category of binary log data
Property is assigned to the corresponding field of class file, so as to create service data object.
Wherein, the operational attribute of binary log data, including at least with the next item down：Wrapped in database-name, database
The data table name that contains, the data field of tables of data before changing after value, SQL statement, database manipulation type, the behaviour of database
The IP address of server, port where making behavior and database.
For example, the structured file that arvo frameworks define：
Defined in the template of structured file various database manipulations title (insert, update, delete,
Create, drop etc.), database manipulation type (DML, DCL, DDL), and data field before changing after value, database, number
According to information such as table, SQL statements.
Step S4, data/address bus is write after service data object is serialized.
After service data object is obtained, it is only necessary to serializing, the inside of Stream Processing module are stated in configuration file
Serializing can be realized automatically, so as to which data object is write into data/address bus.
The embodiment of the present application, based on streaming framework real time parsing binary log data, ensure the real-time of data processing,
By the disaster tolerance mechanism of streaming framework, ack mechanism, preserve processing daily record offset etc. in real time with reference to zookeeper, ensure data
It can obtain not losing during processing and not repeat accurately and reliably effect.
It should be noted that although describing the operation of the application method with particular order in the accompanying drawings, still, this is not required that
Or imply and must perform these operations according to the particular order, or the operation having to carry out shown in whole could realize the phase
The result of prestige.On the contrary, the step of describing in flow chart can change execution sequence.Additionally or alternatively, it is convenient to omit some
Step, multiple steps are merged into a step and performed, and/or a step is decomposed into execution of multiple steps.
Fig. 2 is refer to, as shown in Fig. 2 the embodiment of the present application provides a kind of device 200 of business datum real-time report,
The device 200 includes
Log acquisition unit, for obtaining the binary log data of Service Database in real time；
The embodiment of the present application, Stream Processing framework (i.e. Stream Processing module), it belongs to the stream processing system that main source is increased income,
Such as Storm frameworks.The framework generally includes a host node (Nimbus background programs), and a group working node
(Supervisors), these nodes are coordinated by zookeeper.Storm frameworks are true using upstream data backup and message
The mechanism recognized ensures that message can be handled again after failure.Wherein, message authentication principle：Each operation can it is preceding once
Operation processing message confirmation return.When the processing confirmation of all data records receives, backup i.e. can be by safety
Remove.After failure, received if not all Message Processing confirmations, that data record can be replaced by data source data.
The integrality of data is ensured by this message acknowledgement mechanism, so as to realize that no data is lost.
It can be used for consumption and the business datum resource handled, it is necessary to big data platform to be provided to rear end big data platform
Front end can real-time and accurately obtain the relevant information of business datum.For example, obtain the binarylog (two of Service Database
System daily record) data.Can by checking daily record data, acquisition with operation behavior relevant field attribute, such as the shape of operation
The record of state, operation behavior, and the information of database that operation behavior is directed to, pass through the acquisition of these information so that rear end
Big data platform realizes synchronization, statistics and databases comparison monitoring of business datum etc. according to different business demands.
The embodiment of the present application, log acquisition unit can also include：
Send request unit, for simulating subordinate Service Database, establish subordinate Service Database and Service Database it
Between communication connection, to Service Database send backup request；
Log unit is received, for receiving the binary log data of Service Database push.
The embodiment of the present application, it is not necessary to which embedded any data report logic in operation system, and data backup is not yet
The running status of Service Database is relied on, and accomplishes that noninvasive data report, obtaining, which reduces business datum, reports cost
Effect.
Daily record resolution unit, for field attribute related to business datum in real time parsing binary log data.
During binary log data are obtained, it is also necessary to the extraction content related to business datum, after satisfaction
Continuous miscellaneous service demand.For example, the field attribute of extraction includes title, the action type of various database manipulations, data field
The information such as tables of data, SQL statement of value after before changing, database and its databases storage.Wherein, the operation class of data
Type includes data manipulation language DML, the operations such as it includes insertion, renewal, deleted；DDL DDL, it is included for creating
The various objects built in database, such as insertion, renewal, delete tables of data, view, index, synonym, cluster.Data control
Language DCL processed, for authorize or reclaim access database certain privilege, and control stack room operation affairs occur time and
Effect, and realized by the language to database monitoring etc..
Pass through the parsing of binary log file, it is possible to achieve Disaster Data is recovered, and can also realize data syn-chronization.Two enter
Data of the daily record processed comprising all renewals or all sentences of the data of potential renewal, the sentence is in the form of " event "
Preserve, it describes data change.
Therefore, parsing binary log file can obtain with business datum relevant information, it has taken into full account consumption number
According to multiple business demand, by many kinds of parameters limit in a template so that backend services system can obtain more fully business
Data message.
Class instantiating unit, for the class file of acquisition being instantiated using field attribute, generates service data object.
In the embodiment of the present application, by parsing obtain operational attribute related to business datum in binary log data it
Afterwards, operational attribute is assigned to class file, so as to create service data object, to be supplied to data/address bus to be consumed to rear end.
In the embodiment of the present application, the acquisition of class file can automatically generate according to template, for example, by across speech communication platform
According to the class file of pre-defined template generation.Wherein, across speech communication platform, i.e., across speech communication framework, such as
Protocol buffer, Apache Avro etc..Protocol buffer are a kind of mechanism of serializing and structural data,
With cross-platform, resolution speed is fast, serialized data small volume, autgmentability are high, using it is simple the characteristics of.Apache Avro are
One binary Data Serialization system.Avro is the sub-project for belonging to Hadoop, designed for supporting high-volume number
According to the application of exchange, rely on template (Schema) to realize data structure definition, pattern is represented by JSON objects, Avro also by
Used as a kind of RPC frameworks.Because Avro frameworks can quickly realize RPC, synchronization and asynchronous communication are supported, supports dynamic
The advantages that message, can be as a preferred embodiment of the embodiment of the present application.
For example, in Avro frameworks, the data structure template of binary log data is first defined, then Avro frameworks are according to this
Data structure template can automatically generate class file, and then, stream process framework is mainly responsible for the operation category of binary log data
Property is assigned to the corresponding field of class file, so as to create service data object.
Wherein, the operational attribute of binary log data, including at least with the next item down：Wrapped in database-name, database
The data table name that contains, the data field of tables of data before changing after value, SQL statement, database manipulation type, the behaviour of database
The IP address of server, port where making behavior and database.
For example, the structured file that arvo frameworks define：
Defined in the template of structured file various database manipulations title (insert, update, delete,
Create, drop etc.), database manipulation type (DML, DCL, DDL), and data field before changing after value, database, number
According to information such as table, SQL statements.
In the embodiment of the present application, such instantiating unit can also include class file acquisition subelement and generation object is single
Member：
Class file acquiring unit, for obtaining the class text by across speech communication platform according to pre-defined template generation
Part, template are the data structure template according to binary log data definition by across speech communication platform；
Object unit is generated, for field attribute to be assigned into corresponding field in class file, generates business datum
Object.
Data cell is write, data/address bus is write after being serialized for service data object.
After service data object is obtained, it is only necessary to serializing, the inside of Stream Processing module are stated in configuration file
Serializing can be realized automatically, so as to which data object is write into data/address bus.
The embodiment of the present application, based on streaming framework real time parsing binary log data, ensure the real-time of data processing,
By the disaster tolerance mechanism of streaming framework, ack mechanism, preserve processing daily record offset etc. in real time with reference to zookeeper, ensure data
It can obtain not losing during processing and not repeat accurately and reliably effect.
It should be appreciated that all units or module described in device 200 and each step phase in the method described with reference to figure 1
It is corresponding.Thus, the unit that the operation above with respect to method description and feature are equally applicable to device 200 and wherein included, herein
Repeat no more.Device 200 can be realized in the browser of electronic equipment or other safety applications in advance, can also pass through download
It is loaded into etc. mode in browser or its safety applications of electronic equipment.Corresponding units in device 200 can be set with electronics
Unit in standby cooperates to realize the scheme of the embodiment of the present application.
Fig. 3 is refer to, as shown in figure 3, the embodiment of the present application additionally provides a kind of method that business datum obtains in real time, should
Method includes the service data object subscribed to the embodiment of the present application as shown in Figure 1 from data/address bus and reported.
After service data object is obtained, carried out unserializing and handle to obtain business datum.
Fig. 4 is refer to, as shown in figure 4, the embodiment of the present application additionally provides the device that a kind of business datum obtains in real time
400, the device 400 includes：
Subscriber units 401, for including subscribing to the embodiment of the present application reporting service data as shown in Figure 1 from data/address bus
Method, the service data object reported.
Acquiring unit 402, handle to obtain business number for unserializing after service data object is obtained, will to be carried out the phase
According to.
Below with reference to Fig. 5, it illustrates suitable for for realizing the structural representation of the computer equipment 500 of the embodiment of the present application
Figure.
As shown in figure 5, computer system 500 includes CPU (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into program in random access storage device (RAM) 503 from storage part 508 and
Perform various appropriate actions and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data.
CPU 501, ROM 502 and RAM 503 are connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to always
Line 504.
I/O interfaces 505 are connected to lower component：Importation 506 including keyboard, mouse etc.；Penetrated including such as negative electrode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage part 508 including hard disk etc.；
And the communications portion 509 of the NIC including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net performs communication process.Driver 510 is also according to needing to be connected to I/O interfaces 505.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc., it is arranged on as needed on driver 510, in order to read from it
Computer program be mounted into as needed storage part 508.
Especially, according to embodiments herein, computer software is may be implemented as above with reference to the process of Fig. 1 descriptions
Program.For example, embodiments herein includes a kind of computer program product, it includes being tangibly embodied in machine readable media
On computer program, the computer program include be used for perform Fig. 1 method program code.In such embodiments,
The computer program can be downloaded and installed by communications portion 509 from network, and/or be pacified from detachable media 511
Dress.
Flow chart and block diagram in accompanying drawing, it is illustrated that according to the system of various embodiments of the invention, method and computer journey
Architectural framework in the cards, function and the operation of sequence product.At this point, each square frame in flow chart or block diagram can generation
The part of one module of table, program segment or code, wherein, a part for module, program segment or code includes one or more
The individual executable instruction for being used to realize defined logic function.It should also be noted that some as replace realization in, in square frame
The function of being marked can also be with different from the order marked in accompanying drawing generation.For example, two square frames succeedingly represented are real
It can be performed substantially in parallel on border, they can also be performed in the opposite order sometimes, and this is depending on involved function.
, can be with it should be noted that the combination of each square frame and block diagram in block diagram and/or flow chart and/or the square frame in flow chart
Function or the special hardware based system of operation as defined in execution are realized, or can use specialized hardware and computer
The combination of instruction is realized.
Being described in unit or module involved in the embodiment of the present application can be realized by way of software, can also
Realized by way of hardware.Described unit or module can also be set within a processor, for example, can be described as：
A kind of processor includes log acquisition unit, sends request unit and receives log unit.Wherein, these units or module
Title does not form the restriction to the unit or module in itself under certain conditions, for example, log acquisition unit can also be retouched
State as " being used for the unit for obtaining daily record ".
As on the other hand, present invention also provides a kind of computer-readable recording medium, the computer-readable storage medium
Matter can be the computer-readable recording medium included in the system of above-described embodiment；Can also be individualism, it is unassembled
Enter the computer-readable recording medium in equipment.Computer-readable recording medium storage has one or more than one program, should
Program is used for performing the business datum report method for being described in the application by one or more than one processor.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.People in the art
Member should be appreciated that invention scope involved in the application, however it is not limited to the technology that the particular combination of above-mentioned technical characteristic forms
Scheme, while should also cover in the case where not departing from identical inventive concept, carried out by above-mentioned technical characteristic or its equivalent feature
The other technical schemes for being combined and being formed.Such as features described above has similar work(with (but not limited to) disclosed herein
The technical scheme that the technical characteristic of energy is replaced mutually and formed.
Claims (13)
- A kind of 1. method of business datum real-time report, it is characterised in that this method is performed by Stream Processing module, and it includes：The binary log data of Service Database are obtained in real time；The field attribute related to business datum in binary log data described in real time parsing；The class file of acquisition is instantiated using the field attribute, generates service data object；Data/address bus is write after the service data object is serialized.
- 2. according to the method for claim 1, it is characterised in that described using the field attribute that the class file of acquisition is real Exampleization, service data object is generated, including：The class file by across speech communication platform according to pre-defined template generation is obtained, the template is across language by described Communications platform according to the binary log data definition data structure template；AndThe field attribute is assigned to corresponding field in the class file, generates the service data object.
- 3. according to the method for claim 2, it is characterised in that across the speech communication platform is Data Serialization system Apache Avro。
- 4. according to the method described in claim any one of 1-3, it is characterised in that the field attribute is comprised at least with next ：The data table name that is included in database-name, database, the data field of tables of data before changing after value, SQL statement, Database manipulation type, the IP address of the operation behavior of database and server where database, port.
- 5. according to the method any one of claim 1-4, it is characterised in that the Stream Processing module is Storm journeys Sequence.
- 6. method as claimed in claim 5, it is characterised in that the binary log number for obtaining Service Database in real time According to, including,Simulation subordinate Service Database, the communication connection established between the subordinate Service Database and the Service Database, Backup request is sent to the Service Database；Receive the binary log data of the Service Database push.
- 7. a kind of device of business datum real-time report, it is characterised in that described device includes：Log acquisition unit, for obtaining the binary log data of Service Database in real time；Daily record resolution unit, for field attribute related to business datum in binary log data described in real time parsing；Class instantiating unit, for the class file of acquisition being instantiated using the field attribute, generates service data objectData cell is write, data/address bus is write after being serialized for the service data object.
- 8. device according to claim 7, it is characterised in that the class instantiating unit, in addition to：Class file acquiring unit, for obtaining the class file by across speech communication platform according to pre-defined template generation, institute It is the data structure template by across the speech communication platform according to the binary log data definition to state template；Object unit is generated, for the field attribute to be assigned into corresponding field in the class file, described in generation Service data object.
- 9. device as claimed in claim 7 or 8, it is characterised in that the log acquisition unit, including,Request unit is sent, for simulating subordinate Service Database, establishes the subordinate Service Database and the business datum Communication connection between storehouse, backup request is sent to the Service Database；Log unit is received, for receiving the binary log data of the Service Database push.
- 10. a kind of method that business datum obtains in real time, it is characterised in that this method includes：The service data object obtained from method of the data/address bus subscription as described in claim 1-6 is any；Service data object progress unserializing is handled to obtain business datum.
- 11. the device that a kind of business datum obtains in real time, it is characterised in that the device includes：Subscriber units, the business datum obtained for subscribing to the method as described in claim 1-6 is any from data/address bus Object,Acquiring unit, for handling to obtain business datum service data object progress unserializing.
- 12. a kind of computer equipment, its feature is being, the equipment includes：One or more processors；Storage device, for storing one or more programs, when one or more of programs are by one or more of processing Device performs so that one or more of processors realize the method as described in claim 1-6 is any.
- 13. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the computer program The method as described in claim 1-6 is any is realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711235076.2A CN107885881A (en) | 2017-11-29 | 2017-11-29 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711235076.2A CN107885881A (en) | 2017-11-29 | 2017-11-29 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107885881A true CN107885881A (en) | 2018-04-06 |
Family
ID=61776035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711235076.2A Pending CN107885881A (en) | 2017-11-29 | 2017-11-29 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107885881A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108874964A (en) * | 2018-06-07 | 2018-11-23 | 火烈鸟网络(广州)股份有限公司 | A kind of method and system in monitoring data library |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN109359139A (en) * | 2018-10-24 | 2019-02-19 | 拉扎斯网络科技（上海）有限公司 | Method of data synchronization, system, electronic equipment and computer readable storage medium |
| CN110489469A (en) * | 2019-07-12 | 2019-11-22 | 中国平安人寿保险股份有限公司 | Data transmission method, device, computer equipment and storage medium |
| CN110502566A (en) * | 2019-08-29 | 2019-11-26 | 江苏满运软件科技有限公司 | Near real-time data acquisition method, device, electronic equipment, storage medium |
| CN110691112A (en) * | 2018-07-06 | 2020-01-14 | 优信拍（北京）信息科技有限公司 | Data synchronization method, device and system |
| CN111046050A (en) * | 2018-10-12 | 2020-04-21 | 顺丰科技有限公司 | Data Schema processing method, device, equipment and storage medium |
| CN111563123A (en) * | 2020-05-07 | 2020-08-21 | 北京首汽智行科技有限公司 | Live warehouse metadata real-time synchronization method |
| CN111899087A (en) * | 2020-06-16 | 2020-11-06 | 中国建设银行股份有限公司 | Data providing method and device, electronic equipment and computer readable storage medium |
| CN111988373A (en) * | 2020-07-31 | 2020-11-24 | 广州市百果园信息技术有限公司 | Data processing method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105574082A (en) * | 2015-12-08 | 2016-05-11 | 曙光信息产业(北京)有限公司 | Storm based stream processing method and system |
| CN105824618A (en) * | 2016-03-10 | 2016-08-03 | 浪潮软件集团有限公司 | Real-time message processing method for Storm |
| CN106339274A (en) * | 2015-07-14 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Method and system for obtaining data snapshot |
| CN106446092A (en) * | 2016-09-12 | 2017-02-22 | 浪潮软件股份有限公司 | Flume-based method for analyzing data of semi-structured text file |
| CN106713332A (en) * | 2016-12-30 | 2017-05-24 | 山石网科通信技术有限公司 | Network data processing method, device and system |
| US20170337246A1 (en) * | 2016-05-20 | 2017-11-23 | WASAI technology inc. | Big-data processing accelerator and big-data processing system thereof |
-
2017
- 2017-11-29 CN CN201711235076.2A patent/CN107885881A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106339274A (en) * | 2015-07-14 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Method and system for obtaining data snapshot |
| CN105574082A (en) * | 2015-12-08 | 2016-05-11 | 曙光信息产业(北京)有限公司 | Storm based stream processing method and system |
| CN105824618A (en) * | 2016-03-10 | 2016-08-03 | 浪潮软件集团有限公司 | Real-time message processing method for Storm |
| US20170337246A1 (en) * | 2016-05-20 | 2017-11-23 | WASAI technology inc. | Big-data processing accelerator and big-data processing system thereof |
| CN106446092A (en) * | 2016-09-12 | 2017-02-22 | 浪潮软件股份有限公司 | Flume-based method for analyzing data of semi-structured text file |
| CN106713332A (en) * | 2016-12-30 | 2017-05-24 | 山石网科通信技术有限公司 | Network data processing method, device and system |
Non-Patent Citations (2)
| Title |
|---|
| HAITAOYAO: ""MySQL binlog 增量数据解析服务"", 《HTTPS://WWW.JIANSHU.COM/P/BE3F62D4DCE0》 * |
| KIMIBOB: ""AVRO"", 《HTTPS://WWW.JIANSHU.COM/P/ECBB607809C4》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108874964A (en) * | 2018-06-07 | 2018-11-23 | 火烈鸟网络(广州)股份有限公司 | A kind of method and system in monitoring data library |
| CN110691112A (en) * | 2018-07-06 | 2020-01-14 | 优信拍（北京）信息科技有限公司 | Data synchronization method, device and system |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN109257337B (en) * | 2018-08-28 | 2021-09-07 | 科大国创软件股份有限公司 | Internet of things equipment protocol conversion method and lamp pole operation platform based on same |
| CN111046050A (en) * | 2018-10-12 | 2020-04-21 | 顺丰科技有限公司 | Data Schema processing method, device, equipment and storage medium |
| CN109359139A (en) * | 2018-10-24 | 2019-02-19 | 拉扎斯网络科技（上海）有限公司 | Method of data synchronization, system, electronic equipment and computer readable storage medium |
| CN110489469A (en) * | 2019-07-12 | 2019-11-22 | 中国平安人寿保险股份有限公司 | Data transmission method, device, computer equipment and storage medium |
| CN110489469B (en) * | 2019-07-12 | 2023-12-08 | 中国平安人寿保险股份有限公司 | Data transmission method, device, computer equipment and storage medium |
| CN110502566A (en) * | 2019-08-29 | 2019-11-26 | 江苏满运软件科技有限公司 | Near real-time data acquisition method, device, electronic equipment, storage medium |
| CN110502566B (en) * | 2019-08-29 | 2022-09-09 | 江苏满运软件科技有限公司 | Near real-time data acquisition method and device, electronic equipment and storage medium |
| CN111563123A (en) * | 2020-05-07 | 2020-08-21 | 北京首汽智行科技有限公司 | Live warehouse metadata real-time synchronization method |
| CN111563123B (en) * | 2020-05-07 | 2023-08-22 | 北京首汽智行科技有限公司 | Real-time synchronization method for hive warehouse metadata |
| CN111899087A (en) * | 2020-06-16 | 2020-11-06 | 中国建设银行股份有限公司 | Data providing method and device, electronic equipment and computer readable storage medium |
| CN111988373A (en) * | 2020-07-31 | 2020-11-24 | 广州市百果园信息技术有限公司 | Data processing method and device |
| CN111988373B (en) * | 2020-07-31 | 2024-03-19 | 广州市百果园信息技术有限公司 | Data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107885881A (en) | Business datum real-time report, acquisition methods, device, equipment and its storage medium | |
| CN106096056B (en) | One kind being based on distributed public sentiment data real-time collecting method and system | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| US20150032725A1 (en) | Systems and methods for efficient data ingestion and query processing | |
| US11941068B2 (en) | Case leaf nodes pointing to business objects or document types | |
| CN105224631A (en) | Build the system of the open cloud of industry and the method for establishment XBRL financial statement | |
| WO2021184761A1 (en) | Data access method and apparatus, and data storage method and device | |
| CN104657497A (en) | Mass electricity information concurrent computation system and method based on distributed computation | |
| CN113590576A (en) | Database parameter adjusting method and device, storage medium and electronic equipment | |
| CN108269056A (en) | Government information resources manage system | |
| CN114722119A (en) | Data synchronization method and system | |
| CN106528169A (en) | Web system development reusable method based on AnGo dynamic evolution model | |
| US20210191921A1 (en) | Method, apparatus, device and storage medium for data aggregation | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| CN111427869A (en) | Log system based on block chain | |
| US9189486B2 (en) | Autonomic generation of document structure in a content management system | |
| US10620970B2 (en) | Request processing by a runtime agent in a network system | |
| CN115168474B (en) | Internet of things central station system building method based on big data model | |
| CN106502842A (en) | Data reconstruction method and system | |
| CN116483707A (en) | Test method, test device, test apparatus, test program, and test program | |
| CN115757304A (en) | Log storage method, device and system, electronic equipment and storage medium | |
| CN112100187B (en) | Student learning data storage method and device based on VueJS | |
| WO2019153546A1 (en) | Ten-thousand-level dimension data generation method, apparatus and device, and storage medium | |
| CN114996246A (en) | Data cleaning method for checking multiple fields of table based on NiFi assembly | |
| Rezazadeh et al. | Some guidelines for formal development of web-based applications in b-method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180406 |
|
| RJ01 | Rejection of invention patent application after publication |