CN110083616A - Page data processing method, device, equipment and computer readable storage medium - Google Patents
Page data processing method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110083616A CN110083616A CN201910321415.1A CN201910321415A CN110083616A CN 110083616 A CN110083616 A CN 110083616A CN 201910321415 A CN201910321415 A CN 201910321415A CN 110083616 A CN110083616 A CN 110083616A
- Authority
- CN
- China
- Prior art keywords
- page
- journal file
- preset
- page data
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 35
- 230000009471 action Effects 0.000 claims abstract description 30
- 230000003068 static effect Effects 0.000 claims description 23
- 238000004458 analytical method Methods 0.000 claims description 18
- 238000003780 insertion Methods 0.000 claims description 16
- 230000037431 insertion Effects 0.000 claims description 16
- 230000003139 buffering effect Effects 0.000 claims description 15
- 238000012217 deletion Methods 0.000 claims description 10
- 230000037430 deletion Effects 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 7
- 238000000034 method Methods 0.000 abstract description 16
- 238000004891 communication Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 230000003111 delayed effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 101000804902 Drosophila melanogaster Xaa-Pro aminopeptidase ApepP Proteins 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 229910002056 binary alloy Inorganic materials 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of page data processing methods, this method comprises: obtaining the journal file of source database, the journal file is for recording client to the operation information of the source database；Whether judge in the journal file comprising preset action event type；If so, judging whether preset page key message changes in the journal file；If, page data is then sent to application server corresponding with the client according to the page key message of variation and updates request, so that the background server updates corresponding page data, and updated page data is uploaded to preset content distributing network and is cached.The invention also discloses a kind of page data processing unit, equipment and a kind of computer readable storage mediums.The present invention is able to ascend the loading velocity of customer terminal webpage.
Description
Technical field
The present invention relates to financial technology (Fintech) technical field more particularly to page data processing method, device, set
Standby and computer readable storage medium.
Background technique
With the development of computer technology, the especially continuous development of internet financial technology, more and more technologies are answered
Used in financial field, traditional financial industry gradually changes to financial technology (Finteh), but due to finance, the peace of payment industry
Quan Xing, requirement of real-time, the higher requirement that also technology is proposed.In financial field, Mobile banking, Investment & Financing, finance
The client end APs such as information P usually issues network to WEB server from client end AP P and visits when accessing background application server
Request is asked, to background application requesting page server face display data, application service after the packaged association requests parameter of WEB server
Device obtains data from source database, and data are calculated, process after return.
The defect of aforesaid way is: entire link will lead in the case where financial business high concurrent is requested using clothes
Business device and source database increased pressure, and then there is performance bottleneck, cause customer terminal webpage and loads slow problem.
Summary of the invention
It is a primary object of the present invention to propose a kind of page data processing method, device, equipment and computer-readable deposit
Storage media, it is intended to promote the loading velocity of customer terminal webpage.
To achieve the above object, the present invention provides a kind of page data processing method, the page data processing method packet
Include following steps:
The journal file of source database is obtained, the journal file is for recording operation of the client to the source database
Information；
Whether judge in the journal file comprising preset action event type；
If including preset action event type in the journal file, the preset page in the journal file is judged
Whether key message changes；
If preset page key message changes in the journal file, believed according to the page key of variation
It ceases to application server corresponding with the client and sends page data update request, so that background server update pair
The page data answered, and updated page data is uploaded to preset content distributing network and is cached.
Preferably, the step of journal file for obtaining source database includes:
When source database is MySQL database, socket is established with the MySQL database and is connect, and is created
Build Log backup thread；
The binary log that the MySQL database is generated according to the socket connection and the Log backup thread
File is monitored, and receives the binary log file of the MySQL database transmission.
Preferably, described to judge in the journal file that the step of whether including preset action event type includes:
Judge whether to update comprising the data insertion based on the source database, data in the journal file or data are deleted
Except event type；
If including data insertion, data update or data deletion event based on the source database in the journal file
Type then determines in the journal file comprising preset action event type.
Preferably, described to judge the step of whether preset page key message changes in journal file packet
It includes:
The journal file is sent to preset Distributed Message Queue, so that the Distributed Message Queue will be described
Journal file is forwarded to preset log analysis service；
The journal file is parsed by the log analysis service, it is preset in the journal file to judge
Whether page key message changes.
Preferably, the content distributing network is divided into static buffer and dynamic buffering area, and the static buffer is used for
Preset static page information is cached, the dynamic buffering area is for caching preset dynamic page information, the dynamic page
Information includes the page key message.
In addition, to achieve the above object, the present invention also provides a kind of page data processing unit, the page data processing
Device includes:
Module is obtained, for obtaining the journal file of source database, the journal file is for recording client to described
The operation information of source database；
First judgment module, for whether judging in the journal file comprising preset action event type；
Second judgment module, if judging the day for including preset action event type in the journal file
Whether preset page key message changes in will file；
Processing module, if changing for page key message preset in the journal file, according to variation
The page key message sends page data to application server corresponding with the client and updates request, so that after described
The corresponding page data of platform server update, and updated page data is uploaded to preset content distributing network and is delayed
It deposits.
Preferably, the acquisition module is also used to:
When source database is MySQL database, socket is established with the MySQL database and is connect, and is created
Build Log backup thread；
The binary log that the MySQL database is generated according to the socket connection and the Log backup thread
File is monitored, and receives the binary log file of the MySQL database transmission.
Preferably, the first judgment module is also used to:
Judge whether to update comprising the data insertion based on the source database, data in the journal file or data are deleted
Except event type；
If including data insertion, data update or data deletion event based on the source database in the journal file
Type then determines in the journal file comprising preset action event type.
Preferably, second judgment module is also used to:
The journal file is sent to preset Distributed Message Queue, so that the Distributed Message Queue will be described
Journal file is forwarded to preset log analysis service；
The journal file is parsed by the log analysis service, it is preset in the journal file to judge
Whether page key message changes.
Preferably, the content distributing network is divided into static buffer and dynamic buffering area, and the static buffer is used for
Preset static page information is cached, the dynamic buffering area is for caching preset dynamic page information, the dynamic page
Information includes the page key message.
In addition, to achieve the above object, the present invention also provides a kind of page data processing equipment, the page data processing
Equipment includes: memory, processor and is stored on the memory and at the page data that can be run on the processor
Program is managed, the page data processing routine realizes page data processing method as described above when being executed by the processor
Step.
In addition, to achieve the above object, it is described computer-readable the present invention also provides a kind of computer readable storage medium
Page data processing routine is stored on storage medium, the page data processing routine realizes institute as above when being executed by processor
The step of page data processing method stated.
Page data processing method proposed by the present invention, by obtaining the journal file of source database, when in journal file
Comprising preset action event type and when preset page key message changes, according to the page key message after variation
Page data is sent to application server corresponding with the client and updates request, so that background server updates corresponding page
Face data, and updated page data is uploaded to preset content distributing network and is cached, it ensure that content delivery network
The page data cached in network is consistent with the record in database, thus when client sends accessing page request, Ke Yizhi
It connects and gets newest page info from the content distributing network, without visiting again application server and source database, keep away
Exempt from the case where high concurrent is requested, the problem of performance bottleneck occur in application server and source database, improve client
The loading velocity of the page.
Detailed description of the invention
Fig. 1 is the device structure schematic diagram for the hardware running environment that the embodiment of the present invention is related to；
Fig. 2 is the flow diagram of page data processing method first embodiment of the present invention；
Fig. 3 is the configuration diagram of page data processing system in the embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
As shown in Figure 1, Fig. 1 is the device structure schematic diagram for the hardware running environment that the embodiment of the present invention is related to.
Page data processing equipment of the embodiment of the present invention can be PC machine or server apparatus.
As shown in Figure 1, the page data processing equipment may include: processor 1001, such as CPU, network interface 1004,
User interface 1003, memory 1005, communication bus 1002.Wherein, communication bus 1002 is for realizing between these components
Connection communication.User interface 1003 may include display screen (Display), input unit such as keyboard (Keyboard), optional
User interface 1003 can also include standard wireline interface and wireless interface.Network interface 1004 optionally may include standard
Wireline interface, wireless interface (such as WI-FI interface).Memory 1005 can be high speed RAM memory, be also possible to stable
Memory (non-volatile memory), such as magnetic disk storage.Memory 1005 optionally can also be independently of aforementioned
The storage device of processor 1001.
It will be understood by those skilled in the art that device structure shown in Fig. 1 does not constitute the restriction to equipment, can wrap
It includes than illustrating more or fewer components, perhaps combines certain components or different component layouts.
As shown in Figure 1, as may include that operating system, network are logical in a kind of memory 1005 of computer storage medium
Believe module, Subscriber Interface Module SIM and page data processing routine.
In equipment shown in Fig. 1, network interface 1004 is mainly used for connecting background server, carries out with background server
Data communication；User interface 1003 is mainly used for connecting client (user terminal), carries out data communication with client；And processor
1001 can be used for calling the page data processing routine stored in memory 1005, and execute following operation:
The journal file of source database is obtained, the journal file is for recording operation of the client to the source database
Information；
Whether judge in the journal file comprising preset action event type；
If including preset action event type in the journal file, the preset page in the journal file is judged
Whether key message changes；
If preset page key message changes in the journal file, believed according to the page key of variation
It ceases to application server corresponding with the client and sends page data update request, so that background server update pair
The page data answered, and updated page data is uploaded to preset content distributing network and is cached.
Further, processor 1001 can call the page data processing routine stored in memory 1005, also execute
It operates below:
When source database is MySQL database, socket is established with the MySQL database and is connect, and is created
Build Log backup thread；
The binary log that the MySQL database is generated according to the socket connection and the Log backup thread
File is monitored, and receives the binary log file of the MySQL database transmission.
Further, processor 1001 can call the page data processing routine stored in memory 1005, also execute
It operates below:
Judge whether to update comprising the data insertion based on the source database, data in the journal file or data are deleted
Except event type；
If including data insertion, data update or data deletion event based on the source database in the journal file
Type then determines in the journal file comprising preset action event type.
Further, processor 1001 can call the page data processing routine stored in memory 1005, also execute
It operates below:
The journal file is sent to preset Distributed Message Queue, so that the Distributed Message Queue will be described
Journal file is forwarded to preset log analysis service；
The journal file is parsed by the log analysis service, it is preset in the journal file to judge
Whether page key message changes.
Further, the content distributing network is divided into static buffer and dynamic buffering area, and the static buffer is used
In caching preset static page information, the dynamic buffering area is for caching preset dynamic page information, the dynamic page
Face information includes the page key message.
Based on above-mentioned hardware configuration, page data processing method embodiment of the present invention is proposed.
It is the flow diagram of page data processing method first embodiment of the present invention, the method packet referring to Fig. 2, Fig. 2
It includes:
Step S10 obtains the journal file of source database, and the journal file is for recording client to the source data
The operation information in library；
It is the configuration diagram of page data processing system in the embodiment of the present invention referring to Fig. 3, Fig. 3.Implement in the present invention
In example, page data processing system includes client, content distributing network (Content Delivery Network, abbreviation
CDN), WEB server, application server, source database and page data processing equipment.Wherein, client refers to and server
It is corresponding, the program of local service is provided for client, other than some application programs in local runtime, is typically mounted at
In common client computer, operation of working in coordination with server-side, including web browser, pc client, instant message applications are needed
Deng；CDN is that the content distributing network constructed on network by the Edge Server for being deployed in various regions passes through central platform
The functional modules such as load balancing, content distribution, scheduling, content needed for obtaining user nearby reduces network congestion, improves and use
Family access response speed and hit rate；WEB server refers generally to Website server, refers to and resides at certain seed type meter on internet
The program of calculation machine can provide document to WEB client sides such as browsers, can also place site file, the whole world be allowed to browse simultaneously
Downloading；Application server is that the background server of related service is provided for WEB server, provides the request of parsing WEB terminal, data add
The service such as work；Source database, which refers to, can directly provide the self-sufficiency database of firsthand information or specific data, and user need not look into again
Read other information source.
The present embodiment page data processing method is applied to page data processing equipment, which can be with
It is PC machine or server, the page data processing equipment and source database and application server communication, operation has log to obtain thereon
It takes and log analysis service.
Firstly, page data processing equipment obtains the journal file of source database.Wherein, source database can be relationship type
Database (such as MySQL, Oracle), memory type database (such as shared drive, Redis), non-relational database is (such as
Mongodb), the database (such as Hive, Hbase) of Large data types, the present embodiment are not construed as limiting the type of source database；
For the journal file of source database for recording client to the operation information of the source database, which includes but is not limited to count
It is investigated that inquiry, data insertion, data update, data deletion etc..
In one embodiment, the step of journal file for obtaining source database may further include: when source number
According to library be MySQL database when, establish socket with the MySQL database and connect, and create Log backup thread；
The binary log file that the MySQL database generates is carried out according to the socket connection and the Log backup thread
It monitors, and receives the binary log file of the MySQL database transmission.
MySQL is a kind of Relational DBMS of open source code, uses most common data base administration language
Speech -- structured query language SQL (Structured Query Language) carries out data base administration.Mysql-binlog is
The binary log file of MySQL database, for recording user to the SQL statement of database manipulation (in addition to data query language
Sentence) information.In the present embodiment, when source database is MySQL database, for the binary log for obtaining MySQL database
File binlog, needing to establish socket first with MySQL database, (two programs on network pass through a two-way communication link
The exchange for realizing data is connect, one end of this connection is known as a socket, and socket is also known as " socket ", and application program is usual
Request or response network request are issued to network by " socket ") it connects, after socket connection is established, connected using socket
The data packet based on MySQL agreement is received and sent, to create the binlog dump thread (Log backup thread) of MySQL master library, so
Afterwards, the binary log file that the MySQL database generates is monitored according to socket connection and Log backup thread,
And receive the binary log file that MySQL database is sent.Hereby it is achieved that the binary system day of MySQL database
The acquisition of will file.
Whether step S20 judges in the journal file comprising preset action event type；
In the step, after getting the journal file of source database, whether judge in the journal file comprising preset
Action event type, the event which is used to be expressed as having the data in source database modification to operate
Type.
In one embodiment, whether step S20 may further include: judging in the journal file comprising being based on
The data insertion of the source database, data update or data deletion event type；If comprising being based on institute in the journal file
The data insertion, data update or data deletion event type for stating source database, then determine in the journal file comprising default
Action event type.
Specifically, when comprising insertEvent (data insertion event), updateEvent, (data update in journal file
Event) or when deleteEvent (data deletion event), it is possible to determine that it include preset action event class in the journal file
Type；Include other event types in addition to preset action event type when in journal file, for example only includes data and look into
When inquiry event, then illustrate that the data of the storage in source database do not change, at this time with no treatment, and keeps to source
The journal file of database is monitored.
If including preset action event type in the journal file, S30 is thened follow the steps, judges the journal file
In preset page key message whether change；
If in journal file including preset action event type, illustrate that the library table record of source database is modified,
The journal file is further parsed at this time, judges whether preset page key message becomes in the journal file
It is dynamic.Wherein, page key message can carry out flexible setting according to practical business situation, for example, to Mr. Yu's merchandise display page,
Page key message can be set by information such as price, inventories, it, can be by contact person, connection to Mr. Yu's contact person's displayed page
It is that the information such as phone, address, mailbox are set as page key message；The variation of page key message includes page key message
Newly-increased, update is deleted, for example the commodity price in library table record has become 80 yuan from 100 yuan, and inventory is become from 500
300 etc..
If preset page key message changes in the journal file, S40 is thened follow the steps, according to the institute of variation
It states page key message and sends page data update request to application server corresponding with the client, so that the backstage
The corresponding page data of server update, and updated page data is uploaded to preset content distributing network and is delayed
It deposits.
If preset page key message changes in journal file, need the variation informing application service at this time
Device.Specifically, when the log analysis service of page data processing equipment detects preset page key message in journal file
When changing, page data is sent more to application server corresponding with client according to the page key message of variation
New request updates corresponding page data, and will be on updated page data so that the background server, which enables, runs batch service
WEB server is reached, WEB server updates the timestamp of page data, and updated page data is uploaded to content point
Hairnet network is cached.
It should be noted that being cached with all library table records in above content distribution network in advance, client can receive
The accessing page request at end simultaneously returns to resource for the nearest node of client distribution distance, to improve the page load of client
Speed.It, can be with when subsequent client sends accessing page request after the page data that caches in content distributing network updates
Newest page info is directly got from the content distributing network.
The page data processing method that the present embodiment proposes works as journal file by obtaining the journal file of source database
In comprising preset action event type and when preset page key message changes, believed according to the page key after variation
It ceases to application server corresponding with the client and sends page data update request, so that background server update is corresponding
Page data, and updated page data is uploaded to preset content distributing network and is cached, it ensure that content distribution
The page data cached in network is consistent with the record in database, thus when client sends accessing page request, it can be with
Newest page info is directly got from the content distributing network, without visiting again application server and source database,
It avoids in the case where high concurrent request, the problem of performance bottleneck occur in application server and source database, improve client
The loading velocity in end page face.
Further, it is based on page data processing method first embodiment of the present invention, proposes page data processing of the present invention
Method second embodiment.
In the present embodiment, above-mentioned steps S30 may include: that the journal file is sent to preset distributed message
Queue, so that the journal file is forwarded to preset log analysis service by the Distributed Message Queue；Pass through the day
Will analysis service parses the journal file, to judge whether preset page key message is sent out in the journal file
It changes dynamic.
In the present embodiment, page data processing equipment, which can also be run, is distributed formula message queue, the distributed message
Queue can be Kafka (a kind of distributed post-subscription message system, mainly for the treatment of active stream data),
ActiveMQ (Message Queue, messaging bus), RabbitMQ, RocketMQ etc., when specific implementation, can flexible settings.Day
Will obtains service after getting the log information of source database, which is sent in Distributed Message Queue, point
The journal file is forwarded to log analysis service by cloth message queue, and log analysis service parses journal file, sentences
Whether disconnected wherein preset page key message changes.
Above by the mode of Distributed Message Queue forwarding journal file, is conducive to handle a large amount of stream data and expire
The real-time demand that sufficient page data updates.
Further, in the present embodiment, content distributing network can be divided into static buffer and dynamic buffering area, institute
Static buffer is stated for caching preset static page information, the dynamic buffering area is for caching preset dynamic page letter
Breath, the dynamic page information includes the page key message.
In the present embodiment, static buffer is used to cache almost unchanged information, i.e. static page information, such as HTTP
(Hyper Text Transfer Protocol, hypertext transfer protocol) message packet head, user's operation behavior, text and figure
Piece, webpage other elements etc., wherein HTTP message heading includes but is not limited to expirse, pragma, cache-
Control, last-modified, if-modified-since, ETag, if-none-match, vary etc., user's operation row
To include but is not limited to that address field carriage return, page link jump, new windowing, advance, retrogressing, F5 refreshes, ctrl+F5 forces brush
New etc., text and picture include but is not limited to text, Gif, png, jpg, jpeg, img etc., and the other elements of webpage include but unlimited
In css, html, javascript, apk application packet, ipa application packet, hyperlink, table, list, navigation bar, hovering button,
The files such as java special efficacy, ActiveX, gz, zip, rar, tar, audio-video.Dynamic buffering area is used to cache the letter often changed
Breath, i.e. dynamic page information, the dynamic page information include preset crucial page info, for example, may include price, inventory,
Sales volume, temperature, click volume, descriptive labelling, contact person, telephone number, address, mailbox etc..
When application server enables the race batch service corresponding page data of update every time, batch dynamic page information is only run, such as
This can be improved page data and updates efficiency.
The present invention also provides a kind of page data processing units.The page data processing unit includes:
Module is obtained, for obtaining the journal file of source database, the journal file is for recording client to described
The operation information of source database；
First judgment module, for whether judging in the journal file comprising preset action event type；
Second judgment module, if judging the day for including preset action event type in the journal file
Whether preset page key message changes in will file；
Processing module, if changing for page key message preset in the journal file, according to variation
The page key message sends page data to application server corresponding with the client and updates request, so that after described
The corresponding page data of platform server update, and updated page data is uploaded to preset content distributing network and is delayed
It deposits.
Further, the acquisition module is also used to:
When source database is MySQL database, socket is established with the MySQL database and is connect, and is created
Build Log backup thread；
The binary log that the MySQL database is generated according to the socket connection and the Log backup thread
File is monitored, and receives the binary log file of the MySQL database transmission.
Further, the first judgment module is also used to:
Judge whether to update comprising the data insertion based on the source database, data in the journal file or data are deleted
Except event type；
If including data insertion, data update or data deletion event based on the source database in the journal file
Type then determines in the journal file comprising preset action event type.
Further, second judgment module is also used to:
The journal file is sent to preset Distributed Message Queue, so that the Distributed Message Queue will be described
Journal file is forwarded to preset log analysis service；
The journal file is parsed by the log analysis service, it is preset in the journal file to judge
Whether page key message changes.
Further, the content distributing network is divided into static buffer and dynamic buffering area, and the static buffer is used
In caching preset static page information, the dynamic buffering area is for caching preset dynamic page information, the dynamic page
Face information includes the page key message.
Method performed by above-mentioned each program module can refer to each embodiment of page data caching method of the present invention, herein
It repeats no more.
The present invention also provides a kind of computer readable storage mediums.
Page data processing routine, the page data processing routine are stored on computer readable storage medium of the present invention
The step of page data processing method as described above is realized when being executed by processor.
Wherein, the page data processing routine run on the processor, which is performed realized method, can refer to this
The each embodiment of invention page data processing method, details are not described herein again.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the system that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or system institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or system.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that terminal device (it can be mobile phone,
Computer, server, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (12)
1. a kind of page data processing method, which is characterized in that the page data processing method includes the following steps:
The journal file of source database is obtained, the journal file, which is used to record client, believes the operation of the source database
Breath；
Whether judge in the journal file comprising preset action event type；
If in the journal file including preset action event type, judge that the preset page is crucial in the journal file
Whether information changes；
If preset page key message changes in the journal file, according to the page key message of variation to
Application server corresponding with the client sends page data and updates request, so that background server update is corresponding
Page data, and updated page data is uploaded to preset content distributing network and is cached.
2. page data processing method as described in claim 1, which is characterized in that the journal file for obtaining source database
The step of include:
When source database is MySQL database, socket is established with the MySQL database and is connect, and creates day
Will backup thread；
The binary log file that the MySQL database is generated according to the socket connection and the Log backup thread
It is monitored, and receives the binary log file of the MySQL database transmission.
3. page data processing method as described in claim 1, which is characterized in that it is described judge in the journal file whether
The step of including preset action event type includes:
Judge whether to update comprising the data insertion based on the source database, data in the journal file or data delete thing
Part type；
If including data insertion, data update or data deletion event class based on the source database in the journal file
Type then determines in the journal file comprising preset action event type.
4. page data processing method as claimed any one in claims 1 to 3, which is characterized in that the judgement day
The step of whether preset page key message changes in will file include:
The journal file is sent to preset Distributed Message Queue, so that the Distributed Message Queue is by the log
File is forwarded to preset log analysis service；
The journal file is parsed by the log analysis service, to judge the preset page in the journal file
Whether key message changes.
5. page data processing method as claimed in claim 4, which is characterized in that the content distributing network is divided into static slow
Area and dynamic buffering area are deposited, for caching preset static page information, the dynamic buffering area is used for for the static buffer
Preset dynamic page information is cached, the dynamic page information includes the page key message.
6. a kind of page data buffer storage, which is characterized in that the page data buffer storage includes:
Module is obtained, for obtaining the journal file of source database, the journal file is for recording client to the source number
According to the operation information in library；
First judgment module, for whether judging in the journal file comprising preset action event type；
Second judgment module, if judging the log text for including preset action event type in the journal file
Whether preset page key message changes in part；
Processing module, if changing for page key message preset in the journal file, according to variation
Page key message sends page data to application server corresponding with the client and updates request, so that the backstage takes
Business device updates corresponding page data, and updated page data is uploaded to preset content distributing network and is cached.
7. page data buffer storage as claimed in claim 6, which is characterized in that the acquisition module is also used to:
When source database is MySQL database, socket is established with the MySQL database and is connect, and creates day
Will backup thread；
The binary log file that the MySQL database is generated according to the socket connection and the Log backup thread
It is monitored, and receives the binary log file of the MySQL database transmission.
8. page data buffer storage as claimed in claim 6, which is characterized in that the first judgment module is also used to:
Judge whether to update comprising the data insertion based on the source database, data in the journal file or data delete thing
Part type；
If including data insertion, data update or data deletion event class based on the source database in the journal file
Type then determines in the journal file comprising preset action event type.
9. the page data buffer storage as described in any one of claim 6 to 8, which is characterized in that described second judges mould
Block is also used to:
The journal file is sent to preset Distributed Message Queue, so that the Distributed Message Queue is by the log
File is forwarded to preset log analysis service；
The journal file is parsed by the log analysis service, to judge the preset page in the journal file
Whether key message changes.
10. page data buffer storage as claimed in claim 9, which is characterized in that the content distributing network is divided into static state
Buffer area and dynamic buffering area, the static buffer are used for caching preset static page information, the dynamic buffering area
In caching preset dynamic page information, the dynamic page information includes the page key message.
11. a kind of page data processing equipment, which is characterized in that the page data processing equipment includes: memory, processor
And it is stored in the page data caching program that can be run on the memory and on the processor, the page data caching
The step of page data processing method as described in any one of claims 1 to 5 is realized when program is executed by the processor.
12. a kind of computer readable storage medium, which is characterized in that be stored with page number on the computer readable storage medium
According to caching program, the page data caching program is realized as described in any one of claims 1 to 5 when being executed by processor
The step of page data processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910321415.1A CN110083616A (en) | 2019-04-19 | 2019-04-19 | Page data processing method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910321415.1A CN110083616A (en) | 2019-04-19 | 2019-04-19 | Page data processing method, device, equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110083616A true CN110083616A (en) | 2019-08-02 |
Family
ID=67415875
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910321415.1A Pending CN110083616A (en) | 2019-04-19 | 2019-04-19 | Page data processing method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110083616A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110753316A (en) * | 2019-09-26 | 2020-02-04 | 贝壳技术有限公司 | Information sending method and device, computer readable storage medium and electronic equipment |
| CN110991871A (en) * | 2019-11-29 | 2020-04-10 | 深圳前海微众银行股份有限公司 | Risk monitoring method, device, equipment and computer readable storage medium |
| CN111338821A (en) * | 2020-02-25 | 2020-06-26 | 北京思特奇信息技术股份有限公司 | Method, system and electronic equipment for realizing data load balance |
| CN111506787A (en) * | 2020-03-06 | 2020-08-07 | 百度在线网络技术（北京）有限公司 | Webpage updating method and device, electronic equipment and computer-readable storage medium |
| CN111506786A (en) * | 2020-03-06 | 2020-08-07 | 百度在线网络技术（北京）有限公司 | Webpage updating method and device, electronic equipment and computer-readable storage medium |
| CN111831947A (en) * | 2020-07-27 | 2020-10-27 | 中国工商银行股份有限公司 | Application system, data processing method, computer system, and storage medium |
| CN112269837A (en) * | 2020-11-17 | 2021-01-26 | 珠海大横琴科技发展有限公司 | Data processing method and device |
| CN112860772A (en) * | 2019-11-12 | 2021-05-28 | 武汉微品致远信息科技有限公司 | Static file replacing service storage information data processing system |
| CN113420037A (en) * | 2021-06-27 | 2021-09-21 | 杭州迪普科技股份有限公司 | Real-time asset data changing method and device |
| CN113918526A (en) * | 2021-10-12 | 2022-01-11 | 平安国际智慧城市科技股份有限公司 | Log processing method and device, computer equipment and storage medium |
| CN115374218A (en) * | 2022-08-26 | 2022-11-22 | 苏州浪潮智能科技有限公司 | Data synchronization method, device, equipment and medium for heterogeneous database |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101770484A (en) * | 2008-12-31 | 2010-07-07 | 北大方正集团有限公司 | Method and system for network upgrade real-time release |
| CN104572689A (en) * | 2013-10-17 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Data synchronizing method, device and system |
| CN107341212A (en) * | 2017-06-26 | 2017-11-10 | 努比亚技术有限公司 | A kind of buffering updating method and equipment |
| CN107784086A (en) * | 2017-10-09 | 2018-03-09 | 中国平安人寿保险股份有限公司 | A kind of loading method of webpage, terminal device and computer-readable recording medium |
-
2019
- 2019-04-19 CN CN201910321415.1A patent/CN110083616A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101770484A (en) * | 2008-12-31 | 2010-07-07 | 北大方正集团有限公司 | Method and system for network upgrade real-time release |
| CN104572689A (en) * | 2013-10-17 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Data synchronizing method, device and system |
| CN107341212A (en) * | 2017-06-26 | 2017-11-10 | 努比亚技术有限公司 | A kind of buffering updating method and equipment |
| CN107784086A (en) * | 2017-10-09 | 2018-03-09 | 中国平安人寿保险股份有限公司 | A kind of loading method of webpage, terminal device and computer-readable recording medium |
Non-Patent Citations (1)
| Title |
|---|
| 严国辉等: "电子商务", 30 August 2008, 严国辉等 * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110753316A (en) * | 2019-09-26 | 2020-02-04 | 贝壳技术有限公司 | Information sending method and device, computer readable storage medium and electronic equipment |
| CN110753316B (en) * | 2019-09-26 | 2022-04-26 | 贝壳技术有限公司 | Information sending method and device, computer readable storage medium and electronic equipment |
| CN112860772A (en) * | 2019-11-12 | 2021-05-28 | 武汉微品致远信息科技有限公司 | Static file replacing service storage information data processing system |
| CN112860772B (en) * | 2019-11-12 | 2023-05-26 | 武汉微品致远信息科技有限公司 | Static file substitution service storage information class data processing system |
| CN110991871A (en) * | 2019-11-29 | 2020-04-10 | 深圳前海微众银行股份有限公司 | Risk monitoring method, device, equipment and computer readable storage medium |
| CN111338821A (en) * | 2020-02-25 | 2020-06-26 | 北京思特奇信息技术股份有限公司 | Method, system and electronic equipment for realizing data load balance |
| CN111338821B (en) * | 2020-02-25 | 2023-04-07 | 北京思特奇信息技术股份有限公司 | Method, system and electronic equipment for realizing data load balance |
| CN111506786A (en) * | 2020-03-06 | 2020-08-07 | 百度在线网络技术（北京）有限公司 | Webpage updating method and device, electronic equipment and computer-readable storage medium |
| CN111506787B (en) * | 2020-03-06 | 2023-04-25 | 百度在线网络技术（北京）有限公司 | Method, device, electronic equipment and computer readable storage medium for web page update |
| CN111506787A (en) * | 2020-03-06 | 2020-08-07 | 百度在线网络技术（北京）有限公司 | Webpage updating method and device, electronic equipment and computer-readable storage medium |
| CN111506786B (en) * | 2020-03-06 | 2023-10-27 | 百度在线网络技术（北京）有限公司 | Method, device, electronic equipment and computer readable storage medium for web page update |
| CN111831947A (en) * | 2020-07-27 | 2020-10-27 | 中国工商银行股份有限公司 | Application system, data processing method, computer system, and storage medium |
| CN111831947B (en) * | 2020-07-27 | 2023-08-15 | 中国工商银行股份有限公司 | Application system, data processing method, computer system and storage medium |
| CN112269837A (en) * | 2020-11-17 | 2021-01-26 | 珠海大横琴科技发展有限公司 | Data processing method and device |
| CN113420037A (en) * | 2021-06-27 | 2021-09-21 | 杭州迪普科技股份有限公司 | Real-time asset data changing method and device |
| CN113918526A (en) * | 2021-10-12 | 2022-01-11 | 平安国际智慧城市科技股份有限公司 | Log processing method and device, computer equipment and storage medium |
| CN115374218A (en) * | 2022-08-26 | 2022-11-22 | 苏州浪潮智能科技有限公司 | Data synchronization method, device, equipment and medium for heterogeneous database |
| CN115374218B (en) * | 2022-08-26 | 2024-01-26 | 苏州浪潮智能科技有限公司 | Data synchronization method, device, equipment and medium of heterogeneous database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110083616A (en) | Page data processing method, device, equipment and computer readable storage medium | |
| US11606411B1 (en) | Application state server-side cache for a state-based client-server application | |
| US7062756B2 (en) | Dynamic object usage pattern learning and efficient caching | |
| US20220179791A1 (en) | Invalidation and refresh of multi-tier distributed caches | |
| US8407338B2 (en) | Methods and systems for polling an on demand service | |
| US7702800B2 (en) | Detecting and handling affinity breaks in web applications | |
| US10747670B2 (en) | Reducing latency by caching derived data at an edge server | |
| US20020116582A1 (en) | Batching of invalidations and new values in a web cache with dynamic content | |
| CN101493826B (en) | Database system based on WEB application and data management method thereof | |
| US20040073630A1 (en) | Integrated JSP and command cache for web applications with dynamic content | |
| US6941310B2 (en) | System and method for caching data for a mobile application | |
| US8825749B2 (en) | Method of tracking offline user interaction in a rendered document on a mobile device | |
| Ravi et al. | A survey on dynamic Web content generation and delivery techniques | |
| CN107305576A (en) | The pseudo- static treatment method and apparatus of the page | |
| CN102985921B (en) | There is the client terminal device high speed caching electronic document resources of e-sourcing data base | |
| US9081869B2 (en) | System, method and computer program product for communicating data between a database and a cache | |
| US11055367B2 (en) | Web application architecture for information management | |
| US20020116583A1 (en) | Automatic invalidation dependency capture in a web cache with dynamic content | |
| US6807606B2 (en) | Distributed execution coordination for web caching with dynamic content | |
| CN107329963A (en) | Accelerate the method and apparatus of web page access | |
| Yuan et al. | Evaluation of edge caching/offloading for dynamic content delivery | |
| US20020111992A1 (en) | JSP composition in a cache for web applications with dynamic content | |
| US6823360B2 (en) | Cofetching in a command cache | |
| US20020112125A1 (en) | Command caching to improve network server performance | |
| US10268536B2 (en) | Secure debugging with an encrypted token |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |