WO2015043391A1 - Data synchronization for remote and local databases - Google Patents

Data synchronization for remote and local databases Download PDF

Info

Publication number
WO2015043391A1
WO2015043391A1 PCT/CN2014/086599 CN2014086599W WO2015043391A1 WO 2015043391 A1 WO2015043391 A1 WO 2015043391A1 CN 2014086599 W CN2014086599 W CN 2014086599W WO 2015043391 A1 WO2015043391 A1 WO 2015043391A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
database
local
server
databases
Prior art date
Application number
PCT/CN2014/086599
Other languages
English (en)
French (fr)
Inventor
Xiaowei Ma
Kunxian WANG
Yong Wang
Dewen WANG
Junhao SONG
Original Assignee
Tencent Technology (Shenzhen) Company Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Company Limited filed Critical Tencent Technology (Shenzhen) Company Limited
Publication of WO2015043391A1 publication Critical patent/WO2015043391A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data

Definitions

  • the present application relates to the technical field of database management, and in particular, to methods, systems and servers for synchronizing data among remote and local databases.
  • An Internet based data management system often manages huge volumes of data, and is therefore bound by special requirements for its infrastructure and corresponding data processing process. Specifically, the data management system should reduce its use of computational resources while still delivering satisfactory data processing efficiency.
  • FIG. 1 illustrates a data management system 100 that processes content information communicated over the Internet.
  • the content information is carried in digital codes, and presented to interested audience in different forms (e. g. , a webpage and a user interface) .
  • the content information is normally associated with a destination webpage. When a user clicks on a link to the content information on a browser, the corresponding destination webpage is loaded for display to the user.
  • Information technology for loading and displaying the content information has been drastically developed during the past few years, and widely applied in many fields. For example, such information technology is also used in the field of Internet advertisements, and an Internet advertisement is regarded as one type of content information.
  • An important aspect of the information technology for loading and displaying the content information is associated with distribution of data associated with the content information.
  • the content information needs to be distributed and presented to a specific audience effectively, and therefore, needs to be managed efficiently on a server level prior to being presented to a client device.
  • a large volume of data has to be analyzed and stored for efficient information distribution, presentation and management on servers. This task can be implemented only using a well-arranged infrastructure of the data management system and a corresponding content management method.
  • data management system 100 optionally includes a business intelligence (BI) server 101, a backup (BB) server 102, a BB host machine 103, a WEB server 104, a preview server 105 and a development machine 106.
  • BI server 101 is a data source server mainly used for gathering the content information, such as advertisement data, order data, client data, and visitor information (e. g. , a click-through rate of a website) , from data sources distributed over a computer network.
  • Backup server 102 is a data computation server coupled to BI server 101 and configured to support operation analysis tool systems that operate on the content information. Specifically, backup server 102 synchronizes data from BI server 101 regularly (e. g.
  • BB host machine 103 is a client server used for running a plurality of service programs according to a specific application implemented on machine 103. In some situations, when that different service programs are running simultaneously on machine 103 and competing for limited computational resources, part of each service program is separated and computed on backup server 102, and thereby, BB host 103 is reserved for responding to a user’s requests.
  • WEB server 104 is configured to generate a file cache for all service modules and update the file cache on a daily basis. When a user request is received, it is sent to WEB server 104 and compared with those stored in the file cache of WEB server 104. Only when the file cache does not include the user request, BB host machine 103 is searched for the user request.
  • preview server 105 is configured to synchronize its data with those stored in backup server 102, and generate a preview of its synchronized data.
  • a development environment is created on development machine 106 for data synchronization and preview, such that development machine 106 allows the user to develop a program based on the synchronized data displayed on this development environment.
  • An ORACLE database is usually used when huge volumes of data are involved, and an open-source software KETTLE is used to synchronize data between BI server 101 and BB backup server 102.
  • KETTLE is a JAVA-based tool for data extraction, transformation, and loading (ESL) , and has been widely used as open-source software.
  • ESL data extraction, transformation, and loading
  • KETTLE is optionally compiled based on the JAVA language or the Extensible Markup Language (XML) .
  • KETTLE works under a workflow-based mode, and includes a transformation file (e. g. , a ktr file) and a task file (e. g. , a kjb file) .
  • a data source In the workflow-based mode, a data source, a data destination and a sequence of operations are identified. Further, in accordance with the sequence of operations, source data are identified in a search of the data source, placed in a temporary table, and inserted into a destination table in a database located at the data destination.
  • JDBC Java DataBase Connectivity
  • API Java application programming interface
  • BB backup server 102 in the prior art uses a relatively simple data processing mode, results in a low data processing efficiency, and therefore, wastes its own computational capability. Therefore, there is a need to synchronize data (e. g. , content information) efficiently among remote and local databases that are located at different server machines.
  • data e. g. , content information
  • the application is implemented in a server that has one or more processors, memory and one or more modules, programs or sets of instructions stored in the memory for performing multiple functions. Instructions for performing these functions may be included in a computer program product configured for execution by one or more processors.
  • One aspect of the application is a data processing method implemented at an intermediate server.
  • the method includes sending a user query received from a client server to a remote database server that includes a remote database, wherein the intermediate server is communicatively coupled between the remote database server and the client server, and the client server is configured to implement a software application.
  • the method further includes in response to the user query, synchronizing data in the local database with data in the remote database using a database link that is established on a database layer, wherein both data in the local and remote databases are associated with the software application; generating a materialized view of the data synchronized in the local database, wherein the materialized view is atomically refreshed with respect to the data in the local database; in accordance with a set of business rules associated with the software application, computing the data synchronized in the local database based on the materialized view; and communicating the computed data to the client server.
  • a server that includes one or more processors and memory having at least one program (including instructions) stored thereon, which when executed by the one or more processors causes the processors to perform operations to implement the above data processing method.
  • Another aspect of the application is a non-transitory computer readable storage medium that stores at least one program configured for execution by at least one processor of an electronic device.
  • the at least one program includes instructions for causing a server to perform the operations in the method described above.
  • FIG. 1 illustrates a data management system that processes content information communicated over the Internet.
  • FIG. 2 illustrates a flow chart of a data processing method for synchronizing data in databases in accordance with some embodiments.
  • FIG. 3 illustrates a block diagram of a data computation server (i. e. , an intermediate server) that synchronizes data in a local database with data in a remote database in accordance with some embodiments.
  • a data computation server i. e. , an intermediate server
  • FIG. 4 illustrates an exemplary data management system that includes two or more servers in which database data are synchronized in accordance with some embodiments.
  • FIG. 5 illustrates another exemplary data management system that includes two or more servers in which database data are synchronized in accordance with some embodiments.
  • FIG. 6 illustrates a flow chart for an exemplary data processing method 600 in which an intermediate server synchronizes data in a local database and a remote database in accordance with some embodiments of the application.
  • FIG. 7 illustrates a block diagram of an intermediate server 700 that synchronizes data in a local database and a remote database in accordance with some embodiments of the application.
  • FIG. 2 illustrates a flow chart of a data processing method 200 for synchronizing data in databases in accordance with some embodiments.
  • Method 200 is, optionally, governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a server.
  • Each of the operations shown in FIG. 2 may correspond to instructions stored in a computer memory or non-transitory computer readable storage medium.
  • the computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices.
  • the instructions stored on the computer readable storage medium may include one or more of: source code, assembly language code, object code, or other instruction format that is interpreted by one or more processors.
  • data processing method 200 includes (201) synchronizing source data provided by a database of a data source server to a local database by using a file duplication operation on a database layer.
  • the data source server includes BI server 101
  • data processing method 200 is implemented in BB backup server 102 to synchronize data of a local database in BB backup server 102 with data of a remote database in BI server 101.
  • both the local database in BB backup server 102 and the remote database in BI server 101 are established as ORACLE databases or Microsoft SQL Server databases.
  • the data source server is sometimes called as a remote database server or a BI server 101.
  • both the local database in BB backup server 102 and the remote database in BI server 101 are established as ORACLE databases.
  • ORACLE databases On a database layer, data files are duplicated using a DBlink data synchronization operation of the ORACLE databases.
  • DBlink data synchronization operation when a user intends to access data in another remote database distinct from a local database, a dblink is created in the local database to access the remote database.
  • the dblink in the local database would allow the user to access data in the remote database as if the data in the remote database belong to the local database.
  • the dblink is an object associated with an access path coupling the local and remote databases, and allows the user to search in the remote database and execute a remote program. Because the dblink does not rely on any intermediate temporary table to transfer data, its data transfer rate is improved compared with a KETTLE based method that has been used to transfer data between two databases.
  • both the local database in BB backup server 102 and the remote database in BI server 101 are established as ORACLE databases
  • data files are duplicated or synchronized among these two databases using a data pump supported by the ORACLE databases.
  • the data pump of the ORACLE debases operates based on an end-to-end copy at a data file level of the databases.
  • the data pump does not rely on any intermediate temporary table to transfer data, and is a newer, faster, and more flexible data import and export tool provided by the ORACLE databases.
  • the data pump allows data files to be transferred between the local and remote databases at an enhanced data transfer rate compared with the conventional KETTLE based method.
  • both the local database in BB backup server 102 and the remote database in BI server 101 are established as Microsoft SQL Server databases which have a data synchronization operation similar to the DBlink data synchronization operation in the ORACLE databases. Therefore, a local database established as an SQL Server database uses a DBlink data synchronization operation of the SQL Server database to duplicate data from a remote database.
  • a local database is coupled to more than one remote databases each hosted by a respective data source server and associated with a respective data service type (sometimes called data type) .
  • the data in the more than one remote database is synchronized to the local database according to a predetermined synchronization order. Specifically, respective data in each remote database are associated with a respective data service type, and the data in the remote databases are sequentially synchronized to the local database based on their respective data service type. At a specific instant, only data having one specific data service type is synchronized between the local and remote databases. Further, when a remote database is coupled to and provides data having a specific data service type to more than one local database, the remote database is also configured to provide the data to these local databases sequentially.
  • This sequential order is set to ensure that there is only one downstream local database requesting data from the remote database at a time.
  • the remote database would be protected from a load pressure that occurs when multiple local databases simultaneously request data from the remote database, allowing a high level of stability for the data source server that hosts the remote database.
  • Data processing method 200 further includes (202) executing a materialized view refresh for the data synchronized to the local database, and an atomic refresh is used for each materialized view.
  • the materialized view refers to a virtual table in a computer database, and content of the materialized view is determined according to a query.
  • a materialized view is associated with data that could be arranged according to a series of columns and rows.
  • the data associated with the materialized view are not stored in an actual table format within any remote or local database. Rather, the data associated with the materialized view are dynamically generated according to the query that determines the respective source of each datum in the materialized view.
  • the materialized view is generated by processing (e. g. , joining or clustering) tables in the local database and storing the processed tables. Such database operations normally consumes more time than most of the other database operations. Thus, the materialized view provides the processing results conveniently and fast without repeating these time-consuming operations during the course of data processing. From many perspectives, the materialized view is equivalent to an index of the database used to improve searchability of the database. That is to say, the materialized view is transparent to an application (i. e. , addition and deletion of a materialized view do not affect correctness and effectiveness of SQL statements in corresponding programs of the application) . Also, the materialized view is physically stored in a memory of the server that hosts the local database, and has to be refreshed according to changes that occur to the local database from which the materialized view is generated.
  • processing e. g. , joining or clustering
  • a group of materialized views need to be refreshed together.
  • the entire group of materialized views is forgone from being refreshed.
  • each materialized view is refreshed separately but atomically.
  • the atomic refresh operations on the other materialized views are not affected. Efficiency of data processing is thereby improved as a result of such individualized atomic refresh of the materialized views.
  • the materialized view is associated with a local database implemented as an ORACLE database, and therefore, an ATOMIC_REFRESH parameter used in the ORACLE database.
  • an ATOMIC_REFRESH parameter used in the ORACLE database.
  • the ATOMIC_REFRESH parameter is set to TRUE, an old materialized view is not deleted until a refreshed materialized view is stored in the memory. Stated another way, the ATOMIC_REFRESH parameter set to TRUE forces a DELETE of the old materialized view instead of TRUNCATE and makes the materialized view more "available" at a refresh time.
  • Data processing method 200 further includes (203) performing service logic computation on the data of the local database in accordance with a set of business rules associated with a software application.
  • BB backup server 102 performs service logic computation
  • some preferred embodiments in the application use some preferred data processing methods to synchronize data among different databases, improve processing efficiency associated with specific business rules and enhancing performance of overall data processing of a local computer.
  • an operation mode of the local database is set to a non-archive mode.
  • an online redo log is maintained to keep track of database operations, e. g. , data insertion, deletion, and update.
  • the local database often includes two or more online redo logs. When a first log of the two or more online redo logs is full, the database switches to use a second log of the two or more online redo logs. Further, when the second log is full, the database switches again to use (e. g. , overwrite) the first log of the two or more online redo logs. The above process repeats when the first log is full again. As such, the database tracks database operations by switching between the first and second logs of the two or more online redo logs.
  • non-archive mode of the ORACLE database e. g. , a Noarchivelog mode
  • data in a full log are not archived, when the database switches to another log.
  • an archive mode the data in the full log are archived under the same circumstances (e. g. , when the respective log is full and the database switches to another log) .
  • the database switches to use a second log of the two or more online redo logs, and content of the first log is copied to a specific archive directory.
  • the copied redo log is referred to as an archived redo log.
  • a default setting of an ORACLE database is the archive mode in which the database generates and stores a large amount of log data. Management of these log data compromises performance of the database. Therefore, in some implementations, the operation mode of the ORACLE database is set at the non-archive mode (e. g. , a Noarchivelog mode) . This reduces the amount of log data that need to be tracked and managed by the database, and enhances the performance of the corresponding server (e. g. , BB backup server 102) . Likewise, when other types of databases (e. g. , SQL server databases) are used, their operation modes could be associated with an archive mode and a non-archive mode in which the non-archive mode is used in accordance with data processing method 200.
  • non-archive mode e. g. , a Noarchivelog mode
  • data processing method 200 when service logic computation is performed, data processing method 200 further includes determining the number (e. g. , N) of cores used in a central processing unit (CPU) in a server (e. g. , BB backup server 102) , and allocating a subset of the cores (e. g. , N-1 cores) for parallel service logic computation.
  • N the number of cores used in a central processing unit (CPU) in a server (e. g. , BB backup server 102)
  • a subset of the cores e. g. , N-1 cores
  • data processing method 200 further includes determining the number (e. g. , N) of cores used in a central processing unit (CPU) in a server (e. g. , BB backup server 102) , and allocating a subset of the cores (e. g. , N-1 cores) for parallel service logic computation.
  • single-core computation in the prior art is replaced with multi
  • the remaining one idle core is used to coordinate sequential operations of the CPU cores and manage the other resources in the corresponding server.
  • a structured query language (SQL) statement may be used to specific the number of CPU cores reserved for the service logic computation.
  • the SQL statement optionally reserves N-1 CPU cores of a server having N CPU cores for computing the data synchronized in the local database according to the relevant business rules.
  • the serve stores (204) a result of the above service logic computation, and optionally synchronizes (204) the result of the service logic computation to a client server (e. g. , a server on the client side, BB host 103) .
  • the result of the service logic computation is stored by (a) invalidating an index in a database table, (b) inserting data of the result of the service logic computation into the database table, and (3) after the insertion is finished, reestablishing an index of the database table.
  • concurrent update of an index of a database table consumes more input/output (IO) resources in a memory than reestablishment of an index. Therefore, less I/O resources are required to access the memory of the data computation server when the reestablished index to the database table is used, while data processing performance (including the corresponding efficiency) could be further improved.
  • IO input/output
  • the result of the service logic computation is optionally synchronized to a client server (e. g. , BB host 103 shown in FIG. 1) .
  • a client server e. g. , BB host 103 shown in FIG. 1
  • the present application further discloses a data computation server (i. e. , an intermediate server between a data source server and a client server) .
  • a data computation server i. e. , an intermediate server between a data source server and a client server
  • the data computation server is associated with BB backup server 102.
  • the data source server and the client served coupled by the data computation server are associated with BI server 101 and BB host 103, respectively
  • FIG. 3 illustrates a block diagram of a data computation server 300 (also called an “intermediate server” in this application) that synchronizes data in a local database with data in a remote database in accordance with some embodiments.
  • data computation server 300 optionally includes some or all modules of a first synchronization module 301, a materialization module 302, a computation module 303, a storage module 304, and a second synchronization module 305.
  • First synchronization module 301 is configured to synchronize source data from a remote database of a data source server to a local database by using a file duplication operation on a database layer.
  • Materialization module 302 is configured to execute a materialized view refresh for the data synchronized to the local database, and an atomic refresh is optionally used for each individual materialized view.
  • Computation module 303 is configured to perform service logic computation on the data of the local database.
  • Storage module 304 is configured to store a result of the service logic computation.
  • Second synchronization module 305 is configured to synchronize the result of the service logic computation to a client server.
  • the local database is implemented as an ORACLE database
  • the file duplication operation used in first synchronization module 301 is associated with a DBlink data synchronization operation of the ORACLE database.
  • the dblink is an object associated with a path linking the remote and local databases. The dblink does not use any intermediate temporary table, and its data transfer speed is faster than a KETTLE-based solution used in some prior art.
  • the file duplication operation used in first synchronization module 301 is associated with a data pump of the ORACLE database.
  • the data pump is a newer, faster, and more flexible data import and export tool provided by ORACLE.
  • the data pump use does not use any intermediate temporary table. Therefore, the data transfer speed of the data pump is faster than a KETTLE-based solution used in some prior art.
  • first synchronization module 301 is configured to set a synchronization order for the data in the local database according to their respective data types (sometimes also called service types) .
  • data types sometimes also called service types
  • only data of one specific service type is synchronized at a time, such that stability of the data source server may be enhanced by limiting the communication between the local and remote databases to this specific service type at this specific time.
  • computation module 303 is further configured to operate on data of the local database based on a non-archive mode, and improves data processing efficiency of data computation server 300.
  • the databases are implemented as ORACLE databases that maintain detailed logs about their operations by default.
  • such a logging function consumes a considerable amount of computational resources in the intermediate server that needs to reserve the recourses to synchronize data in the local database with data in the remote database.
  • the operation mode of the database is changed to the non-archive mode for the purposes of reducing use of the computational resources for the log function and thereby enhancing the data processing performance of the intermediate server.
  • the operation mode of the local database is operationally set as an archive mode or a non-archive mode.
  • the non-archive mode is enabled on these other databases for the purpose of reserving the computational resources for the database synchronization function.
  • the computation module 303 is further configured to determine the number of cores (e. g. , N) in a CPU in the intermediate server (e. g. , data computation server 300) that hosts the local database, and allocate a subset of the CPU cores (e. g. , N-1) for parallel service logic computation (e. g. , computing the data in the local database according to the set of business rules) .
  • N the number of cores
  • N e. g. , N
  • parallel service logic computation e. g. , computing the data in the local database according to the set of business rules
  • storage module 304 is configured to: (a) invalidate an index in a database table, (b) insert data of the result of the service logic computation into the database table, and (3) after the insertion is finished, reestablish an index of the database table.
  • IO input/output
  • the result of the service logic computation is optionally synchronized to a client server (e. g. , BB host 103 shown in FIG. 1) .
  • a client server e. g. , BB host 103 shown in FIG. 1
  • FIG. 4 illustrates an exemplary data management system 400 that includes two or more servers in which database data are synchronized in accordance with some embodiments.
  • Data management system 400 at least includes: a BI server 401, and a network media information operation analysis tool system (BB) host 403.
  • BI server 401 is configured to provide source data of network media information that are stored in a remote database in BI server 401.
  • BB host 403 is configured to respond to a user request to provide some data associated with the source data of network media information.
  • data management system 400 further includes a BB backup server 402, i. e. , an intermediate server that transfers the source data of network media information in BI server 401 to BB host 403.
  • BB backup server 402 further includes a data computation server (e. g. , server 300) . More details on data computation server 300 are explained above with reference to FIG. 3.
  • a data computation server e. g. , server 300
  • BI server 401 and BB host 403 are used as a data source server and a client server, respectively.
  • FIG. 5 illustrates another exemplary data management system 500 that includes two or more servers in which database data are synchronized in accordance with some embodiments.
  • Data management system 500 includes a BI server 401, a BB backup server 402, a BB host 403, a WEB server 404, a preview server 405 and a development server 406.
  • WEB server 404 is coupled to BB host 403 and configured to cache data for BB host 403. For example, a user request is received by WEB server 404, and WEB server 404 searches its local cache to respond to the user request. If a response is identified in the local cache of WEB server 404, corresponding data are extracted from the local cache and provided in response to the user request. However, if no response is identified in the local cache of WEB server 404, WEB server 404 sends the user requester further to BB host. By these means, WEB server 404 reduces BB host’s load for processing the user requests.
  • data management system 500 further includes a preview server 405 and a development machine 406.
  • Preview server 405 is coupled between BB backup server 402 and development machine 406.
  • Preview server 405 is configured to synchronize data of BB backup server 405, and enable a concurrent preview and an iterative test of the data.
  • development machine 406 performs program development in a development environment created by the preview server 405.
  • FIG. 6 illustrates a flow chart for an exemplary data processing method 600 in which an intermediate server synchronizes data in a local database and a remote database in accordance with some embodiments of the application.
  • Method 600 is, optionally, governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a server.
  • Each of the operations shown in FIG. 6 may correspond to instructions stored in a computer memory or non-transitory computer readable storage medium.
  • the computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices.
  • the instructions stored on the computer readable storage medium may include one or more of: source code, assembly language code, object code, or other instruction format that is interpreted by one or more processors.
  • Method 600 is implemented on an intermediate server that includes a local database.
  • the intermediate server sends (602) a user query received from a client server to a remote database server (also called a data source server) that includes a remote database.
  • the intermediate server is communicatively coupled between the remote database server and the client server, and the client server is configured to implement a software application based on at least part of data in the local database.
  • the intermediate server synchronizes (604) data in the local database with data in the remote database using a database link that is established on a database layer.
  • Both data in the local and remote databases are associated with the software application implemented on the client server.
  • the remote and local databases are implemented as Oracle databases, and the database link is associated with a DBlink function in the Oracle databases.
  • the database link is associated with an Oracle Data Pump feature in the Oracle databases.
  • the remote and local databases are implemented as SQL Server databases, and the database link used to synchronize the data in the local and remote databases is associated with a DBlink function in the Oracle databases.
  • the data in the local database are sequentially synchronized with the corresponding data in the remote database according their respective data types.
  • the local database is set at a Noarchivelog mode in which the local database does not perform a recovery operation or an online backup.
  • the intermediate server further generates (606) a materialized view of the data synchronized in the local database, and this materialized view is atomically refreshed with respect to the data in the local database.
  • the intermediate server computes (608) the data synchronized in the local database based on the materialized view, and communicates (610) the computed data to the client server for the purpose of implementing the software application.
  • computing the data synchronized in the local database further includes determining the number of cores that are included in the central processing unit (CPU) of the intermediate server; and assigning a subset of the cores in the CPU to compute the data synchronized in the local database based on the materialized view.
  • CPU central processing unit
  • the intermediate server disables a database index associated with the local database, stores the computed data in the local database and modifies the database index to reflect that the computed data are stored in the local database.
  • FIG. 7 illustrates a block diagram of an intermediate server system 700 that synchronizes data in a local database and a remote database in accordance with some embodiments of the application.
  • server system 700 at least includes one or more processors 710 (e. g. , central processing units) and a memory 720 for storing data, programs and instructions for execution by one or more processors 710.
  • server system 700 further includes one or more communication interfaces 730, an input/output (I/O) interface 740, and one or more communication buses 750 that interconnect these components.
  • processors 710 e. g. , central processing units
  • memory 720 for storing data, programs and instructions for execution by one or more processors 710.
  • server system 700 further includes one or more communication interfaces 730, an input/output (I/O) interface 740, and one or more communication buses 750 that interconnect these components.
  • I/O input/output
  • I/O interface 740 includes an input unit 742 and a display unit 744.
  • input unit 742 include a keyboard, a mouse, a touch pad, a game controller, a function key, a trackball, a joystick, a microphone, a camera and the like.
  • display unit 744 displays information that is inputted by the user or provided to the user for review. Examples of display unit 744 include, but are not limited to, a liquid crystal display (LCD) and an organic light-emitting diode (OLED) display.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • input unit 742 and display unit 744 are integrated on a touch-sensitive display that displays a graphical user interface (GUI) .
  • GUI graphical user interface
  • communication buses 750 include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • communication interfaces 730 further include a receiver 732 and a transmitter 734.
  • memory 720 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.
  • memory 720 includes one or more storage devices remotely located from the one or more processors 710.
  • memory 720, or alternatively the non-volatile memory device (s) within memory 720 includes a non-transitory computer readable storage medium.
  • memory 720 or alternatively the non-transitory computer readable storage medium of memory 720 stores the following programs, modules and data structures, instructions, or a subset thereof:
  • ⁇ Operating System 701 that includes procedures for handling various basic system services and for performing hardware dependent tasks;
  • I/O interface module 702 that includes procedures for handling various basic input and output functions through one or more input and output devices, wherein I/O interface module 702 further includes an interface display module that controls displaying of a graphical user interface;
  • ⁇ Communication module 703 that is used for connecting server system 700 to other computational devices (e. g. , servers and client devices) , via one or more network communication interfaces 750 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;
  • ⁇ Data synchronization module 704 that synchronizes data in a local database 705 with data in a remote database maintained at a data source server;
  • server system 700 includes at least data computation server 300, and data synchronization module 704 on server system 700 includes modules 301-305 included in data computation server 300. More details on data synchronization module 704 are explained above with reference to FIGS. 1-6, and particularly, FIGS. 2 and 3.
  • multiple functional modules are optionally integrated in one processing unit.
  • each functional module exists separately from other functional modules.
  • a subset of functional modules are integrated in one processing unit.
  • the integrated processing unit is implemented in the form of hardware, software, or a combination of both. All the aforementioned functional modules are optionally located at one terminal or network node, or distributed on a plurality of terminals or network nodes of a computer network.
  • each of the above functional modules is optionally implemented by a data processing program executed by a data processing device (e. g. , a computer) .
  • the data processing program usually stored in one storage medium, the program is directly read from the storage medium or the program is installed or duplicated to a storage device (for example, a hard disk and/or a memory) of the data processing device for execution.
  • the storage medium may use any type of recording manner such as a paper storage medium (such as a paper tape) , a magnetic storage medium (such as a floppy disk, a hard disk, and a flash memory) , an optical storage medium (such as a CD-ROM) , and a magneto-optical storage medium (such as an MO) .
  • the present application further discloses a storage medium, where a data processing program is stored in the storage medium, and the data processing program is configured to execute any one of the embodiments of the above method according to the present application.
  • the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting, " that a stated condition precedent is true, depending on the context.
  • the phrase “if it is determined [that a stated condition precedent is true] “ or “if [astated condition precedent is true] “ or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.
  • stages that are not order dependent may be reordered and other stages may be combined or broken out. While some reordering or other groupings are specifically mentioned, others will be obvious to those of ordinary skill in the art and so do not present an exhaustive list of alternatives. Moreover, it should be recognized that the stages could be implemented in hardware, firmware, software or any combination thereof.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
PCT/CN2014/086599 2013-09-30 2014-09-16 Data synchronization for remote and local databases WO2015043391A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201310460009.6A CN104519103B (zh) 2013-09-30 2013-09-30 网络数据的同步处理方法、服务器及相关系统
CN201310460009.6 2013-09-30

Publications (1)

Publication Number Publication Date
WO2015043391A1 true WO2015043391A1 (en) 2015-04-02

Family

ID=52742026

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/086599 WO2015043391A1 (en) 2013-09-30 2014-09-16 Data synchronization for remote and local databases

Country Status (2)

Country Link
CN (1) CN104519103B (zh)
WO (1) WO2015043391A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108197164A (zh) * 2017-12-14 2018-06-22 泰康保险集团股份有限公司 业务数据保存方法及装置
CN108573060A (zh) * 2018-04-26 2018-09-25 北京恒华伟业科技股份有限公司 一种基于spring boot插件的数据同步方法和系统
CN113220794A (zh) * 2016-02-29 2021-08-06 株式会社特迈数据 用于同步异构数据库之间的查询结果的方法、服务器和计算机可读介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105117423B (zh) * 2015-07-31 2018-08-21 中冶南方工程技术有限公司 一种基于智能体的数据库变更自主选择响应方法
CN105025110B (zh) * 2015-07-31 2018-05-08 中冶南方工程技术有限公司 一种多客户端数据修改消息精确推送方法
CN106888238B (zh) * 2015-12-15 2020-05-05 阿里巴巴集团控股有限公司 一种数据同步方法及装置
CN107291806B (zh) * 2017-05-16 2021-04-06 浙江工业大学 一种Web可视化环境中的数据视图副本迭代方法
CN111597402A (zh) * 2020-05-21 2020-08-28 北京字节跳动网络技术有限公司 更新图索引数据的方法、装置、电子设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101405729A (zh) * 2006-03-23 2009-04-08 微软公司 具有增量式视图维护的映射体系结构
CN101477524A (zh) * 2008-12-11 2009-07-08 金蝶软件（中国）有限公司 基于物化视图的系统性能优化方法及系统
US20120158795A1 (en) * 2010-12-21 2012-06-21 Sybase, Inc. Entity triggers for materialized view maintenance

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100487700C (zh) * 2006-07-11 2009-05-13 阿里巴巴集团控股有限公司 数据仓库中的数据处理方法及系统
CN102325153B (zh) * 2011-07-12 2014-08-06 北京新媒传信科技有限公司 一种服务开发方法和系统

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101405729A (zh) * 2006-03-23 2009-04-08 微软公司 具有增量式视图维护的映射体系结构
CN101477524A (zh) * 2008-12-11 2009-07-08 金蝶软件（中国）有限公司 基于物化视图的系统性能优化方法及系统
US20120158795A1 (en) * 2010-12-21 2012-06-21 Sybase, Inc. Entity triggers for materialized view maintenance

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113220794A (zh) * 2016-02-29 2021-08-06 株式会社特迈数据 用于同步异构数据库之间的查询结果的方法、服务器和计算机可读介质
CN108197164A (zh) * 2017-12-14 2018-06-22 泰康保险集团股份有限公司 业务数据保存方法及装置
CN108573060A (zh) * 2018-04-26 2018-09-25 北京恒华伟业科技股份有限公司 一种基于spring boot插件的数据同步方法和系统

Also Published As

Publication number Publication date
CN104519103A (zh) 2015-04-15
CN104519103B (zh) 2018-10-26

Similar Documents

Publication Publication Date Title
WO2015043391A1 (en) Data synchronization for remote and local databases
US10642861B2 (en) Multi-instance redo apply
Corbellini et al. Persisting big-data: The NoSQL landscape
US8589361B2 (en) Reduced disk space standby
US9529881B2 (en) Difference determination in a database environment
US9886443B1 (en) Distributed NFS metadata server
US20130110873A1 (en) Method and system for data storage and management
US20070174292A1 (en) Autonomic recommendation and placement of materialized query tables for load distribution
US20160019279A1 (en) High throughput data modifications using blind update operations
JP7263297B2 (ja) ハイブリッドクラウド弾性スケーリングおよび高性能データ仮想化のためのリアルタイムクロスシステムデータベースレプリケーション
CN107077453B (zh) 用于使用集群缓存进行数据库查询的并行优化的系统和方法
CN101184106A (zh) 一种移动数据库的关联事务处理方法
US10650024B2 (en) System and method of replicating data in a distributed system
CN106716409A (zh) 基于查询需求从行存储数据库自适应地构建和更新列存储数据库的方法和系统
US20150193526A1 (en) Schemaless data access management
CN104216955A (zh) 一种操作数据及管理事务的方法、装置及分布式系统
US11386078B2 (en) Distributed trust data storage system
US11048728B2 (en) Dependent object analysis
CN105426119A (zh) 一种存储设备及数据处理方法
US11144373B2 (en) Data pipeline using a pluggable topology connecting components without altering code of the components
US20210157814A1 (en) Complex query rewriting
CN114661690A (zh) 多版本并发控制和日志清除方法、节点、设备和介质
US20240143594A1 (en) Offloading graph components to persistent storage for reducing resident memory in distributed graph processing
US11429388B2 (en) Parallel load of mapping containers for database system start and restart operations
US20180173805A1 (en) Application programming interface for detection and extraction of data changes

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14848536

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase
32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 23/08/2016)

122 Ep: pct application non-entry in european phase

Ref document number: 14848536

Country of ref document: EP

Kind code of ref document: A1