WO2017016336A1 - 数据处理及查询方法、装置 - Google Patents

数据处理及查询方法、装置 Download PDF

Info

Publication number
WO2017016336A1
WO2017016336A1 PCT/CN2016/085558 CN2016085558W WO2017016336A1 WO 2017016336 A1 WO2017016336 A1 WO 2017016336A1 CN 2016085558 W CN2016085558 W CN 2016085558W WO 2017016336 A1 WO2017016336 A1 WO 2017016336A1
Authority
WO
WIPO (PCT)
Prior art keywords
olap
server
olap server
data
query
Prior art date
Application number
PCT/CN2016/085558
Other languages
English (en)
French (fr)
Inventor
吕燕
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017016336A1 publication Critical patent/WO2017016336A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the present invention relates to the field of communications, and in particular to a data processing and query method and apparatus.
  • OLAP Online Analytical Processing
  • Design enables users to perform fast, stable, and interactive access based on multidimensional models to achieve complex analysis and data prediction.
  • OLAP is a tool for data analysts. At this stage, many types of OLAP are stand-alone versions, which support a limited number of concurrent users. Therefore, in a high concurrency scenario, OLAP cannot process the user's analysis request; and as the data source size increases, when the fact table data size reaches the TB (Terabyte terabytes) level, OLAP will not be able to analyze the data.
  • TB Transmissionbyte terabytes
  • the present invention provides a data processing and query method and apparatus, so as to solve at least the problem that the OLAP existing in the related art cannot quickly respond to user requirements in the case of high concurrency and large amount of data stored in the database.
  • a data processing method comprising: a load balancing server receiving a query request for requesting query of predetermined data; the load balancing server selecting, from a cluster of online analytical processing OLAP servers, according to a selection policy An OLAP server that processes the query request, wherein the OLAP server cluster includes at least two OLAP servers and each of the OLAP server clusters are independent of each other; the load balancing server forwards the query request Give the OLAP server.
  • the method further includes: the load balancing server determining the selected OLAP server A failure occurs; the load balancing server selects a new OLAP server for processing the query request from other OLAP servers in the OLAP server cluster according to the selection policy; the load balancing server will The query request is forwarded to the new OLAP server.
  • the selection policy includes at least one of: a load condition of an OLAP server in the OLAP server cluster, and an OLAP server load policy.
  • a data query method comprising: an online analytical processing OLAP server receiving a query request sent by a data requester for requesting querying predetermined data, wherein the OLAP server is negative
  • the OLAP server queries the predetermined data according to the query request, including: the OLAP server queries the global data cache server for the predetermined data according to the query request, where the global data cache server Data is pre-cached, and the cached data is distributedly stored in more than two nodes of the global data cache server, the global data cache server supporting adding and/or deleting nodes.
  • the OLAP server after the OLAP server queries the predetermined data in the global data cache server according to the query request, the OLAP server includes the database when the predetermined data is not cached in the global data cache server. Querying the predetermined data; the OLAP server caches the queried predetermined data into the global data cache server.
  • the method further includes: determining, by the OLAP server, that a multi-dimensional model for querying data of one OLAP server in the OLAP server cluster is changed; the OLAP server is to use a multi-dimensional model of the OLAP server The changed multidimensional model is synchronized; and/or the OLAP server determines that the multidimensional model of the OLAP server for querying data has changed; the OLAP server sends to other OLAP servers in the OLAP server cluster a change notification, wherein the change notification is used to identify a change in a multidimensional model of the OLAP server.
  • the method further includes: the OLAP server synchronizing the multidimensional model of the OLAP server into a global data cache server.
  • the selection policy includes at least one of: a load condition of an OLAP server in the OLAP server cluster, and an OLAP server load policy.
  • a data processing apparatus which is applied to a load balancing server, and includes: a first receiving module configured to receive a query request for requesting query for predetermined data; and a first selecting module configured to The selection policy selects, from the online analytical processing OLAP server cluster, an OLAP server for processing the query request, wherein the OLAP server cluster includes at least two OLAP servers and each OLAP server in the OLAP server cluster is mutually Independent; a first forwarding module configured to forward the query request to the OLAP server.
  • the apparatus further includes: a first determining module, configured to determine that the selected OLAP server is faulty; and a second selecting module configured to select from other OLAP servers in the OLAP server cluster according to the selection policy Selecting a new OLAP server for processing the query request; and a second forwarding module configured to forward the query request to a new OLAP server.
  • a first determining module configured to determine that the selected OLAP server is faulty
  • a second selecting module configured to select from other OLAP servers in the OLAP server cluster according to the selection policy Selecting a new OLAP server for processing the query request
  • a second forwarding module configured to forward the query request to a new OLAP server.
  • the selection policy includes at least one of: a load condition of an OLAP server in the OLAP server cluster, and an OLAP server load policy.
  • a data query device which is applied to an online analytical processing OLAP server, and includes: a second receiving module configured to receive a query request sent by a data requester for requesting querying predetermined data,
  • the OLAP server is an OLAP server selected by the load balancing server from the OLAP server cluster according to the selection policy, the OLAP server cluster includes at least two OLAP servers, and each OLAP server in the OLAP server cluster is independent of each other;
  • a query module is configured to query the predetermined data according to the query request; and the returning module is configured to return the queried predetermined data to the data requester.
  • the first query module includes: querying the predetermined data in a global data cache server according to the query request, wherein the data is pre-cached in the global data cache server, and the cached data is Distributedly stored in more than two nodes of the global data cache server, the global data cache server supports adding and/or deleting nodes.
  • the device further includes: a second query module, configured to query the predetermined data from a database when the predetermined data is not cached in the global data cache server; and the cache module is configured to query The predetermined data is cached in the global data cache server.
  • a second query module configured to query the predetermined data from a database when the predetermined data is not cached in the global data cache server
  • the cache module is configured to query The predetermined data is cached in the global data cache server.
  • the apparatus further includes: a second determining module, configured to determine that a multi-dimensional model for querying data of one OLAP server in the OLAP server cluster has changed; a first synchronization module, configured to The multi-dimensional model of the OLAP server is synchronized with the changed multi-dimensional model; and/or the third determining module is configured to determine that the multi-dimensional model of the OLAP server for querying data has changed; the sending module is set to The other OLAP servers in the OLAP server cluster send change notifications, wherein the change notifications are used to identify changes in the multidimensional model of the OLAP server.
  • the apparatus further includes: a second synchronization module configured to synchronize the multidimensional model of the OLAP server into the global data cache server.
  • the selection policy includes at least one of: a load condition of an OLAP server in the OLAP server cluster, and an OLAP server load policy.
  • Another embodiment of the present invention provides a computer storage medium storing execution instructions for performing the method in the above embodiments.
  • a load balancing server is used to receive a query request for requesting query of predetermined data; and the load balancing server selects an OLAP server for processing the query request from an online analytical processing OLAP server cluster according to a selection policy, wherein
  • the OLAP server cluster includes at least two OLAP servers and each OLAP server in the OLAP server cluster is independent of each other; the load balancing server forwards the query request to the OLAP server, in an OLAP server cluster
  • Each OLAP server is independent of each other, and each OLAP server can perform data processing for the requesting party, which solves the problem that the OLAP existing in the related technology cannot quickly respond to the user's demand in the case of high concurrency and large amount of data stored in the database, and further It achieves the effect of responding quickly to user demands in the case of high concurrency and large amount of data stored in the database, and improves user experience.
  • FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention.
  • FIG. 2 is a flowchart of a data query method according to an embodiment of the present invention.
  • FIG. 3 is a block diagram showing the structure of a data processing apparatus according to an embodiment of the present invention.
  • FIG. 4 is a block diagram showing a preferred structure of a data processing apparatus according to an embodiment of the present invention.
  • FIG. 5 is a structural block diagram of a data query apparatus according to an embodiment of the present invention.
  • FIG. 6 is a block diagram showing a preferred structure of a data querying apparatus according to an embodiment of the present invention.
  • FIG. 7 is a block diagram showing another preferred structure of a data querying apparatus according to an embodiment of the present invention.
  • FIG. 8 is a block diagram showing still another preferred structure of a data query apparatus according to an embodiment of the present invention.
  • FIG. 9 is a component diagram of a multidimensional data analysis system in accordance with an embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of an OLAP cluster architecture component 92 in a multi-dimensional analysis system according to an embodiment of the present invention.
  • FIG. 11 is a block diagram showing the structure of a multi-dimensional model synchronization component 96 in a multi-dimensional analysis system according to an embodiment of the present invention
  • FIG. 12 is a schematic structural diagram of a data cache management component 910 in a multi-dimensional analysis system according to an embodiment of the present invention.
  • FIG. 1 is a flowchart of a data processing method according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps:
  • Step S102 The load balancing server receives a query request for requesting query for predetermined data.
  • Step S104 The load balancing server selects, from the online analytical processing OLAP server cluster, an OLAP server for processing the query request according to the selection policy, where the OLAP server cluster includes at least two OLAP servers and each OLAP server in the OLAP server cluster Independent;
  • Step S106 the load balancing server forwards the query request to the OLAP server.
  • the load balancing server receives a query request for requesting query for predetermined data; the load balancing server selects, from the online analytical processing OLAP server cluster, for processing the query request according to the selection policy.
  • An OLAP server wherein the OLAP server cluster includes at least two OLAP servers and each OLAP server in the OLAP server cluster is independent of each other; the load balancing server forwards the query request to the OLAP server, because each OLAP server in the OLAP server cluster is independent of each other,
  • Each OLAP server can serve as a stand-alone OLAP server for the requester's query request, and solves the problem that the OLAP existing in the related technology cannot quickly respond to the user's demand in the case of high concurrency and large amount of data stored in the database.
  • the effect of quickly responding to user demands in the case of high concurrency and large amount of data stored in the database is achieved, and the user experience is improved.
  • the data processing method may further include: the load balancing server determines that the selected OLAP server is faulty; and the load balancing server selects from other OLAP servers in the OLAP server cluster according to the selection policy. A new OLAP server for processing query requests; the load balancing server sends query requests to the new OLAP server.
  • the load balancing server can select a new OLAP server from the OLAP cluster architecture to serve the requester again, thereby improving the OLAP cluster architecture. The fault tolerance rate ensures that the requester's request can be satisfied.
  • the load balancing server may select an OLAP server from the OLAP server cluster according to the selection policy, and the selection policy may include at least one of the following: a load condition of the OLAP server in the OLAP server cluster, and an OLAP server load policy. .
  • each OLAP in the OLAP server cluster can be fully utilized, effectively preventing load imbalance in the OLAP server cluster.
  • FIG. 2 is a flowchart of a data query method according to an embodiment of the present invention. As shown in FIG. 2, the method includes the following steps:
  • Step S202 the online analysis processing OLAP server receives a query request sent by the data requester for requesting query for predetermined data, wherein the OLAP server is an OLAP server selected by the load balancing server from the OLAP server cluster according to the selection policy, and the OLAP server cluster includes At least two OLAP servers and each OLAP server in the OLAP server cluster are independent of each other;
  • Step S204 the OLAP server queries the predetermined data according to the query request
  • step S206 the OLAP server returns the queried predetermined data to the data requester.
  • the OLAP server receives a query request sent by the data requester for requesting querying the predetermined data, wherein the OLAP server is an OLAP server selected by the load balancing server from the OLAP server cluster according to the selection policy, and the OLAP server cluster includes at least two Each OLAP server and each OLAP server in the OLAP server cluster are independent of each other; the OLAP server queries the predetermined data according to the query request; the OLAP server returns the queried predetermined data to the data requester, because each OLAP server in the OLAP server cluster is independent of each other, Both can serve as a stand-alone OLAP server for the requester's query request, and solve the problem that the OLAP existing in the related technology cannot quickly respond to the user's demand in the case of high concurrency and large amount of data stored in the database, thereby achieving the problem. In the case of high concurrency and large amount of data stored in the database, the effect of responding to user requirements is quickly improved, and
  • Step S204 has various implementation manners, for example, according to data input and output (Input & Output, referred to as IO)
  • the interface queries the data from the database.
  • step S204 may include: the OLAP server queries the global data cache server for predetermined data according to the query request, wherein the global data cache server pre-caches the data, and the cached
  • the data is distributedly stored in more than two nodes of the global data cache server, which supports adding and/or deleting nodes.
  • the OLAP server can directly query data from the cache, reducing the number of data IO queries and improving the efficiency of the query.
  • the global data cache server adopts a distributed data storage mode, can store a large amount of data, and can also be extended, and can solve the problem of low query speed caused by querying the massive data of the requester to a certain extent.
  • the OLAP server queries the database from the database when the predetermined data is not cached in the global data cache server. Data; the OLAP server caches the scheduled data that is queried to the global data cache server.
  • the OLAP server fails to query the data in the global cache data server, the OLAP can also query the data from the database through the data IO query, thereby improving the reliability of the query data, and the query is The data is cached to the global data cache server, which ensures that the next time the data is queried, it can be directly searched from the global data cache server, which improves the efficiency of querying data.
  • the data query method may further include: determining, by the OLAP server, that a multidimensional model for querying data of an OLAP server in the OLAP server cluster has changed; the OLAP server is configured to use the multidimensional model of the OLAP server The changed multidimensional model is synchronized; and/or the OLAP server determines that the multidimensional model of the OLAP server for querying data has changed; the OLAP server sends a change notification to other OLAP servers in the OLAP server cluster, wherein the change notification is used A change in the multidimensional model that identifies the OLAP server.
  • the multidimensional model on an OLAP server may change due to user configuration, etc., but other OLAP servers in the OLAP server cluster have not changed, and the multidimensional changes through other OLAP servers
  • the model is synchronized, so that when the requesting party's query request is accessed again, each OLAP server can be selected only according to the selection strategy, and the multi-dimensional model on each OLAP server is not considered to be synchronized, thereby improving the efficiency of selection.
  • the OLAP server whose multi-dimensional model changes actively sends change notifications, which can improve the efficiency of synchronization.
  • the process of determining that the multi-dimensional model in one OLAP server in the OLAP server cluster changes and performs synchronization or notification in this embodiment may occur at any position of the data query method as shown in FIG. 2, for example, at step S202. Before or after, before or after step S206, it may be a real-time synchronization process or a timing synchronization process.
  • the OLAP server synchronizes the multidimensional model of the OLAP server to the global data cache server.
  • the data of the multidimensional model can be backed up, and each OLAP server can also synchronize the multidimensional model from the global data cache server.
  • the selection policy may include at least one of the following: a load condition of an OLAP server in an OLAP server cluster, and an OLAP server load policy.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • a storage medium such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
  • a data processing and querying device is provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has not been described again.
  • the term “module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • FIG. 3 is a structural block diagram of a data processing apparatus according to an embodiment of the present invention. As shown in FIG. 3, the apparatus is applied to a load balancing server, including a first receiving module 32, a first selecting module 34, and a first forwarding module 36. The device will be described.
  • the first receiving module 32 is configured to receive a query request for requesting querying the predetermined data.
  • the first selecting module 34 is connected to the first receiving module 32, and is configured to select a query request from the OLAP server cluster according to the selection policy.
  • the OLAP server is processed, wherein the OLAP server cluster includes at least two OLAP servers and each OLAP server in the OLAP server cluster is independent of each other; the first forwarding module 36 is connected to the first selection module 34, and is configured to query the query. Forward to the OLAP server.
  • FIG. 4 is a block diagram of a preferred structure of a data processing apparatus according to an embodiment of the present invention. As shown in FIG. 4, the apparatus includes a first determining module 42 and a second selecting module 44, in addition to all the modules shown in FIG.
  • the second forwarding module 46 will be described below.
  • the first determining module 42 is connected to the first selecting module 34, and is configured to determine that the selected OLAP server is faulty.
  • the second selecting module 44 is connected to the first determining module 42 and configured to be from the OLAP server cluster according to the selection policy.
  • the other OLAP servers select a new OLAP server for processing the query request;
  • the second forwarding module 46 is coupled to the second selection module 44 and configured to forward the query request to the new OLAP server.
  • the selection policy includes at least one of the following: a load condition of an OLAP server in an OLAP server cluster, and an OLAP server load policy.
  • FIG. 5 is a structural block diagram of a data query device according to an embodiment of the present invention. As shown in FIG. 5, the device is applied to an OLAP server, and includes a second receiving module 52. The first query module 54 and the return module 56 are described below.
  • the second receiving module 52 is configured to receive a query request sent by the data requester for requesting querying the predetermined data, where the OLAP server is an OLAP server selected by the load balancing server from the OLAP server cluster according to the selection policy, and the OLAP server cluster includes The at least two OLAP servers and the OLAP servers in the OLAP server cluster are independent of each other; the first query module 54 is connected to the second receiving module 52, and is configured to query predetermined data according to the query request; the returning module 56 is connected to the first The query module 54 is configured to return the queried predetermined data to the data Requester.
  • the first query module 54 may include: querying the predetermined data in the global data cache server according to the query request, wherein the data is pre-cached in the global data cache server, and the cached data is distributedly stored in the Among the two or more nodes of the global data cache server, the global data cache server supports adding and/or deleting nodes.
  • FIG. 6 is a block diagram of a preferred structure of a data query apparatus according to an embodiment of the present invention. As shown in FIG. 6, the apparatus includes: a second query module 62 and a cache module 64, in addition to all the modules shown in FIG. The device will be described.
  • the second query module 62 is connected to the first query module 54 and configured to query the predetermined data from the database when the predetermined data is not cached in the global data cache server.
  • the cache module 64 is connected to the second query module 62 and configured. To cache the scheduled data that is queried to the global data cache server.
  • the connection relationship shown in FIG. 6 is only an example. The connection relationship between the second query module 62 and the cache module 64 and the module shown in FIG. 5 may be various. For example, the return module 56 may also be connected to the second query. Module 62 is interposed between cache module 64.
  • FIG. 7 is another block diagram of another preferred structure of a data query apparatus according to an embodiment of the present invention.
  • the apparatus includes, in addition to all the modules shown in FIG. 5, a second determining module 72 and a first synchronization.
  • Module 74, and/or, further includes a third determination module 76 and a transmission module 78, the apparatus being described below.
  • the second determining module 72 is connected to the returning module 56, and is configured to determine that a multi-dimensional model for querying data of one OLAP server in the OLAP server cluster has changed; the first synchronization module 74 is connected to the second determining module 72. And the third determining module 76 is connected to the returning module 56, and is configured to determine that the multidimensional model of the OLAP server for querying data has changed; The module 78, coupled to the third determining module 76, is configured to send a change notification to other OLAP servers in the OLAP server cluster, wherein the change notification is used to identify changes in the multidimensional model of the OLAP server.
  • connection relationship shown in FIG. 7 is only an example.
  • the connection relationship between the second determination module 72 and each module in FIG. 5 may be multiple.
  • the second determination module 72 may be connected to the second receiving module 52.
  • the third determining module 76 can also be connected to the second receiving module 52 or connected to the first query module 54 to determine that the multi-dimensional structure of an OLAP server in the OLAP server cluster changes. And notification or synchronization can occur in the data query device in real time.
  • FIG. 8 is a block diagram of still another preferred structure of a data query apparatus according to an embodiment of the present invention. As shown in FIG. 8, the apparatus includes: a second synchronization module 82, in addition to all the modules shown in FIG. The device is described.
  • the second synchronization module 82 coupled to the first synchronization module 74 and/or the transmission module 78, is configured to synchronize the multidimensional model of the OLAP server to the global data cache server.
  • the foregoing selection policy may include at least one of the following: a load condition of an OLAP server in an OLAP server cluster, and an OLAP server load policy.
  • the embodiments of the present invention can be applied to a multi-dimensional data analysis system with high concurrency and massive data, and FIG. 9 is based on the present invention.
  • the multidimensional data analysis system component diagram of the embodiment, as shown in FIG. 9, the multidimensional data analysis system includes: a global data cache component 94 (corresponding to the global data cache server described above), and a multidimensional model synchronization component 96 (corresponding to the second receiving The module 52, the second determining module 72, the first synchronization module 74, the third determining module 76 and the sending module 78), the dynamic session cluster component 98 and the data cache management component 910 (corresponding to the first query module 54, the second query) Module 62, cache module 64 and return module 56).
  • a global data cache component 94 corresponding to the global data cache server described above
  • a multidimensional model synchronization component 96 corresponding to the second receiving The module 52, the second determining module 72, the first synchronization module 74, the third determining module 76 and the sending module 78
  • the multidimensional data analysis system is also an OLAP cluster architecture component 92 (equivalent to the above OLAP server cluster). Since the OLAP cluster architecture component 92 is composed of distributed, unowned architectures of various OLAP servers, it is not shown in FIG. A component diagram of an OLAP server is explicitly shown in FIG. 9. The other OLAP servers are identical to the component diagrams of the OLAP server in FIG. 9, and each OLAP server constitutes an OLAP cluster architecture component 92.
  • the multidimensional data analysis system will be described below.
  • OLAP cluster architecture component 92 OLAP cluster architecture adopts a masterless cluster architecture, and each OLAP instance (ie, OLAP server) is an independent service instance.
  • the OLAP cluster implements the functions of unified service address, service forwarding, and load sharing through the load balancing server.
  • Global data cache component 94 global data cache component 94 is a distributed, scalable architecture. As the data source size increases, OLAP can utilize the global data cache component 94 to improve the data cache ceiling caused by increased data size.
  • the Segment type has the largest amount of data. Its structure contains models, dimension members, column axes, The row axis data, and the multi-Dimensional eXpressions (MDX) information that is initiated.
  • MDX multi-Dimensional eXpressions
  • Each service instance in the OLAP cluster first hits the data from the global cache data component 94 when performing data analysis. If the data is not found, the data storage IO query data is initiated, and the data analyzed in this time is stored in the global cache data. Component 94, the number of data storage IO requests will be further reduced.
  • OLAP puts the data in the analysis process into memory, and the subsequent analysis requests first hit the data from the memory, reducing the number of requests for data storage IO.
  • the multi-dimensional model synchronization component 96 the multi-dimensional model synchronization component 96 is responsible for tracking and calculating the stability of the cluster members, and synchronizing other service instances with the service instance with the latest and most complete multi-dimensional model as a reference.
  • a multidimensional model is configured on each OLAP service instance, and each OLAP service instance can independently receive an analysis request.
  • the multidimensional model synchronization component 96 avoids inconsistencies in multidimensional models on different service instances in an OLAP cluster.
  • the dynamic session cluster component 98 ensures that the session is not lost, provided that the data in the session is serializable.
  • the data cache management component 910, the data cache management component 910 functions include: hit data, data cache, cache data update.
  • the user initiates a data analysis operation. After the operation parameters are parsed, the data cache management component 910 is sent to the data cache management component 910 to search according to the model-dimension member-data segment order, and the matching data is returned.
  • the data cache management component 910 caches the new user data analysis results into the global cache and updates the global cache data.
  • Data source changes such as dimension member data update, metric data update
  • data cache management component 910 is responsible for cleaning up the corresponding data in the global cache to ensure data accuracy.
  • the user-initiated data query request is received by the multi-bit model synchronization component 96, and the data query request is sent to the data cache management component 910, and the data cache management component 910 requests the global data cache according to the data query request.
  • the data is looked up in component 94. If the data fails to be found, data cache management component 910 queries the data in the database based on data IO and caches the queried data to global data cache component 94.
  • the dynamic session cluster component 98 will confirm that the OLAP service instance has failed, and the load balancing server again selects the OLAP service instance to serve the user.
  • the multidimensional model synchronization component 96 in each OLAP service instance synchronizes the changed multidimensional model. Thereby the user's services in each OLAP service instance are guaranteed.
  • each OLAP server in the OLAP cluster architecture component 92 is independent of each other, and each OLAP server can be used for a user.
  • the data query service, the OLAP cluster architecture component 92, and the global data cache component 94, the multidimensional model synchronization component 96, the dynamic session cluster component 98, and the data cache management component 910 are one of the OLAP cluster architecture components 92. The role of the service instance is explained.
  • the embodiment is designed according to the componentization method, and can realize multi-dimensional analysis of high concurrency and massive data.
  • the OLAP cluster architecture component 92 is an OLAP server without a primary cluster, and each OLAP server is an independent service instance.
  • the load balancing software is set up at the access layer, and the load balancing software implements the load balancing capability service of the HTTP layer. It publishes a unified service address; it is responsible for client access, and connects the client to the OLAP service instance according to the server load and server load policy; it is responsible for service jump, and forwards the client to the corresponding function according to the client's request. On the server; it is responsible for server disaster recovery. If the server being serviced is down, the multi-dimensional analysis operation initiated by the user who has previously transferred to the server can jump to other serviceable instances.
  • Global data cache component 94 global data cache component 94 is an extensible data distribution and clustering platform that supports object level data caching. Each node in the cluster has the same status and no main architecture; data and data backup are distributed; nodes are dynamically added to remove the cluster, and nodes are aware of each other.
  • the global data cache component 94 provides security mechanisms such as a socket interceptor, authenticates the access node, and controls the node to access the cluster.
  • the global data is basically distributed among the nodes in the cluster.
  • a node member When a node member is down, its backup data copy contains the same data, and the data copy is redistributed on the remaining active nodes, so that no data will be lost. .
  • the new node When a node newly joins the global data cache cluster, the new node assigns responsibility for data access and loads a portion of all data.
  • each global data cache node can be deployed in a unified manner with each OLAP service instance, that is, each running OLAP service instance has a global data cache function. This can omit the installation, deployment, and security authentication of the global data cache component 94.
  • FIG. 11 is a structural block diagram of a multi-dimensional model synchronization component 96 in a multi-dimensional analysis system according to an embodiment of the present invention.
  • the multi-dimensional model synchronization component 96 includes a monitoring module 112, a stability calculation module 114, and a multi-dimensional model update module 116. .
  • the multidimensional model synchronization components 96 on each OLAP service instance can communicate with each other.
  • the monitoring module 112 is configured to monitor the multi-dimensional model change notification, and the multi-dimensional model change on a certain OLAP service instance sends a broadcast notification to perform the multi-dimensional model update of the example.
  • the monitoring module monitors the link between the instance and other instances, and the stability calculation module 114 is configured to calculate the stability coefficient of the present example and other instances, and select the OLAP service instance with the highest stability.
  • the multidimensional model update module 116 configured to interact with the multidimensional model adaptation component, is responsible for updating the multidimensional model of the present example.
  • Dynamic session cluster component 98 which interacts with global data cache component 94, and user session data is stored in a global data cache.
  • the dynamic session cluster component 98 ensures that the session is not lost, and the user is shielded from the disaster recovery policy of the OLAP cluster.
  • the steps before and after the multi-dimensional analysis operation are highly correlated.
  • the complete multi-dimensional model analysis data should be placed in the user session, including the model and the data source corresponding to the current analysis.
  • the MDX statement currently being analyzed, the current data analysis result, the filtering information, the sorting information, whether the row or column is switched, etc., and the user session is guaranteed to be serializable.
  • FIG. 12 is a schematic structural diagram of a data cache management component 910 in a multi-dimensional analysis system according to an embodiment of the present invention. As shown in FIG. 12, the data cache management component 910 includes: a global data cache proxy 122 and a data cache loader 124. The data cache management component 910 is described.
  • the global data caching agent 122 connects to the external global data cache.
  • the main functions of the global data caching agent 122 are the get (GET) and deposit (PUT) functions.
  • the data cache loader 124 interacts with the global data caching agent 122. When loading data, it first looks up the data from the cache. If there is no hit, it interacts with the data source adaptation component to obtain the data and put the data into the cache.
  • the data cache management component 910 interacts with the data source adaptation component. When the data source is sent, the data cache loader hits the data and cleans it up.
  • the OLAP server cluster and the global data distributed node cache method can be used. Users can efficiently perform online multidimensional data analysis in high concurrency and massive data storage mode.
  • each of the above modules may be implemented by software or hardware.
  • the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the modules are located in multiple In the processor.
  • Embodiments of the present invention also provide a storage medium.
  • the foregoing storage medium may be configured to store program code for performing the following steps:
  • the storage medium is further arranged to store program code for performing the following steps:
  • the OLAP server is an OLAP server selected by the load balancing server from the OLAP server cluster according to the selection policy, and the OLAP server cluster includes at least two OLAP servers and Each OLAP server in the OLAP server cluster is independent of each other;
  • the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), and a Random Access Memory (RAM).
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
  • the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
  • the invention is not limited to any specific combination of hardware and software.
  • the data processing and query method and apparatus provided by the embodiments of the present invention have the following beneficial effects: the OLAP existing in the related art cannot solve the user quickly in the case of high concurrency and large amount of data stored in the database.
  • the problem of demand in turn, achieves the effect of quickly responding to user demands in the case of high concurrency and large amount of data stored in the database, and improves the user experience.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本发明提供了一种数据处理及查询方法、装置，其中，该数据处理方法包括：负载均衡服务器接收用于请求查询预定数据的查询请求；负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器，其中，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；负载均衡服务器将查询请求转发给OLAP服务器。通过本发明，解决了相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题，进而达到了在高并发、数据库中存储的数据量大的情况下快速响应用户需求的效果，提高了用户的体验度。

Description

数据处理及查询方法、装置 技术领域
本发明涉及通信领域，具体而言，涉及一种数据处理及查询方法、装置。
背景技术
联机分析处理(Online Analytical Processing，简称为OLAP)是数据仓库系统的一种应用，它针对特定的分析主题，设计多种可能的观察方式，设计相应的分析主题结构(即进行事实表和维度表设计)，使用户在多维模型的基础上进行快速、稳定、交互式的访问，以达到复杂分析和数据预测的作用。
OLAP作为面向数据分析人员的工具，现阶段很多种类的OLAP都是单机版，支持用户并发数有限。因此，在高并发场景下，OLAP无法处理用户的分析请求；并且随着数据源规模增大，当事实表数据量规模达到TB(Terabyte万亿字节)级别规模，OLAP将无法分析数据。
针对相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题，目前尚未提出有效的解决方案。
发明内容
本发明提供了一种数据处理及查询方法、装置，以至少解决相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题。
根据本发明的一个方面，提供了一种数据处理方法，包括：负载均衡服务器接收用于请求查询预定数据的查询请求；所述负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器，其中，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；所述负载均衡服务器将所述查询请求转发给所述OLAP服务器。
可选地，所述负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器之后，还包括：所述负载均衡服务器确定选择的所述OLAP服务器发生故障；所述负载均衡服务器根据所述选择策略从所述OLAP服务器集群中的其他OLAP服务器中选择一个新的用于对所述查询请求进行处理的OLAP服务器；所述负载均衡服务器将所述查询请求转发给新的OLAP服务器。
可选地，所述选择策略包括以下至少之一：所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
根据本发明的另一个方面，提供了一种数据查询方法，包括：联机分析处理OLAP服务器接收数据请求方发送的用于请求查询预定数据的查询请求，其中，所述OLAP服务器为负 载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；所述OLAP服务器根据所述查询请求查询所述预定数据；所述OLAP服务器将查询到的所述预定数据返回给所述数据请求方。
可选地，所述OLAP服务器根据所述查询请求查询所述预定数据，包括：所述OLAP服务器根据所述查询请求在全局数据缓存服务器中查询所述预定数据，其中，所述全局数据缓存服务器中预先缓存有数据，且缓存的所述数据被分布地存储在所述全局数据缓存服务器的两个以上节点中，所述全局数据缓存服务器支持增加和/或删除节点。
可选地，所述OLAP服务器根据所述查询请求在全局数据缓存服务器中查询所述预定数据之后，包括：当所述全局数据缓存服务器中未缓存所述预定数据时，所述OLAP服务器从数据库中查询所述预定数据；所述OLAP服务器将查询到的所述预定数据缓存至所述全局数据缓存服务器中。
可选地，所述方法还包括：所述OLAP服务器确定所述OLAP服务器集群中的一个OLAP服务器的用于查询数据的多维模型发生了变化；所述OLAP服务器将所述OLAP服务器的多维模型与发生变化的多维模型进行同步处理；和/或，所述OLAP服务器确定所述OLAP服务器的用于查询数据的多维模型发生了变化；所述OLAP服务器向所述OLAP服务器集群中的其他OLAP服务器发送变更通知，其中，所述变更通知用于标识所述OLAP服务器的多维模型发生的变化。
可选地，所述方法还包括：所述OLAP服务器将所述OLAP服务器的多维模型同步到全局数据缓存服务器中。
可选地，所述选择策略包括以下至少之一：所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
根据本发明的另一个方面，提供了一种数据处理装置，应用于负载均衡服务器，包括：第一接收模块，设置为接收用于请求查询预定数据的查询请求；第一选择模块，设置为根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器，其中，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；第一转发模块，设置为将所述查询请求转发给所述OLAP服务器。
可选地，所述装置还包括：第一确定模块，设置为确定选择的所述OLAP服务器发生故障；第二选择模块，设置为根据所述选择策略从所述OLAP服务器集群中的其他OLAP服务器中选择一个新的用于对所述查询请求进行处理的OLAP服务器；第二转发模块，设置为将所述查询请求转发给新的OLAP服务器。
可选地，所述选择策略包括以下至少之一：所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
根据本发明的再一个方面，还提供了一种数据查询装置，应用于联机分析处理OLAP服务器，包括：第二接收模块，设置为接收数据请求方发送的用于请求查询预定数据的查询请求，其中，所述OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；第一查询模块，设置为根据所述查询请求查询所述预定数据；返回模块，设置为将查询到的所述预定数据返回给所述数据请求方。
可选地，所述第一查询模块包括：根据所述查询请求在全局数据缓存服务器中查询所述预定数据，其中，所述全局数据缓存服务器中预先缓存有数据，且缓存的所述数据被分布地存储在所述全局数据缓存服务器的两个以上节点中，所述全局数据缓存服务器支持增加和/或删除节点。
可选地，所述装置还包括：第二查询模块，设置为当所述全局数据缓存服务器中未缓存所述预定数据时，从数据库中查询所述预定数据；缓存模块，设置为将查询到的所述预定数据缓存至所述全局数据缓存服务器中。
可选地，所述装置还包括：第二确定模块，设置为确定所述OLAP服务器集群中的一个OLAP服务器的用于查询数据的多维模型发生了变化；第一同步模块，设置为将所述OLAP服务器的多维模型与发生变化的多维模型进行同步处理；和/或，第三确定模块，设置为确定所述OLAP服务器的用于查询数据的多维模型发生了变化；发送模块，设置为向所述OLAP服务器集群中的其他OLAP服务器发送变更通知，其中，所述变更通知用于标识所述OLAP服务器的多维模型发生的变化。
可选地，所述装置还包括：第二同步模块，设置为将所述OLAP服务器的多维模型同步到全局数据缓存服务器中。
可选地，所述选择策略包括以下至少之一：所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
本发明另一实施例提供了一种计算机存储介质，所述计算机存储介质存储有执行指令，所述执行指令用于执行上述实施例中的方法。
通过本发明，采用负载均衡服务器接收用于请求查询预定数据的查询请求；所述负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器，其中，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；所述负载均衡服务器将所述查询请求转发给所述OLAP服务器的方法，OLAP服务器集群中的各个OLAP服务器相互独立，各个OLAP服务器都可以为请求方进行数据处理，解决了相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题，进而达到了在高并发、数据库中存储的数据量大的情况下快速响应用户需求的效果，提高了用户的体验度。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是根据本发明实施例的数据处理方法的流程图；
图2是根据本发明实施例的数据查询方法的流程图；
图3是根据本发明实施例的数据处理装置的结构框图；
图4是根据本发明实施例的数据处理装置的优选结构框图；
图5是根据本发明实施例的数据查询装置的结构框图；
图6是根据本发明实施例的数据查询装置的优选结构框图；
图7是根据本发明实施例的数据查询装置的另一优选结构框图；
图8是根据本发明实施例的数据查询装置的再一优选结构框图；
图9是根据本发明实施例的多维数据分析系统组件图；
图10是根据本发明实施例的多维分析系统中OLAP集群架构组件92的结构示意图；
图11是根据本发明实施例中多维分析系统中多维模型同步组件96的结构框图；
图12是根据本发明实施例的多维分析系统中数据缓存管理组件910的结构示意图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
在本实施例中提供了一种数据处理方法，图1是根据本发明实施例的数据处理方法的流程图，如图1所示，该流程包括如下步骤：
步骤S102，负载均衡服务器接收用于请求查询预定数据的查询请求；
步骤S104，负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对查询请求进行处理的OLAP服务器，其中，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；
步骤S106，负载均衡服务器将查询请求转发给OLAP服务器。
通过上述步骤，负载均衡服务器接收用于请求查询预定数据的查询请求；负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的 OLAP服务器，其中，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；负载均衡服务器将查询请求转发给OLAP服务器，因为OLAP服务器集群中的各个OLAP服务器相互独立，各个OLAP服务器都可以作为一个独立的OLAP服务器为请求方的查询请求进行服务，解决了相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题，进而达到了在高并发、数据库中存储的数据量大的情况下快速响应用户需求的效果，提高了用户的体验度。
在一个可选实施例中，在上述步骤S104之后，该数据处理方法还可以包括：负载均衡服务器确定选择的OLAP服务器发生故障；负载均衡服务器根据选择策略从OLAP服务器集群中的其他OLAP服务器中选择一个新的用于对查询请求进行处理的OLAP服务器；负载均衡服务器将查询请求发送给新的OLAP服务器。在该可选实施例中，当处理当前查询请求的OLAP服务器发生故障之后，负载均衡服务器可以再次从OLAP集群架构中选择一个新的OLAP服务器为请求方进行服务，从而提高了该OLAP集群架构的容错率，保证了请求方的请求可以得到满足。
在一个可选的实施例中，负载均衡服务器可以根据选择策略从OLAP服务器集群中选择OLAP服务器，该选择策略可以包括以下至少之一：OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。在该可选实施例中，可以使得OLAP服务器集群中的各个OLAP得到充分利用，有效防止了OLAP服务器集群中负载失衡。
在本发明实施例中还提供了一种数据查询方法，图2是根据本发明实施例的数据查询方法的流程图，如图2所示，该方法包括以下步骤：
步骤S202，联机分析处理OLAP服务器接收数据请求方发送的用于请求查询预定数据的查询请求，其中，OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；
步骤S204，OLAP服务器根据查询请求查询预定数据；
步骤S206，OLAP服务器将查询到的预定数据返回给数据请求方。
通过上述步骤，OLAP服务器接收数据请求方发送的用于请求查询预定数据的查询请求，其中，OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；OLAP服务器根据查询请求查询预定数据；OLAP服务器将查询到的预定数据返回给数据请求方，因为OLAP服务器集群中的各个OLAP服务器相互独立，都可以作为一个独立的OLAP服务器为请求方的查询请求进行服务，解决了相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题，进而达到了在高并发、数据库中存储的数据量大的情况下快速响应用户需求的效果，提高了用户的体验度。
步骤S204有多种实现方式，例如可以根据数据输入输出(Input&Output，简称为IO) 接口从数据库中查询数据，在一个可选实施例中，步骤S204可以包括：OLAP服务器根据查询请求在全局数据缓存服务器中查询预定数据，其中，全局数据缓存服务器中预先缓存有数据，且缓存的数据被分布地存储在全局数据缓存服务器的两个以上节点中，全局数据缓存服务器支持增加和/或删除节点。在该可选实施例中，OLAP服务器可以直接从缓存中查询数据，减少了数据IO查询的次数，提高了查询的效率。并且，全局数据缓存服务器中采取分布式的数据存储方式，能够存储海量数据，还可扩展，能够在一定程度上解决请求方的海量数据的查询导致的查询速度低的问题。
在一个可选的实施例中，OLAP服务器根据查询请求在全局数据缓存服务器中查询预定数据之后，包括：当所述全局数据缓存服务器中未缓存预定数据时，OLAP服务器从数据库中查询所述预定数据；OLAP服务器将查询到的预定数据缓存至全局数据缓存服务器中。在该可选实施例中，当OLAP服务器在全局缓存数据服务器中查询数据失败的情况下，OLAP还可以通过数据IO查询从数据库中查询数据，从而提高了查询数据的可靠性，并且，将查询到的数据缓存到全局数据缓存服务器，保证了下次查询该数据时可以直接从全局数据缓存服务器中查找，提高了查询数据的效率。
在一个可选的实施例中，该数据查询方法，还可以包括：OLAP服务器确定OLAP服务器集群中的一个OLAP服务器的用于查询数据的多维模型发生了变化；OLAP服务器将OLAP服务器的多维模型与发生变化的多维模型进行同步处理；和/或，OLAP服务器确定OLAP服务器的用于查询数据的多维模型发生了变化；OLAP服务器向OLAP服务器集群中的其他OLAP服务器发送变更通知，其中，变更通知用于标识OLAP服务器的多维模型发生的变化。在该可选实施例中，由于用户配置等原因，某一OLAP服务器上的多维模型可能会发生变化，但是，OLAP服务器集群中的其他OLAP服务器却没有发生变化，通过其他OLAP服务器对变化的多维模型进行同步，使得再一次接入请求方的查询请求时，可以只根据选择策略选择各个OLAP服务器，不必考虑各个OLAP服务器上的多维模型是否同步的问题，提高了选择的效率。同时，多维模型发生变化的OLAP服务器主动发送变更通知，可以提高同步的效率。
该实施例中的确定OLAP服务器集群中的一个OLAP服务器中的多维模型发生变化，并进行同步或者通知的过程，可以发生在如图2所示的数据查询方法的任何位置，例如发生在步骤S202之前或之后、步骤S206之前或者之后，它可以是一个实时的同步过程，也可以是定时的同步过程。
在一个可选的实施例中，OLAP服务器将OLAP服务器的多维模型同步到全局数据缓存服务器中。从而可以将多维模型的数据进行备份，同时各个OLAP服务器也可以从全局数据缓存服务器中对多维模型进行同步。
在一个可选的实施例中，选择策略可以包括以下至少之一：OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。从而使得OLAP服务器集群中的各个OLAP服务器得到有效利用，防止了OLAP服务器集群的负载失衡。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方 法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。
在本实施例中还提供了一种数据处理及查询装置，该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图3是根据本发明实施例的数据处理装置的结构框图，如图3所示，该装置应用于负载均衡服务器，包括第一接收模块32、第一选择模块34及第一转发模块36，下面对该装置进行说明。
第一接收模块32，设置为接收用于请求查询预定数据的查询请求；第一选择模块34，连接于上述第一接收模块32，设置为根据选择策略从OLAP服务器集群中选择用于对查询请求进行处理的OLAP服务器，其中，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；第一转发模块36，连接于上述第一选择模块34，设置为将查询请求转发给OLAP服务器。
图4是根据本发明实施例的数据处理装置的优选结构框图，如图4所示，该装置除包括图3所示的所有模块外，还包括第一确定模块42、第二选择模块44及第二转发模块46，下面对该装置进行说明。
第一确定模块42，连接于上述第一选择模块34，设置为确定选择的OLAP服务器发生故障；第二选择模块44，连接于上述第一确定模块42，设置为根据选择策略从OLAP服务器集群中的其他OLAP服务器中选择一个新的用于对所述查询请求进行处理的OLAP服务器；第二转发模块46，连接于上述第二选择模块44，设置为将查询请求转发给新的OLAP服务器。
在一个可选的实施例中，选择策略包括以下至少之一：OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
本发明实施例还提供了一种数据查询装置，图5是根据本发明实施例的数据查询装置的结构框图，如图5所示，该装置应用于OLAP服务器中，包括第二接收模块52、第一查询模块54及返回模块56，下面对该装置进行说明。
第二接收模块52，设置为接收数据请求方发送的用于请求查询预定数据的查询请求，其中，OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；第一查询模块54，连接于上述第二接收模块52，设置为根据查询请求查询预定数据；返回模块56，连接于上述第一查询模块54，设置为将查询到的预定数据返回给数据 请求方。
在一个可选实施例中，第一查询模块54可以包括：根据查询请求在全局数据缓存服务器中查询预定数据，其中，全局数据缓存服务器中预先缓存有数据，且缓存的数据被分布地存储在全局数据缓存服务器的两个以上节点中，全局数据缓存服务器支持增加和/或删除节点。
图6是根据本发明实施例的数据查询装置的优选结构框图，如图6所示，该装置除包括图5所示的所有模块外，还包括：第二查询模块62和缓存模块64，下面对该装置进行说明。
第二查询模块62，连接于上述第一查询模块54，设置为当全局数据缓存服务器中未缓存预定数据时，从数据库中查询预定数据；缓存模块64，连接于上述第二查询模块62，设置为将查询到的预定数据缓存至全局数据缓存服务器中。图6所示的连接关系仅仅是一种示例，第二查询模块62和缓存模块64与图5中所示的模块的连接关系可以有多种，例如，返回模块56也可以连接于第二查询模块62与缓存模块64之间。
图7是根据本发明实施例的数据查询装置的另一优选结构框图，如图7所示，该装置除包括图5所示的所有模块外，还包括：第二确定模块72和第一同步模块74，和/或，还包括第三确定模块76和发送模块78，下面对该装置进行说明。
第二确定模块72，连接于上述返回模块56，设置为确定OLAP服务器集群中的一个OLAP服务器的用于查询数据的多维模型发生了变化；第一同步模块74，连接于上述第二确定模块72，设置为将OLAP服务器的多维模型与发生变化的多维模型进行同步处理；第三确定模块76，连接于上述返回模块56，设置为确定OLAP服务器的用于查询数据的多维模型发生了变化；发送模块78，连接于上述第三确定模块76，设置为向OLAP服务器集群中的其他OLAP服务器发送变更通知，其中，变更通知用于标识OLAP服务器的多维模型发生的变化。
图7中所示的连接关系仅是一种示例，第二确定模块72和图5中的各个模块的连接关系可以是多种，例如，该第二确定模块72可以连接于第二接收模块52或者连接于第一查询模块54，同理，第三确定模块76还可以连接于第二接收模块52或者连接于第一查询模块54，确定OLAP服务器集群中的某一OLAP服务器的多维结构发生变化并进行通知或者同步，可以实时的发生在该数据查询装置中。
图8是根据本发明实施例的数据查询装置的再一优选结构框图，如图8所示，该装置除包括图7所示的所有模块外，还包括：第二同步模块82，下面对该装置进行说明。
第二同步模块82，连接于上述第一同步模块74和/或发送模块78，设置为将OLAP服务器的多维模型同步到全局数据缓存服务器中。
在一个可选的实施例中，上述选择策略可以包括以下至少之一：OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
下面结合实际应用环境对本发明实施例进行说明。
本发明实施例可以应用于高并发、海量数据的多维数据分析系统，图9是根据本发明实 施例的多维数据分析系统组件图，如图9所示，该多维数据分析系统包括：全局数据缓存组件94(相当于上述全局数据缓存服务器)、多维模型同步组件96(相当于上述第二接收模块52、第二确定模块72、第一同步模块74、第三确定模块76和发送模块78)、动态会话集群组件98和数据缓存管理组件910(相当于上述第一查询模块54、第二查询模块62、缓存模块64和返回模块56)。该多维数据分析系统还OLAP集群架构组件92(相当于上述OLAP服务器集群)，由于该OLAP集群架构组件92是由各个OLAP服务器分布式、无主架构构成的，没有在图9中展示，仅在图9中明确展示了一个OLAP服务器的组件图，其他OLAP服务器与图9中的OLAP服务器的组件图相同，各个OLAP服务器构成OLAP集群架构组件92。下面对该多维数据数据分析体统进行说明。
OLAP集群架构组件92，OLAP集群架构采用无主集群架构方式，每个OLAP实例(即OLAP服务器)都是一个独立的服务实例。OLAP集群对外通过负载均衡服务器实现统一服务地址、服务前转、负荷分担的功能。
全局数据缓存组件94，全局数据缓存组件94是一个分布式、可扩展架构。当数据源规模增大，OLAP可以利用全局数据缓存组件94改进数据规模增大引起的数据缓存上限问题。
OLAP放入内存中的数据分为三种：模型(Schema)、维度成员(Member)、数据段(Segment)，其中Segment类型的数据量最大，它的结构包含了模型、维度成员、列轴、行轴数据，以及发起的多维分析查询语句(Multi-Dimensional eXpressions，简称为MDX)信息等。
OLAP集群中的各个服务实例在进行数据分析时，先从全局缓存数据组件94中命中数据，如果找不到数据，则发起数据存储IO查询数据，并将本次分析的数据存入全局缓存数据组件94，数据存储IO请求次数将进一步减少。
为了提高数据分析效率，OLAP将分析过程中的数据放入内存中，后续发起的分析请求先从内存中命中数据，减少了数据存储IO的请求次数。
多维模型同步组件96，多维模型同步组件96负责跟踪、计算集群成员稳定性，以具备最新、最完整的多维模型的服务实例为基准，对其它服务实例进行同步。
无主架构OLAP服务器集群中，每个OLAP服务实例上都配置了多维模型，每个OLAP服务实例可以独立接收分析请求。多维模型同步组件96避免了OLAP集群中不同服务实例上的多维模型出现不一致的现象。
动态会话集群组件98，动态会话集群组件98保证会话不丢失，前提条件是会话中的数据是可序列化的。
假设我们有3个OLAP服务实例A、B、C，当A宕机时，A上的用户发起的请求会转向到B或C，A上的用户会话不会丢失，用户可以继续发起OLAP分析操作。
数据缓存管理组件910，数据缓存管理组件910功能包括：命中数据，数据缓存，缓存数据更新。
用户发起数据分析操作，操作参数经过解析后，传入数据缓存管理组件910，按照模型-维度成员-数据段顺序查找，找到匹配数据便返回。
数据缓存管理组件910将新的用户数据分析结果缓存到全局缓存中，更新全局缓存数据。
数据源变更，例如维度成员数据更新、度量指标数据更新，数据缓存管理组件910负责清理全局缓存中相应数据，以保证数据的准确性。
在本发明上述实施例中，通过多位模型同步组件96接收用户发起的数据查询请求，将该数据查询请求发送给数据缓存管理组件910，数据缓存管理组件910根据该数据查询请求在全局数据缓存组件94中查找数据，如果查找数据失败，则数据缓存管理组件910根据数据IO查询数据库中的数据，并将查询到的数据缓存至全局数据缓存组件94。同时，在当前服务的OLAP服务实例发生故障的情况下，动态会话集群组件98会确认该OLAP服务实例发生故障，由负载均衡服务器再次选择OLAP服务实例为用户进行服务。并且，在当前服务的OLAP服务实例的多维模型发生变更的情况下，各个OLAP服务实例中的多维模型同步组件96会对该发生变更的多维模型进行同步。从而使得用户在各个OLAP服务实例的服务都得到保障。
图10是根据本发明实施例的多维分析系统中OLAP集群架构组件92的结构示意图，如图10所示，OLAP集群架构组件92中的各个OLAP服务器相互独立，每个OLAP服务器都可以为用户进行数据查询服务，下面对该OLAP集群架构组件92，以及全局数据缓存组件94、多维模型同步组件96、动态会话集群组件98、数据缓存管理组件910在该OLAP集群架构组件92中的其中一个OLAP服务实例上的作用进行说明。
本实施例按照组件化方法设计，可以实现高并发、海量数据的多维分析。
OLAP集群架构组件92，为OLAP服务器无主集群，每个OLAP服务器是独立的服务实例。
在接入层架设负载均衡软件，负载均衡软件实现HTTP层的负载均衡能力服务。它对外发布统一服务地址；它负责客户端的接入，根据服务器负载情况和服务器负荷策略将客户端接入到OLAP服务实例；它负责服务跳转，根据客户端的请求将客户前转到对应的功能服务器上；它负责服务器容灾，若正在服务的服务器宕机，原前转到该服务器的用户后续发起的多维分析操作可以跳转到其它可服务的实例上去。
全局数据缓存组件94，全局数据缓存组件94是一个可扩展的数据分发和集群平台，支持对象级数据缓存。集群中每个节点地位相同，无主架构；数据和数据备份都是分布式的；节点动态加入移除集群，节点之间相互感知存在。
全局数据缓存组件94提供安全机制如套接字(Socket)截获器，对接入节点进行身份认证，控制节点接入集群。
全局数据基本是平均分布在集群中的各个节点中，当一个节点成员宕机，它的备份数据副本含有相同的数据，数据副本会重新分布在剩下的活动节点上，这样就没有数据会丢失。 当一个节点新加入全局数据缓存集群，新节点会赋予数据存取的责任，加载所有数据的一部分。
在一个可选的实施例中，可以进行每个全局数据缓存节点与每个OLAP服务实例合一部署，即每个运行的OLAP服务实例同时具备全局数据缓存功能。这样可以省略全局数据缓存组件94安装、部署、安全认证的工作。
图11是根据本发明实施例中多维分析系统中多维模型同步组件96的结构框图，如图11所示，该多维模型同步组件96包括监控模块112、稳定性计算模块114、多维模型更新模块116。各个OLAP服务实例上的多维模型同步组件96之间可以相互通讯。
监控模块112，设置为监听多维模型变更通知，某个OLAP服务实例上多维模型发生变更发送广播通知，进行本实例多维模型更新。
监控模块监听本实例与其它实例之间的链路，稳定性计算模块114，设置为计算本实例与其它实例的稳定性系数，并选取稳定性最高的OLAP服务实例。
多维模型更新模块116，设置为与多维模型适配组件交互，负责更新本实例的多维模型。
动态会话集群组件98，该动态会话集群组件98与全局数据缓存组件94交互，用户会话数据保存在全局数据缓存中。动态会话集群组件98保证会话不丢失，对用户屏蔽OLAP集群的容灾策略。
多维分析操作前后步骤关联性强，为保证客户端操作的连贯性，保证服务器宕机用户操作不受影响，用户会话中应放入完整的多维模型分析数据，包括模型、当前分析对应的数据源、当前正在分析的MDX语句、当前的数据分析结果、过滤信息、排序信息、是否行列切换等，用户会话要保证可序列化。
图12是根据本发明实施例的多维分析系统中数据缓存管理组件910的结构示意图，如图12所示，数据缓存管理组件910包括：全局数据缓存代理122、数据缓存加载器124，下面对该数据缓存管理组件910进行说明。
全局数据缓存代理122，连接外部全局数据缓存，全局数据缓存代理122主要的功能是获取(GET)和存入(PUT)功能。
数据缓存加载器124，它与全局数据缓存代理122交互，加载数据时首先从缓存中查找数据，如果没有命中则与数据源适配组件交互，获取数据并将数据放入缓存中。
数据缓存管理组件910与数据源适配组件交互，当发送数据源变更，数据缓存加载器命中数据，并进行清理。
在上述实施例中，采用OLAP服务器集群以及全局数据分布式节点缓存的方法，可以使 得用户在高并发、海量数据的存储模式下，高效进行在线多维数据分析。
需要说明的是，上述各个模块是可以通过软件或硬件来实现的，对于后者，可以通过以下方式实现，但不限于此：上述模块均位于同一处理器中；或者，上述模块分别位于多个处理器中。
本发明的实施例还提供了一种存储介质。可选地，在本实施例中，上述存储介质可以被设置为存储用于执行以下步骤的程序代码：
S1，接收用于请求查询预定数据的查询请求；
S2，根据选择策略从OLAP服务器集群中选择用于对查询请求进行处理的OLAP服务器，其中，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；
S3，将所述查询请求转发给所述OLAP服务器。
可选地，存储介质还被设置为存储用于执行以下步骤的程序代码：
S1，接收数据请求方发送的用于请求查询预定数据的查询请求，其中，OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，OLAP服务器集群中包括至少两个OLAP服务器且OLAP服务器集群中的各个OLAP服务器相互独立；
S2，根据查询请求查询所述预定数据；
S3，将查询到的预定数据返回给数据请求方。
可选地，在本实施例中，上述存储介质可以包括但不限于：U盘、只读存储器(Read-Only Memory，简称为ROM)、随机存取存储器(Random Access Memory，简称为RAM)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
可选地，本实施例中的具体示例可以参考上述实施例及可选实施方式中所描述的示例，本实施例在此不再赘述。
显然，本领域的技术人员应该明白，上述的本发明的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。
工业实用性
如上所述，本发明实施例提供的一种数据处理及查询方法、装置具有以下有益效果：解决了相关技术中存在的OLAP在高并发、数据库中存储的数据量大的情况下无法快速响应用户需求的问题，进而达到了在高并发、数据库中存储的数据量大的情况下快速响应用户需求的效果，提高了用户的体验度。

Claims (18)

  1. 一种数据处理方法，包括：
    负载均衡服务器接收用于请求查询预定数据的查询请求；
    所述负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器，其中，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；
    所述负载均衡服务器将所述查询请求转发给所述OLAP服务器。
  2. 根据权利要求1所述的方法，其中，所述负载均衡服务器根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器之后，还包括：
    所述负载均衡服务器确定选择的所述OLAP服务器发生故障；
    所述负载均衡服务器根据所述选择策略从所述OLAP服务器集群中的其他OLAP服务器中选择一个新的用于对所述查询请求进行处理的OLAP服务器；
    所述负载均衡服务器将所述查询请求转发给新的OLAP服务器。
  3. 根据权利要求1至2中任一项所述的方法，其中，所述选择策略包括以下至少之一：
    所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
  4. 一种数据查询方法，包括：
    联机分析处理OLAP服务器接收数据请求方发送的用于请求查询预定数据的查询请求，其中，所述OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的OLAP服务器，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；
    所述OLAP服务器根据所述查询请求查询所述预定数据；
    所述OLAP服务器将查询到的所述预定数据返回给所述数据请求方。
  5. 根据权利要求4所述的方法，其中，所述OLAP服务器根据所述查询请求查询所述预定数据，包括：
    所述OLAP服务器根据所述查询请求在全局数据缓存服务器中查询所述预定数据，其中，所述全局数据缓存服务器中预先缓存有数据，且缓存的所述数据被分布地存储在所述全局数据缓存服务器的两个以上节点中，所述全局数据缓存服务器支持增加和/或删除节点。
  6. 根据权利要求5所述的方法，其中，所述OLAP服务器根据所述查询请求在全局数据缓存服务器中查询所述预定数据之后，包括：
    当所述全局数据缓存服务器中未缓存所述预定数据时，所述OLAP服务器从数据库 中查询所述预定数据；
    所述OLAP服务器将查询到的所述预定数据缓存至所述全局数据缓存服务器中。
  7. 根据权利要求4所述的方法，其中，还包括：
    所述OLAP服务器确定所述OLAP服务器集群中的一个OLAP服务器的用于查询数据的多维模型发生了变化；所述OLAP服务器将所述OLAP服务器的多维模型与发生变化的多维模型进行同步处理；和/或，
    所述OLAP服务器确定所述OLAP服务器的用于查询数据的多维模型发生了变化；所述OLAP服务器向所述OLAP服务器集群中的其他OLAP服务器发送变更通知，其中，所述变更通知用于标识所述OLAP服务器的多维模型发生的变化。
  8. 根据权利要求7所述的方法，其中，还包括：
    所述OLAP服务器将所述OLAP服务器的多维模型同步到全局数据缓存服务器中。
  9. 根据权利要求4至7中任一项所述的方法，其中，所述选择策略包括以下至少之一：
    所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
  10. 一种数据处理装置，应用于负载均衡服务器，包括：
    第一接收模块，设置为接收用于请求查询预定数据的查询请求；
    第一选择模块，设置为根据选择策略从联机分析处理OLAP服务器集群中选择用于对所述查询请求进行处理的OLAP服务器，其中，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；
    第一转发模块，设置为将所述查询请求转发给所述OLAP服务器。
  11. 根据权利要求10所述的装置，其中，还包括：
    第一确定模块，设置为确定选择的所述OLAP服务器发生故障；
    第二选择模块，设置为根据所述选择策略从所述OLAP服务器集群中的其他OLAP服务器中选择一个新的用于对所述查询请求进行处理的OLAP服务器；
    第二转发模块，设置为将所述查询请求转发给新的OLAP服务器。
  12. 根据权利要求10至11中任一项所述的装置，其中，所述选择策略包括以下至少之一：
    所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
  13. 一种数据查询装置，应用于联机分析处理OLAP服务器，包括：
    第二接收模块，设置为接收数据请求方发送的用于请求查询预定数据的查询请求，其中，所述OLAP服务器为负载均衡服务器根据选择策略从OLAP服务器集群中选择的 OLAP服务器，所述OLAP服务器集群中包括至少两个OLAP服务器且所述OLAP服务器集群中的各个OLAP服务器相互独立；
    第一查询模块，设置为根据所述查询请求查询所述预定数据；
    返回模块，设置为将查询到的所述预定数据返回给所述数据请求方。
  14. 根据权利要求13所述的装置，其中，所述第一查询模块包括：
    根据所述查询请求在全局数据缓存服务器中查询所述预定数据，其中，所述全局数据缓存服务器中预先缓存有数据，且缓存的所述数据被分布地存储在所述全局数据缓存服务器的两个以上节点中，所述全局数据缓存服务器支持增加和/或删除节点。
  15. 根据权利要求14所述的装置，其中，还包括：
    第二查询模块，设置为当所述全局数据缓存服务器中未缓存所述预定数据时，从数据库中查询所述预定数据；
    缓存模块，设置为将查询到的所述预定数据缓存至所述全局数据缓存服务器中。
  16. 根据权利要求13所述的装置，其中，还包括：
    第二确定模块，设置为确定所述OLAP服务器集群中的一个OLAP服务器的用于查询数据的多维模型发生了变化；第一同步模块，用于将所述OLAP服务器的多维模型与发生变化的多维模型进行同步处理；和/或，
    第三确定模块，设置为确定所述OLAP服务器的用于查询数据的多维模型发生了变化；发送模块，用于向所述OLAP服务器集群中的其他OLAP服务器发送变更通知，其中，所述变更通知用于标识所述OLAP服务器的多维模型发生的变化。
  17. 根据权利要求16所述的装置，其中，还包括：
    第二同步模块，设置为将所述OLAP服务器的多维模型同步到全局数据缓存服务器中。
  18. 根据权利要求13至16中任一项所述的装置，其中，所述选择策略包括以下至少之一：
    所述OLAP服务器集群中的OLAP服务器的负载情况、OLAP服务器负荷策略。
PCT/CN2016/085558 2015-07-30 2016-06-13 数据处理及查询方法、装置 WO2017016336A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510459153.7A CN106708826A (zh) 2015-07-30 2015-07-30 数据处理及查询方法、装置
CN201510459153.7 2015-07-30

Publications (1)

Publication Number Publication Date
WO2017016336A1 true WO2017016336A1 (zh) 2017-02-02

Family

ID=57884061

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/085558 WO2017016336A1 (zh) 2015-07-30 2016-06-13 数据处理及查询方法、装置

Country Status (2)

Country Link
CN (1) CN106708826A (zh)
WO (1) WO2017016336A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857768A (zh) * 2018-12-29 2019-06-07 电大在线远程教育技术有限公司 一种大数据聚合查询方法
CN112698941A (zh) * 2020-12-22 2021-04-23 浙江中控技术股份有限公司 一种基于动态负载均衡的实时数据库查询方法
CN113489777A (zh) * 2021-07-01 2021-10-08 厦门悦讯信息科技股份有限公司 一种物联网设备集群化数据采集的方法及系统
CN113691611A (zh) * 2021-08-23 2021-11-23 湖南大学 一种区块链的分布式高并发事务处理方法及系统、设备、存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107622124B (zh) * 2017-09-28 2021-02-02 深圳市华傲数据技术有限公司 基于块数据的数据查询方法及系统
CN107943615B (zh) * 2017-11-06 2020-08-18 许继集团有限公司 基于分布式集群的数据处理方法与系统
CN108234616A (zh) * 2017-12-25 2018-06-29 深圳华强聚丰电子科技有限公司 一种高可用分布式web缓存系统和方法
CN108763300B (zh) * 2018-04-19 2020-07-31 北京奇艺世纪科技有限公司 一种数据查询方法及装置
CN109445934B (zh) * 2018-09-26 2024-03-29 中国平安人寿保险股份有限公司 查询请求的分配方法及系统
CN113791904B (zh) * 2021-09-13 2022-11-08 北京百度网讯科技有限公司 用于处理查询输入的方法、装置、设备和可读存储介质
CN114003180A (zh) * 2021-11-11 2022-02-01 中国建设银行股份有限公司 一种基于跨机房Hadoop集群的数据处理方法及装置

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102724105A (zh) * 2011-03-30 2012-10-10 腾讯科技（深圳）有限公司 一种负载均衡方法和装置
CN103188304A (zh) * 2011-12-30 2013-07-03 东月创意科技股份有限公司 多伺服器系统负荷平衡机制
CN103297502A (zh) * 2013-05-08 2013-09-11 青岛海信传媒网络技术有限公司 一种负载均衡系统及方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8170981B1 (en) * 2010-12-08 2012-05-01 Dassault Systemes Enovia Corporation Computer method and system for combining OLTP database and OLAP database environments
CN102693291A (zh) * 2012-05-15 2012-09-26 浪潮电子信息产业股份有限公司 一种实现SQL-Server负载均衡集群的方法
CN103227838B (zh) * 2013-05-10 2015-09-30 中国工商银行股份有限公司 一种多重负载均衡处理装置与方法
CN104424258B (zh) * 2013-08-28 2020-06-16 腾讯科技（深圳）有限公司 多维数据查询的方法、查询服务器、列存储服务器及系统
US10109085B2 (en) * 2014-01-08 2018-10-23 Walmart Apollo, Llc Data perspective analysis system and method
CN104504165A (zh) * 2015-01-29 2015-04-08 云南电网公司带电作业分公司 基于海量数据的综合管理分析系统

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102724105A (zh) * 2011-03-30 2012-10-10 腾讯科技（深圳）有限公司 一种负载均衡方法和装置
CN103188304A (zh) * 2011-12-30 2013-07-03 东月创意科技股份有限公司 多伺服器系统负荷平衡机制
CN103297502A (zh) * 2013-05-08 2013-09-11 青岛海信传媒网络技术有限公司 一种负载均衡系统及方法

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857768A (zh) * 2018-12-29 2019-06-07 电大在线远程教育技术有限公司 一种大数据聚合查询方法
CN109857768B (zh) * 2018-12-29 2023-09-08 电大在线远程教育技术有限公司 一种大数据聚合查询方法
CN112698941A (zh) * 2020-12-22 2021-04-23 浙江中控技术股份有限公司 一种基于动态负载均衡的实时数据库查询方法
CN113489777A (zh) * 2021-07-01 2021-10-08 厦门悦讯信息科技股份有限公司 一种物联网设备集群化数据采集的方法及系统
CN113691611A (zh) * 2021-08-23 2021-11-23 湖南大学 一种区块链的分布式高并发事务处理方法及系统、设备、存储介质
CN113691611B (zh) * 2021-08-23 2022-11-22 湖南大学 一种区块链的分布式高并发事务处理方法及系统、设备、存储介质

Also Published As

Publication number Publication date
CN106708826A (zh) 2017-05-24

Similar Documents

Publication Publication Date Title
WO2017016336A1 (zh) 数据处理及查询方法、装置
CN111480154B (zh) 批量数据摄取的方法、系统和介质
US10992740B2 (en) Dynamically balancing partitions within a distributed streaming storage platform
US11520770B2 (en) System and method for providing high availability data
US10545842B2 (en) Automated local database connection affinity and failover
EP2904763B1 (en) Load-balancing access to replicated databases
US8700773B2 (en) Load balancing using redirect responses
US9715522B2 (en) Information processing apparatus and control method
US8484417B2 (en) Location updates for a distributed data store
US8954391B2 (en) System and method for supporting transient partition consistency in a distributed data grid
US8108623B2 (en) Poll based cache event notifications in a distributed cache
US8352634B2 (en) On-demand propagation of routing information in distributed computing system
US8984328B2 (en) Fault tolerance in a parallel database system
US9830235B1 (en) Cooperative fault tolerance and load balancing
US9659078B2 (en) System and method for supporting failover during synchronization between clusters in a distributed data grid
US20120278817A1 (en) Event distribution pattern for use with a distributed data grid
US10540369B2 (en) Org sync suspend and resume data sync
CN112492022A (zh) 提高数据库可用性的集群、方法、系统及存储介质
Sun et al. Adaptive trade‐off between consistency and performance in data replication
Pippal et al. High availability of databases for cloud
Palma et al. DHTJoin: processing continuous join queries using DHT networks
Arrieta-Salinas et al. Epidemia: Variable consistency for transactional cloud databases
Hong et al. Global-scale event dissemination on mobile social channeling platform
Sukhija et al. Load balancing and fault tolerance mechanisms for scalable and reliable big data analytics

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16829703

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16829703

Country of ref document: EP

Kind code of ref document: A1