CN112783954A - Data access method and device and server - Google Patents

Data access method and device and server Download PDF

Info

Publication number
CN112783954A
CN112783954A CN201911080246.3A CN201911080246A CN112783954A CN 112783954 A CN112783954 A CN 112783954A CN 201911080246 A CN201911080246 A CN 201911080246A CN 112783954 A CN112783954 A CN 112783954A
Authority
CN
China
Prior art keywords
access
interface
data
storage system
group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911080246.3A
Other languages
Chinese (zh)
Other versions
CN112783954B (en
Inventor
邵冬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingsoft Cloud Network Technology Co Ltd
Beijing Kingsoft Cloud Technology Co Ltd
Original Assignee
Beijing Kingsoft Cloud Network Technology Co Ltd
Beijing Kingsoft Cloud Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingsoft Cloud Network Technology Co Ltd, Beijing Kingsoft Cloud Technology Co Ltd filed Critical Beijing Kingsoft Cloud Network Technology Co Ltd
Priority to CN201911080246.3A priority Critical patent/CN112783954B/en
Publication of CN112783954A publication Critical patent/CN112783954A/en
Application granted granted Critical
Publication of CN112783954B publication Critical patent/CN112783954B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/70Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer
    • G06F21/78Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer to assure secure storage of data

Abstract

The invention provides a data access method, a data access device and a server; at least one group of interface groups is operated in the server, each group of interface groups comprises a plurality of access interfaces which are arranged in a chain structure, and each access interface corresponds to a data storage system; when a data access request from a service layer is received, determining a target interface group to be accessed according to the request; and accessing the data storage system corresponding to the access interface in the target interface group according to the arrangement sequence of the plurality of access interfaces in the target interface group until the data access corresponding to the data access request is completed. In the invention, a plurality of access interfaces in the interface group are arranged in a chain structure, if the data storage system corresponding to the access interface accessed currently is abnormal, the data storage system corresponding to the next access interface can be accessed according to the arrangement sequence of the access interfaces in the interface group until the data access is completed, thereby improving the stability of data access service and improving the safety of data storage.

Description

Data access method and device and server
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data access method, an apparatus, and a server.
Background
In the related art, a multi-element storage system is mostly adopted for storing data, the multi-element storage system comprises a plurality of data storage systems, and if one data storage system fails, other normal data storage systems can still continue to provide data services; however, the access logic of multiple data storage systems is usually stored in the code logic, and is difficult to modify in real time, once the access logic has problems or the data in the data storage system has problems, there is a high probability that the whole multiple storage system needs to be shut down, upgraded and modified, which easily causes the whole multiple storage system to have faults, and is not favorable for the safety of data and the stability of service.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a data access method, an apparatus and a server, so as to improve stability of a data access service and improve security of data storage.
In a first aspect, an embodiment of the present invention provides a data access method, where the method is applied to a server running with an access interface, where the server runs with at least one group of interface groups; each interface group comprises a plurality of access interfaces arranged in a chain structure; each access interface corresponds to a data storage system; the method comprises the following steps: receiving a data access request from a service layer; determining a target interface group to be accessed according to the data access request; and accessing the data storage system corresponding to the access interface in the target interface group according to the arrangement sequence of the plurality of access interfaces in the target interface group until the data access corresponding to the data access request is completed.
With reference to the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect, where the step of determining, according to the data access request, a target interface group to be accessed includes: determining an access mode of the data access request; and determining the interface group corresponding to the access mode as a target interface group to be accessed.
With reference to the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, where in the interface group, an access address of a next access interface is stored in an access interface other than a last access interface; according to the arrangement sequence of a plurality of access interfaces in the target interface group, accessing the data storage system corresponding to the plurality of access interfaces in the target interface group until completing the data access corresponding to the data access request, comprising the following steps: determining a first access interface in the target interface group as a current access interface; accessing a data storage system corresponding to the current access interface, and if the data storage system corresponding to the current access interface is abnormal in access, acquiring an access address of a next access interface from the current access interface; and determining the access interface corresponding to the access address as an updated current access interface, and continuing to execute the step of accessing the data storage system corresponding to the current access interface until the data access corresponding to the data access request is completed.
With reference to the second possible implementation manner of the first aspect, an embodiment of the present invention provides the second possible implementation manner of the first aspect, where the step of obtaining an access address of a next access interface from a current access interface includes: acquiring an access strategy of a current access interface; if the access policy indicates to continue accessing the next access interface, acquiring an access address of the next access interface from the current access interface; and if the access policy indicates to stop accessing, returning a signal of abnormal access to the service layer.
With reference to the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, where after the step of completing data access corresponding to the data access request is completed, the method further includes: determining a target interface of data corresponding to the returned data access request in the target interface group; and if the target interface is not the first access interface in the target interface group, writing data corresponding to the data access request into the data storage system corresponding to the access interface arranged before the target interface through the access interface arranged before the target interface in the target interface group.
With reference to any one of the first aspect to the fourth possible implementation manner of the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, where the interface group, the arrangement order of the multiple access interfaces in the interface group, and the data storage system corresponding to each access interface are disposed in a zookeeper.
In a second aspect, an embodiment of the present invention further provides a data access apparatus, where the apparatus is disposed in a server running with an access interface, and the server runs with at least one group of interface groups; each interface group comprises a plurality of access interfaces arranged in a chain structure; each access interface corresponds to a data storage system; the device includes: the request receiving module is used for receiving a data access request from a service layer; the interface group determining module is used for determining a target interface group to be accessed according to the data access request; and the data access module is used for accessing the data storage system corresponding to the access interface in the target interface group according to the arrangement sequence of the plurality of access interfaces in the target interface group until the data access corresponding to the data access request is completed.
With reference to the second aspect, an embodiment of the present invention provides a first possible implementation manner of the second aspect, where the interface group determining module is further configured to: determining an access mode of the data access request; and determining the interface group corresponding to the access mode as a target interface group to be accessed.
With reference to the second aspect, an embodiment of the present invention provides a second possible implementation manner of the second aspect, where in the interface group, access addresses of next access interfaces are stored in access interfaces other than a last access interface; the data access module is further configured to: determining a first access interface in the target interface group as a current access interface; accessing a data storage system corresponding to the current access interface, and if the data storage system corresponding to the current access interface is abnormal in access, acquiring an access address of a next access interface from the current access interface; and determining the access interface corresponding to the access address as an updated current access interface, and continuing to execute the step of accessing the data storage system corresponding to the current access interface until the data access corresponding to the data access request is completed.
In a third aspect, an embodiment of the present invention further provides a server, including a processor and a memory, where the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the data access method.
In a fourth aspect, embodiments of the present invention also provide a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the data access method described above.
The embodiment of the invention has the following beneficial effects:
the data access method, the device and the server run in the server, wherein each group of interface group comprises a plurality of access interfaces arranged in a chain structure, each access interface corresponds to a data storage system, and when a data access request from a service layer is received, a target interface group to be accessed is determined according to the data access request; and accessing the data storage systems corresponding to the multiple access interfaces in the target interface group according to the arrangement sequence of the multiple access interfaces in the target interface group until the data access corresponding to the data access request is completed. In the mode, the plurality of access interfaces in the interface group are arranged in a chain structure, if the data storage system corresponding to the currently accessed access interface is abnormal, the data storage system corresponding to the next access interface can be accessed according to the arrangement sequence of the plurality of access interfaces in the interface group until the data access is completed, so that the stability of data access service is improved, and the safety of data storage is also improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention as set forth above.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic view of an application scenario of data access according to an embodiment of the present invention;
fig. 2 is a flowchart of a data access method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an interface set according to an embodiment of the present invention;
FIG. 4 is a flow chart of another data access method provided by the embodiments of the present invention;
FIG. 5 is a flow chart of another data access method provided by an embodiment of the invention;
fig. 6 is a schematic structural diagram of a data access device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the following embodiments, and it should be understood that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For ease of understanding, an application scenario of data access is first shown, and as shown in fig. 1, the scenario includes a service layer, an access interface layer and a data storage system connected in sequence; the service layer, the access interface layer and the data storage system are usually operated on a server; they may run on different servers, or any two of them on the same server, or all three on the same server. The service layer can be used for interacting with a user side or other systems and receiving service requests sent by the user side or other systems; generating a data access request according to the received service request, sending the data access request to an access interface layer, and accessing the data storage system through the access interface layer to complete the received service request; the service request may be a request for inquiring service data and the like.
The service data corresponding to different service requests can be stored in different data storage systems, and correspondingly, access interfaces corresponding to the data storage systems are arranged on the access interface layer. When a single metadata storage mode is adopted, that is, a certain service data is only stored in one data storage system, if the system fails, the data in the system may not be accessible, and even the risk of losing the data occurs, which affects the security and stability of data storage. In order to improve the security of data storage, the multivariate data system can be used to store the service data, that is, the same service data is stored in a plurality of data storage systems, and when one data storage system fails, the other data storage systems can be accessed to obtain the service data.
In the access interface layer, a plurality of access interfaces corresponding to the data storage system storing the same service data can be used as an interface group; taking the scenario shown in fig. 1 as an example, the access interface layer may include two interface groups, i.e., an interface group 1 and an interface group 2; the interface group 1 comprises an access interface 1 and an access interface 2, and the interface group 2 comprises an access interface 3 and an access interface 4; access interface 1 corresponds to data storage system 1, access interface 2 corresponds to data storage system 2, access interface 3 corresponds to data storage system 3, and access interface 4 corresponds to data storage system 4; in general, the data stored in the data storage system 1 and the data stored in the data storage system 2 are the same and belong to the same data source; the data stored in the data storage system 3 and the data storage system 4 are the same and belong to the same data source.
Compared with a data access scene based on a single metadata storage mode, the data access scene based on the multi-element storage system has higher stability of data access service; however, in the related art, access logic to a data storage system is generally stored in code logic, and is difficult to modify in real time; when access logic is in problem, or data in the data storage system is in problem, there is a high probability that the whole multi-element storage system needs to be shut down, upgraded and modified, which is likely to cause the whole multi-element storage system to malfunction, and is not favorable for data security and service stability. Based on this, the embodiment of the invention provides a data access method, a data access device and a server, which can be applied to data access scenes of various service systems, such as an enterprise management system, various operation platforms, a cloud storage system and the like.
Referring first to FIG. 2, a data access method is shown; the method is applied to a server with an access interface, wherein at least one group of interface groups runs on the server; each interface group comprises a plurality of access interfaces arranged in a chain structure; each access interface corresponds to a data storage system.
The chain structure arrangement can be realized by the following modes: according to the arrangement sequence, except the last access interface, each access interface stores the access address of the next access interface, and the specific form of the access address can be a physical address, a relative address, a path, an address pointer and the like; as shown in fig. 3, taking an example that the interface group includes a first access interface, a second access interface, and a third access interface, assuming that the arrangement order is the first access interface, the second access interface, and the third access interface, the first access interface stores an access address of the second access interface, and the second access interface stores an access address of the third access interface; the access address of the second access interface can be obtained through the first access interface, if the access address of the second access interface is required, the second access interface can be accessed through the access address of the second access interface, and the like, so that the chain arrangement of the plurality of access interfaces is formed.
The method comprises the following steps:
step S200, receiving a data access request from a service layer.
The data access request can be to store data in a certain data source, delete data of a certain data source, replace data of a certain data source or query data of a certain data source, and the like; the data sources correspond to a plurality of data storage systems for storing the same data; the data access request will typically include information about the data source, such as a contract identifier.
Step S202, according to the data access request, determining a target interface group to be accessed.
Specifically, the corresponding interface group can be searched in the interface group of the access interface layer through the relevant information of the data source, such as the data access content, the access type and other information, in the data access request; when the related information of the data source is the appointed identification, the corresponding interface group should also preset the appointed identification; confirming the interface group as a target interface group through the appointed identification; and accessing the data storage system corresponding to the data source through the access interface in the target interface group.
Step S204, according to the arrangement sequence of the plurality of access interfaces in the target interface group, accessing the data storage system corresponding to the access interface in the target interface group until the data access corresponding to the data access request is completed.
The access Interface is typically an Application Programming Interface (API); for example, DAO (Data Access Object, also referred to as DAO Object) is a common Object-oriented Access interface (also referred to as database interface); the DAO mode is a standard J2EE (Java 2Platform Enterprise Edition) design mode, and by using this mode, the underlying data access operations can be separated from the upper layer business logic, so that developers can concentrate more on writing data access code.
The arrangement sequence can be set by a related technician and stored in a set storage position, and meanwhile, the access addresses of all the access interfaces are stored in the arrangement sequence according to the arrangement sequence, wherein the access address is in the last access interface of the access interfaces; after the target interface group is determined, the arrangement order of the target interface group can be acquired from the set storage position, and the data storage system corresponding to the access interface is accessed from the first access interface in the arrangement order.
The above arrangement order can be regarded as a part of the access logic of the multi-element storage system, and in order to reduce the influence of the solidification condition of the access logic on the data access service, the arrangement order can be viewed and modified by the related technical personnel without stopping the working state of the whole system; for example, ZooKeeper may be introduced in the configuration process of the access interface layer, and the above arrangement order is stored in the ZooKeeper; the ZooKeeper is distributed, open source code distributed application program coordination service, and is software for providing consistency service for distributed application; the ZooKeeper can encapsulate complex and error-prone key services, and provide simple and easy-to-use interfaces for relevant technicians, so that the relevant technicians directly modify the arrangement sequence. In the specific implementation process, other software similar to the ZooKeeper function can be written to realize the characteristic of easy modification of the arrangement sequence.
In the process of accessing the data storage system corresponding to the access interface, when the content of the data access request is different, the specific access strategy is also different. For example, when the data access request is to query data of a certain data source, the data storage system corresponding to the first access interface in the target interface group is accessed first, and if the data corresponding to the data access request is found in the data storage system, the data access corresponding to the data access request is completed; if the data corresponding to the data access request cannot be found in the data storage system, the data storage system corresponding to the data access request can be accessed through the second access interface in sequence to find the data corresponding to the data access request, and so on, until the data corresponding to the data access request is found, the access to the data storage system is stopped. If the data corresponding to the data access request is not found until the data storage system corresponding to the last access interface in the target interface group is accessed, the data access is considered to be abnormal, and a signal of the data access abnormity can be returned to the service layer.
When the data access request is to store data into a certain data source, only the data storage system corresponding to the first access interface in the target interface group may be accessed, and after the data is stored in the data storage system, the data access corresponding to the data access request can be considered to be completed; the synchronization process of the data in other data storage systems in the same data source can be completed by other preset processes.
The data access method is applied to a server with access interfaces, at least one group of interface groups is operated on the server, each group of interface groups comprises a plurality of access interfaces arranged in a chain structure, each access interface corresponds to a data storage system, and when a data access request from a service layer is received, a target interface group to be accessed is determined according to the data access request; and accessing the data storage systems corresponding to the multiple access interfaces in the target interface group according to the arrangement sequence of the multiple access interfaces in the target interface group until the data access corresponding to the data access request is completed. In the mode, the plurality of access interfaces in the interface group are arranged in a chain structure, if the data storage system corresponding to the currently accessed access interface is abnormal, the data storage system corresponding to the next access interface can be accessed according to the arrangement sequence of the plurality of access interfaces in the interface group until the data access is completed, so that the stability of data access service is improved, and the safety of data storage is also improved.
The embodiment of the invention also provides another data access method, which is realized on the basis of the method of the embodiment; the method mainly describes a specific process of determining a target interface group to be accessed according to a data access request (realized through the following steps S402-S406), a specific process of accessing a data storage system corresponding to an access interface in the target interface group according to an arrangement sequence (realized through the following steps S410-S420), and a process of performing data synchronization on the data storage systems of the same data source after completing data access (realized through the following steps S422-S426); as shown in fig. 4, the method comprises the steps of:
step S400, receiving a data access request from a service layer.
Step S402, determining the access mode of the data access request.
Because the data access request usually includes data source related information of the data to be accessed and a processing mode of the data to be accessed, such as writing, deleting or querying, and when the corresponding data storage system can be subjected to addition, deletion, modification and querying through the same access interface, such as a subclass access interface DAO based on BaseDAO, the access mode may be related to only the data source related information of the data to be accessed; if the related information is the contract identifier of the data source, the access mode may be the contract identifier.
Step S404, determining the interface group corresponding to the access mode as a target interface group to be accessed; specifically, according to the data source related information in the access mode, the interface group that can access the data storage system corresponding to the data source is searched, and the interface group is determined as the target interface group to be accessed.
Step S406, determining a first access interface in the target interface group as a current access interface.
The arrangement sequence of the access interfaces in the interface group may be stored in a preset storage location in advance, and after the arrangement sequence of the target interface is obtained, the first access interface may be used as the current access interface.
And step S408, accessing the data storage system corresponding to the current access interface.
Generally speaking, an access interface may be composed of a plurality of components, and each component is mutually matched to access a data storage system corresponding to the access interface; for example, a typical DAO implementation has several components: the system comprises a DAO factory class, a DAO interface, a concrete class for realizing the DAO interface, a data transmission object and the like. The DAO interface defines the method of data access, and the specific class implementing the DAO interface contains the logic to access data from a particular data source.
Step S410, if the access of the data storage system corresponding to the current access interface is abnormal, step S412 is executed; otherwise, step S420 is executed.
If the data access corresponding to the data access request cannot be completed in the data storage system corresponding to the current access interface, the data storage system can be considered to be in an access abnormal state; if the data corresponding to the data access request cannot be found, or the data corresponding to the data access request cannot be deleted, etc.
Step S412, if the current access interface is the last access interface in the target interface group, executing step S414; otherwise, step S416 is executed.
When the data storage system corresponding to the current access interface is accessed abnormally, if the current access interface is the last access interface in the target interface group according to the arrangement sequence in the target interface group, each access interface in the target interface group is accessed, and each data storage system is in an access abnormal state, at this time, the access to the data storage system through the interface can be finished. In practice, however, because multiple data storage systems are employed, there is less chance of each data storage system failing than it would be if a single data storage system were employed.
Step S414, returns an access exception signal to the service layer.
When all the access interfaces in the target interface group are accessed and data access corresponding to the data access request cannot be completed, a signal indicating abnormal access can be generated and returned to the service layer as a reply to the data access request sent by the service layer.
Step S416, acquiring the access address of the next access interface from the current access interface; specifically, in the same interface group, according to the arrangement order of the plurality of access interfaces, the access address of the next access interface is stored in the access interfaces except the last access interface; if the current access interface is not the last access interface, the access address of the next access interface can be read from the current access interface.
Step S418, determining the access interface corresponding to the access address as the updated current access interface, and executing step S408; after the access address of the next access interface is obtained, the next access interface of the current access interface may be used as the updated current access interface, and the step S408 is continuously executed to access the data storage system corresponding to the current access interface.
Step S420, completing the data access corresponding to the data access request.
Under the condition that the data storage system corresponding to the current access interface is normally accessed, data writing, data searching and other operations corresponding to the access request are executed in the data storage system through the access interface, data access corresponding to the data access request is completed, and at the moment, data corresponding to the data access request acquired from the data storage system can be returned to the service layer.
Step S422, determining the target interface of the data corresponding to the returned data access request in the target interface group; and when the data access corresponding to the data access request is completed, determining the current access interface as a target interface for returning the data corresponding to the data access request.
Step S424, if the target interface is not the first access interface in the target interface group, execute step S426; if the target interface is the first access interface in the set of target interfaces, ending.
Specifically, the ranking order of the access interfaces of the target interface in the target interface group can be checked to determine whether the target interface is the first access interface in the target interface group. If the target interface is the first access interface in the target interface group, it indicates that the data storage system corresponding to the first access interface is in a normal access state, but the access state of the data storage system corresponding to the other access interface in the target interface group is unknown, and at this time, the data access process may be directly ended.
In step S426, data corresponding to the data access request is written to the data storage system corresponding to the access interface arranged before the target interface through the access interface arranged before the target interface in the target interface group.
If the target interface is not the first access interface in the target interface group, it indicates that the data storage system corresponding to the access interface before the target interface is in an access abnormal state, and the data corresponding to the data access request is not found in the data storage system with access abnormality, at this time, the data corresponding to the data access request needs to be written into the data storage systems through the corresponding access interface, and this action is called write-back; the access interface that can implement write back is the active interface.
After receiving a data access request from a service layer, determining an access mode of the data access request, determining an interface group corresponding to the access mode as a target interface group to be accessed, then accessing a data storage system corresponding to a first access interface according to an arrangement sequence in the target interface group, if the data storage system is in an abnormal access state, acquiring an access address of a next access interface from the first access interface, and continuing to access the data storage system corresponding to the next access interface through the next access interface until the data access corresponding to the data access request is completed; if the access interface adopted when the data access is finished is not the first access interface, the data corresponding to the data access request is written into the data storage system corresponding to the access interface arranged in front of the access interface in the target interface group, so that the data synchronization of the data storage systems of the same data source is realized, the data storage safety is improved, and the stability of the data access service is also improved.
The embodiment of the invention also provides another data access method, which is realized on the basis of the method of the embodiment; the method mainly describes a process of performing subsequent processing based on an access strategy of an access interface when the access of a data storage system corresponding to the access interface is abnormal by combining an access interface such as a common Data Access Object (DAO).
When the access interface is a DAO, the DAO is grouped according to the access mode and the strategy combination to form a chain interface group (also called Chaingroup); in one interface group, the DAOs are grouped together in a linked list. Each DAO corresponds to a data storage system; and the data storage systems corresponding to the DAOs of the same interface group belong to the same data source. When the data storage system is used for storing metadata (data about data, which is information describing data attributes), it may also be referred to as a metadata storage cluster; the cluster can be a KTS (Kingsoft Table Service) cluster, a KVDB (distributed Key-Value data storage Service) cluster, or a Redis (log-type and Key-Value database based on memory and also persistent, and API for providing multiple languages) cluster; the same type of storage cluster, if two different clusters, needs to provide two different DAOs. The access interface layer composed of DOAs may be referred to as a DAOSource layer, and correspondingly, the chained connection manner of DAOs in the DAOSource layer may be referred to as a ChainGroupService layer (data access chained combination service layer).
As shown in fig. 5, the method includes the steps of:
step S500, receiving a data access request from a service layer.
When the access interface is a DAO, the data access request generally includes functions provided by BaseDAO, such as mput (multiple data write), put (data write), mget (multiple data read), get (data read), mdelet (multiple data delete), delete (data delete), recallPut (data write back), and recallMput, which may be directly called by the service layer.
Step S502, according to the data access request, determining a target interface group to be accessed.
Step S504, determining a first access interface in the target interface group as a current access interface; specifically, the DAO object of the head of the chaining table is taken as the current access interface.
And step S506, accessing the data storage system corresponding to the current access interface.
When accessing the corresponding data storage system through the DAO, calling the corresponding internal function according to the function provided by BaseDAO in the data access request, such as calling an _ { mput, … } function when the function in the data access request is mput; these internal functions invoke their own handle … functions, which are more bottom-level than the internal functions, and interact directly with the data storage system, according to different policies. If the function in the data access request is get or mget, a recallPut or recallPut function may be called subsequently to write back the data.
Step S508, if the access of the data storage system corresponding to the current access interface is abnormal, step S510 is executed; otherwise, step S520 is executed.
Step S510, an access policy of the current access interface is obtained.
The access policy of an access interface is typically included in the components (also called objects) of the access interface; if each DAO access interface contains a daresult object, the access policy of the DAO is stored in the daresult object. The access strategy comprises a plurality of types corresponding to different types of data access requests; if the data access request is to write data into a certain data source, the access policy includes a corresponding write mode, such as asynchronous write or synchronous write, the asynchronous write refers to a program mainline to continue executing subsequent logic without waiting for a write result, and the write task is handed to other auxiliary threads to be completed; synchronous writing means that the writing task is completed in the main thread, and subsequent logic can be performed. The access policy can be embodied by the parameter asynW in the database object: when the parameter is true, the access strategy that the writing in the DAO access interface adopts an asynchronous writing mode is represented; when the parameter is false, the access strategy that the writing in the DAO access interface adopts a synchronous writing mode is represented; when the property of this parameter is modified, the corresponding access policy is also changed.
When write back is involved, the above access policy may also determine whether synchronous or asynchronous is employed in write back mode; the write-back refers to when one DAO executes a query task, no result is queried, then a next DAO is queried, and a result is queried, then, in order to ensure data consistency, the query result needs to be written into a data source of a previous DAO, and the action is called write-back. Asynchronous write back means that the task of write back is not completed in the main thread, the main thread continues to perform subsequent logic, and other auxiliary threads complete the write action; synchronous write back is just the opposite of asynchronous write back, where the write task is completed in the main thread and the program can continue to execute after the completion of the write is confirmed. This access policy can be embodied by the parameter asynRcW in the database object: when the parameter is true, the method indicates that an asynchronous write-back access strategy is adopted in a write-back mode of the DAO access interface; when the parameter is false, it indicates that an asynchronous write-back access policy is adopted in the write-back mode of the DAO access interface.
The access policy may further include a processing mode for exception of access to the data storage system; the access strategy of the current access interface can be that when the access of the data storage system corresponding to the current access interface is abnormal, the next access interface is continuously accessed; it is also possible to stop continuing the access. The parameter of thwex in the DAORule object can embody the access policy, and when the parameter is true, the parameter indicates that if the access of the data storage system corresponding to the DAO object is abnormal (namely the DAO object is failed to be accessed), the data access flow is terminated; when the parameter is false, it indicates that if the access of the data storage system corresponding to the DAO object is abnormal, the next DAO object is continuously accessed.
Step S512, if the access policy indicates to continue accessing the next access interface, step S514 is executed; if the access policy indicates to stop the access, step S522 is performed.
And combining the access policy of the DAO, namely when the obtained thwEx parameter is false, the access policy only continues to access the next access interface, and when the obtained thwEx parameter is true, the access policy indicates to stop accessing.
Step S514, if the current access interface is the last access interface in the target interface group, step S522 is executed; otherwise, step S516 is executed.
Step S516, an access address of the next access interface is obtained from the current access interface.
Step S518, determining the access interface corresponding to the access address as the updated current access interface, and executing step S506; after the function execution of the current DAO is completed, the function of the next DAO is directly called.
Step S520, finishing data access corresponding to the data access request; specifically, when the data storage system corresponding to the current DAO is in a normal access state, the DAO provides functions such as handle { mput/put … } to implement a real read-write operation on the corresponding metadata storage.
Step S522, returns a signal of access exception to the service layer.
In a specific implementation process, the interface group, the arrangement order of the multiple access interfaces in the interface group, and the data storage system corresponding to each access interface may all be set in the zookeeper. For example, configuration items of the correspondence relationship of the access interface and the data storage system may be set in/ufa _ ctrl/DAO _ source directory of zookeeper; regarding the chain arrangement in the interface group, the configuration items can be set in the/ufa _ ctrl/DAO _ group directory of zookeeper. Related technicians can modify the configuration through a zookeeper tool, and adverse effects of access logic solidification on stability of data access services are avoided.
The data access method enables the support of various metadata storage systems and flexible access strategies through the chained DAO object organization mode and the support of flexible access strategies, and enables the support of new metadata storage systems to be simpler.
Corresponding to the above embodiment of the data access method, the embodiment of the present invention further provides a data access device, where the device is disposed in a server running with access interfaces, and the server runs with at least one interface group; each interface group comprises a plurality of access interfaces arranged in a chain structure; each access interface corresponds to a data storage system; as shown in fig. 6, the apparatus includes:
a request receiving module 600, configured to receive a data access request from a service layer;
an interface group determining module 602, configured to determine, according to the data access request, a target interface group to be accessed;
the data access module 604 is configured to access the data storage system corresponding to the access interface in the target interface group according to the arrangement order of the multiple access interfaces in the target interface group until data access corresponding to the data access request is completed.
The data access device is arranged on a server running with access interfaces, at least one group of interface groups runs on the server, each group of interface groups comprises a plurality of access interfaces arranged in a chain structure, each access interface corresponds to a data storage system, and when a data access request from a service layer is received, a target interface group to be accessed is determined according to the data access request; and accessing the data storage systems corresponding to the multiple access interfaces in the target interface group according to the arrangement sequence of the multiple access interfaces in the target interface group until the data access corresponding to the data access request is completed. In the mode, the plurality of access interfaces in the interface group are arranged in a chain structure, if the data storage system corresponding to the currently accessed access interface is abnormal, the data storage system corresponding to the next access interface can be accessed according to the arrangement sequence of the plurality of access interfaces in the interface group until the data access is completed, so that the stability of data access service is improved, and the safety of data storage is also improved.
Further, the interface group determining module is further configured to: determining an access mode of the data access request; and determining the interface group corresponding to the access mode as a target interface group to be accessed.
During specific implementation, in the interface group, the access address of the next access interface is stored in the access interfaces except the last access interface; further, the data access module is further configured to: determining a first access interface in the target interface group as a current access interface; accessing a data storage system corresponding to the current access interface, and if the data storage system corresponding to the current access interface is abnormal in access, acquiring an access address of a next access interface from the current access interface; and determining the access interface corresponding to the access address as an updated current access interface, and continuing to execute the step of accessing the data storage system corresponding to the current access interface until the data access corresponding to the data access request is completed.
Further, the data access module is further configured to: acquiring an access strategy of a current access interface; if the access policy indicates to continue accessing the next access interface, acquiring an access address of the next access interface from the current access interface; and if the access policy indicates to stop accessing, returning a signal of abnormal access to the service layer.
Further, the above apparatus further comprises: the target interface determining module is used for determining a target interface which returns data corresponding to the data access request in the target interface group; and the data writing module is used for writing data corresponding to the data access request into the data storage system corresponding to the access interface arranged before the target interface through the access interface arranged before the target interface in the target interface group if the target interface is not the first access interface in the target interface group.
In a specific implementation, the interface group, the arrangement order of the plurality of access interfaces in the interface group, and the data storage system corresponding to each access interface are provided in the zookeeper.
The data access device provided in the embodiment of the present invention has the same implementation principle and technical effect as those of the foregoing data access method embodiment, and for brief description, reference may be made to corresponding contents in the foregoing data access method embodiment for a part not mentioned in the data access device embodiment.
An embodiment of the present invention further provides a server, as shown in fig. 7, where the server includes a processor 130 and a memory 131, the memory 131 stores machine executable instructions capable of being executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the data access method.
Further, the server shown in fig. 7 further includes a bus 132 and a communication interface 133, and the processor 130, the communication interface 133 and the memory 131 are connected through the bus 132.
The memory 131 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 132 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus.
The processor 130 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 130. The processor 130 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the foregoing embodiment in combination with the hardware thereof.
The embodiment of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the data access method.
The data access method and apparatus provided in the embodiments of the present invention and the computer program product of the server include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (11)

1. A data access method is applied to a server running with an access interface, and at least one group of interface groups runs on the server; each of the interface groups comprises a plurality of access interfaces arranged in a chain structure; each access interface corresponds to a data storage system; the method comprises the following steps:
receiving a data access request from a service layer;
determining a target interface group to be accessed according to the data access request;
and accessing the data storage system corresponding to the access interface in the target interface group according to the arrangement sequence of the plurality of access interfaces in the target interface group until the data access corresponding to the data access request is completed.
2. The method of claim 1, wherein the step of determining a set of target interfaces to be accessed based on the data access request comprises:
determining an access mode of the data access request;
and determining the interface group corresponding to the access mode as a target interface group to be accessed.
3. The method according to claim 1, wherein in the interface group, the access address of the next access interface is stored in the access interfaces except the last access interface;
the step of accessing the data storage system corresponding to the multiple access interfaces in the target interface group according to the arrangement sequence of the multiple access interfaces in the target interface group until the data access corresponding to the data access request is completed includes:
determining a first access interface in the target interface group as a current access interface;
accessing the data storage system corresponding to the current access interface, and if the data storage system corresponding to the current access interface is abnormal in access, acquiring an access address of a next access interface from the current access interface;
and determining the access interface corresponding to the access address as an updated current access interface, and continuing to execute the step of accessing the data storage system corresponding to the current access interface until the data access corresponding to the data access request is completed.
4. The method of claim 3, wherein the step of obtaining the access address of the next access interface from the current access interface comprises:
obtaining an access strategy of the current access interface;
if the access policy indicates to continue accessing the next access interface, acquiring an access address of the next access interface from the current access interface;
and if the access policy indicates to stop accessing, returning an access exception signal to the service layer.
5. The method of claim 1, wherein after the step of completing the data access corresponding to the data access request, the method further comprises:
determining a target interface which returns data corresponding to the data access request in the target interface group;
and if the target interface is not the first access interface in the target interface group, writing the data corresponding to the data access request into the data storage system corresponding to the access interface arranged before the target interface through the access interface arranged before the target interface in the target interface group.
6. The method of any of claims 1 to 5, wherein the set of interfaces, the order of the plurality of access interfaces in the set of interfaces, and the data storage system to which each of the access interfaces corresponds are provided in a zookeeper.
7. A data access device is characterized in that the device is arranged on a server running with an access interface, and at least one group of interface groups runs on the server; each of the interface groups comprises a plurality of access interfaces arranged in a chain structure; each access interface corresponds to a data storage system; the device comprises:
the request receiving module is used for receiving a data access request from a service layer;
the interface group determining module is used for determining a target interface group to be accessed according to the data access request;
and the data access module is used for accessing the data storage system corresponding to the access interface in the target interface group according to the arrangement sequence of the plurality of access interfaces in the target interface group until the data access corresponding to the data access request is completed.
8. The apparatus of claim 7, wherein the interface group determination module is further configured to:
determining an access mode of the data access request;
and determining the interface group corresponding to the access mode as a target interface group to be accessed.
9. The apparatus according to claim 7, wherein in the interface group, access addresses of next access interfaces are stored in access interfaces except for the last access interface;
the data access module is further configured to:
determining a first access interface in the target interface group as a current access interface;
accessing the data storage system corresponding to the current access interface, and if the data storage system corresponding to the current access interface is abnormal in access, acquiring an access address of a next access interface from the current access interface;
and determining the access interface corresponding to the access address as an updated current access interface, and continuing to execute the step of accessing the data storage system corresponding to the current access interface until the data access corresponding to the data access request is completed.
10. A server comprising a processor and a memory, the memory storing machine executable instructions executable by the processor, the processor executing the machine executable instructions to implement the method of any one of claims 1 to 6.
11. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to implement the method of any of claims 1 to 6.
CN201911080246.3A 2019-11-06 2019-11-06 Data access method, device and server Active CN112783954B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911080246.3A CN112783954B (en) 2019-11-06 2019-11-06 Data access method, device and server

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911080246.3A CN112783954B (en) 2019-11-06 2019-11-06 Data access method, device and server

Publications (2)

Publication Number Publication Date
CN112783954A true CN112783954A (en) 2021-05-11
CN112783954B CN112783954B (en) 2024-04-05

Family

ID=75747723

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911080246.3A Active CN112783954B (en) 2019-11-06 2019-11-06 Data access method, device and server

Country Status (1)

Country Link
CN (1) CN112783954B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742364A (en) * 2021-09-10 2021-12-03 拉卡拉支付股份有限公司 Data access method, data access device, electronic equipment, storage medium and program product

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050114616A1 (en) * 2002-11-18 2005-05-26 Arm Limited Access control in a data processing apparatus
CN1737943A (en) * 2004-02-26 2006-02-22 松下电器产业株式会社 Memory interface and data processing system
US20060168472A1 (en) * 2005-01-25 2006-07-27 Inventec Corporation Data storage unit failure condition responding method and system
US7240143B1 (en) * 2003-06-06 2007-07-03 Broadbus Technologies, Inc. Data access and address translation for retrieval of data amongst multiple interconnected access nodes
US7739543B1 (en) * 2003-04-23 2010-06-15 Netapp, Inc. System and method for transport-level failover for loosely coupled iSCSI target devices
CN104283976A (en) * 2013-07-10 2015-01-14 中兴通讯股份有限公司 Switching method of network interfaces and access equipment
CN108541312A (en) * 2016-12-30 2018-09-14 谷歌有限责任公司 The multi-modal transmission of packetized data
CN108763274A (en) * 2018-04-09 2018-11-06 北京三快在线科技有限公司 Recognition methods, device, electronic equipment and the storage medium of access request
CN109165182A (en) * 2018-08-07 2019-01-08 神州融安科技(北京)有限公司 The means of communication and device
CN109491587A (en) * 2017-09-11 2019-03-19 华为技术有限公司 The method and device of data access

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050114616A1 (en) * 2002-11-18 2005-05-26 Arm Limited Access control in a data processing apparatus
US7739543B1 (en) * 2003-04-23 2010-06-15 Netapp, Inc. System and method for transport-level failover for loosely coupled iSCSI target devices
US7240143B1 (en) * 2003-06-06 2007-07-03 Broadbus Technologies, Inc. Data access and address translation for retrieval of data amongst multiple interconnected access nodes
CN1737943A (en) * 2004-02-26 2006-02-22 松下电器产业株式会社 Memory interface and data processing system
US20060168472A1 (en) * 2005-01-25 2006-07-27 Inventec Corporation Data storage unit failure condition responding method and system
CN104283976A (en) * 2013-07-10 2015-01-14 中兴通讯股份有限公司 Switching method of network interfaces and access equipment
CN108541312A (en) * 2016-12-30 2018-09-14 谷歌有限责任公司 The multi-modal transmission of packetized data
CN109491587A (en) * 2017-09-11 2019-03-19 华为技术有限公司 The method and device of data access
CN108763274A (en) * 2018-04-09 2018-11-06 北京三快在线科技有限公司 Recognition methods, device, electronic equipment and the storage medium of access request
CN109165182A (en) * 2018-08-07 2019-01-08 神州融安科技(北京)有限公司 The means of communication and device

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
MINH CHAU NGUYEN ETC.: "Data Storage Adapter in Big Data Platform", 《2015 8TH INTERNATIONAL CONFERENCE ON DATABASE THEORY AND APPLICATION (DTA)》, pages 6 - 9 *
刘汉波: "对象存储系统的I/O处理与缓存研究", 《中国优秀硕士学位论文全文数据库(信息科技辑)》, pages 137 - 24 *
陈惟康: "网络环境下分布式存储系统的研究与设计", 《中国优秀硕士学位论文全文数据库(信息科技辑)》, pages 137 - 9 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742364A (en) * 2021-09-10 2021-12-03 拉卡拉支付股份有限公司 Data access method, data access device, electronic equipment, storage medium and program product
CN113742364B (en) * 2021-09-10 2023-12-26 拉卡拉支付股份有限公司 Data access method, device, electronic equipment, storage medium and program product

Also Published As

Publication number Publication date
CN112783954B (en) 2024-04-05

Similar Documents

Publication Publication Date Title
US7548920B2 (en) Systems and methods of accessing and updating recorded data via an inter-object proxy
CN111104392B (en) Database migration method and device, electronic equipment and storage medium
CN110032599B (en) Data structure reading and updating method and device, and electronic equipment
US11327905B2 (en) Intents and locks with intent
CN108599973B (en) Log association method, device and equipment
US11907260B2 (en) Compare processing using replication log-injected compare records in a replication environment
EP2778962B1 (en) Silo-aware databases
US11704114B2 (en) Data structures for managing configuration versions of cloud-based applications
US20200342008A1 (en) System for lightweight objects
US20210382711A1 (en) Data structures for managing configuration versions of cloud-based applications
CN111177703A (en) Method and device for determining data integrity of operating system
CN112685391B (en) Service data migration method and device, computer equipment and storage medium
CN112783954B (en) Data access method, device and server
CN111651235A (en) Virtual machine set task management method and device
US9009731B2 (en) Conversion of lightweight object to a heavyweight object
CN115729724A (en) Fault injection method, fault test system, electronic device and readable storage medium
CN115658357A (en) Method and apparatus for replaying blockchain transactions
CN115237444A (en) Concurrent control method, device and equipment based on version number and storage medium
CN110493326B (en) Zookeeper-based cluster configuration file management system and method
CN111258873B (en) Test method and device
US20240037079A1 (en) Automated validation of database deployments
US11928238B1 (en) Protecting customer data using different domains
US11934879B1 (en) Data processing using application sequence identifiers in cloud environments
CN111414162B (en) Data processing method, device and equipment thereof
CN114255004A (en) Transaction processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant