CN106331166A - Access method and device of storage resource - Google Patents

Access method and device of storage resource Download PDF

Info

Publication number
CN106331166A
CN106331166A CN201610886961.6A CN201610886961A CN106331166A CN 106331166 A CN106331166 A CN 106331166A CN 201610886961 A CN201610886961 A CN 201610886961A CN 106331166 A CN106331166 A CN 106331166A
Authority
CN
China
Prior art keywords
lun
memory node
copy lun
ownership
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610886961.6A
Other languages
Chinese (zh)
Other versions
CN106331166B (en
Inventor
龚飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Macrosan Technologies Co Ltd
Original Assignee
Macrosan Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Macrosan Technologies Co Ltd filed Critical Macrosan Technologies Co Ltd
Priority to CN201610886961.6A priority Critical patent/CN106331166B/en
Publication of CN106331166A publication Critical patent/CN106331166A/en
Application granted granted Critical
Publication of CN106331166B publication Critical patent/CN106331166B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Hardware Redundancy (AREA)

Abstract

The application provides an access method and device of storage resource, applied on the storage node of multivariable control system, wherein the method provides two duplicate LUN for users: a main duplicate Lun and a subordinative duplicate Lun, namely two entity LUN, the two duplicates LUN have the same property, synchronous data, and belong to different storage nodes, when one storage node fails, the duplicate LUN on the other storage node can provide normal storage service for users, which improves reliability of storage system.

Description

A kind of access method storing resource and device
Technical field
The application relates to technical field of network storage, particularly relates to a kind of access method storing resource and device.
Background technology
Society now, information system has become social production ability and the important support of enterprise's normal operation, and data are The foundation stone of information system, business continuance is the guarantee of information system availability, and increasing user selects data to preserve In special storage system.
For high-end storage user, it is generally selected storage system with multiple controllers, the control of storage system with multiple controllers Device number is more, and controller failure tolerance is higher, and front-end access path and addressing means more, high-end storage can be met User's requirement to storage system reliability.But, the entity LUN only one of which copy of current multi-controller storage, data can The highest by property.
Summary of the invention
In view of this, the application provides a kind of access method storing resource and device, provide the user one higher can Many controlled storage systems by property.
Specifically, the application is achieved by the following technical solution:
The application provides a kind of access method storing resource, the storage joint that described method is applied in many controlled storage systems On point, described many controlled storage systems include multiple memory node, and the method includes:
When memory node receives the write IO request that application server sends, according to the LUN carried in described write IO request Identification information obtains the primary copy LUN corresponding with described LUN identification information and major-minor from the LUN relation list item of local record The ownership memory node of this LUN, described LUN relation list item is for recording returning of LUN identification information, copy LUN and copy LUN Belong to the corresponding relation of memory node, wherein, described copy LUN include primary copy LUN and from copy LUN, described primary copy LUN and The described attribute from copy LUN is identical, and belongs to different memory nodes;
When the ownership memory node of described primary copy LUN is this memory node, submit described write IO request to this storage Primary copy LUN process on node;And to from copy LUN synchronous write I/O data;Process completing write IO request and complete to write After I/O data synchronizes, result is returned described application server;
When the ownership memory node of described primary copy LUN is not this memory node, the ownership to described primary copy LUN is deposited Storage node forwards described write IO request, so that the ownership memory node of described primary copy LUN submits described write IO request to self The primary copy LUN process of node, and to from copy LUN synchronous write I/O data;Receive the ownership memory node of described primary copy LUN Process completing write IO request and complete to write the result sent after I/O data synchronizes, described result is returned described Application server.
Further, described method also includes:
When memory node receives the read I O request that application server sends, according to the LUN carried in described read I O request Identification information obtains the ownership storage of the copy LUN corresponding with described LUN identification information from the LUN relation list item of local record Node;
When the ownership memory node of described copy LUN comprises this memory node, described read I O request is submitted to and originally deposits Copy LUN process on storage node, and after read I O request has processed, result is returned described application server;
When the ownership memory node of described copy LUN does not comprises this memory node, store joint from the ownership of copy LUN The ownership memory node selecting a copy LUN in point forwards described read I O request, so that the ownership storage of the copy LUN selected Node submits described read I O request to copy LUN process on own node;Reception processes the copy LUN's of described read I O request The result that ownership memory node sends, returns described result to described application server.
Further, described method also includes:
When the ownership memory node fault of primary copy LUN, and it is this memory node from the ownership memory node of copy LUN Time, obtaining the differentiated identification of arbitration equipment record, described differentiated identification is used for identifying described primary copy LUN and described from copy The data of LUN are the most consistent;
When described differentiated identification is data consistent, primary copy LUN will be defined as from copy LUN on this memory node, And be advertised to other memory node, so that other memory node redefines primary copy LUN.
Further, described method also includes:
This memory node record difference bitmap, described difference bitmap increases the position of data newly for recording in primary copy LUN, And when there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is that data are inconsistent;
When the ownership memory node of the primary copy LUN of fault recovers normal, the major-minor of normal memory node will be recovered This LUN is defined as from copy LUN, and synchronizes this memory node primary copy to this from copy LUN according to the difference bitmap of record The newly-increased data of LUN, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
Further, described method also includes:
When from the ownership memory node fault of copy LUN, and the ownership memory node of primary copy LUN is this memory node Time, recording difference bitmap, described difference bitmap is for recording the position of newly-increased data in primary copy LUN, and newly-increased number is occurring According to time, notify described arbitration equipment update described differentiated identification be that data are inconsistent;
When recovering normal from the ownership memory node of copy LUN of fault, according to the difference bitmap recorded to this from pair This LUN synchronizes the newly-increased data of primary copy LUN, and after completing newly-increased data syn-chronization, notice arbitration equipment updates differentiated identification For data consistent.
The application also provides for a kind of access device storing resource, and described device is applied to the storage in many controlled storage systems On node, described many controlled storage systems include multiple memory node, and this device includes:
Acquiring unit, when memory node receives the write IO request that application server sends, please according to the described IO that writes The LUN identification information carried is asked to obtain the major-minor corresponding with described LUN identification information from the LUN relation list item of local record The ownership memory node of this LUN and primary copy LUN, described LUN relation list item be used for recording LUN identification information, copy LUN with And the corresponding relation of the ownership memory node of copy LUN, wherein, described copy LUN includes primary copy LUN and from copy LUN, institute State primary copy LUN identical with the described attribute from copy LUN, and belong to different memory nodes;
Processing unit, for when the ownership memory node of described primary copy LUN is this memory node, asks the described IO that writes Ask the primary copy LUN process submitted on this memory node;And to from copy LUN synchronous write I/O data;Complete at write IO request After managing and completing to write I/O data synchronization, result is returned described application server;When the ownership of described primary copy LUN is deposited When storage node is not this memory node, forward described write IO request to the ownership memory node of described primary copy LUN, so that described Described write IO request is submitted to the primary copy LUN process of own node by the ownership memory node of primary copy LUN, and to from copy LUN synchronous write I/O data;The ownership memory node receiving described primary copy LUN completes write IO request process and is completing to write IO The result sent after data syn-chronization, returns described result to described application server.
Further,
Described acquiring unit, when being additionally operable to the read I O request that memory node receives application server transmission, according to described The LUN identification information carried in read I O request obtains corresponding with described LUN identification information from the LUN relation list item of local record The ownership memory node of copy LUN;
Described processing unit, is additionally operable to when comprising this memory node in the ownership memory node of described copy LUN, by institute State read I O request and submit the copy LUN process on this memory node to, and after read I O request has processed, result is returned Described application server;When the ownership memory node of described copy LUN does not comprises this memory node, from returning of copy LUN Belong to and memory node selects the ownership memory node of a copy LUN forward described read I O request, so that the copy LUN selected Ownership memory node submits described read I O request to copy LUN process on own node;Reception processes described read I O request The result that the ownership memory node of copy LUN sends, returns described result to described application server.
Further, described device also includes:
Trouble unit, is used for when the ownership memory node fault of primary copy LUN, and from the ownership memory node of copy LUN When being this memory node, obtain the differentiated identification of arbitration equipment record, described differentiated identification be used for identifying described primary copy LUN and The described data from copy LUN are the most consistent;When described differentiated identification is data consistent, by this memory node from copy LUN is defined as primary copy LUN, and is advertised to other memory node, so that other memory node redefines primary copy LUN.
Further,
Described trouble unit, is additionally operable to this memory node record difference bitmap, and described difference bitmap is used for recording primary copy The position of newly-increased data in LUN, and when there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is data Inconsistent;When the ownership memory node of the primary copy LUN of fault recovers normal, the primary copy of normal memory node will be recovered LUN is defined as from copy LUN, and synchronizes this memory node primary copy LUN to this from copy LUN according to the difference bitmap of record Newly-increased data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
Further, described device also includes:
Trouble unit, for working as the ownership memory node fault from copy LUN, and the ownership memory node of primary copy LUN When being this memory node, recording difference bitmap, described difference bitmap increases the position of data newly for recording in primary copy LUN, and When there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is that data are inconsistent;When fault from copy When the ownership memory node of LUN recovers normal, synchronize primary copy LUN new to this from copy LUN according to the difference bitmap of record Increase data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
By above description it can be seen that the application has provided the user two copy LUN (primary copy LUN and from copy LUN, i.e. two entity LUN), the attribute of the two copy LUN is identical, data syn-chronization and belong to different memory nodes, because of This, can play the effect of data backup, and when a memory node fault, the copy LUN on another memory node can be normal Provide the user storage service, it is ensured that customer service is not interrupted, improve the reliability of storage system.
Accompanying drawing explanation
Fig. 1 is the many controlled storage systems schematic diagram shown in the application one exemplary embodiment;
Fig. 2 is a kind of access method flow chart storing resource shown in the application one exemplary embodiment;
Fig. 3 is the many controlled storage systems schematic diagram after LUN is acted on behalf of in the establishment shown in the application one exemplary embodiment;
Fig. 4 is that the basis of a kind of device place equipment that accesses storing resource shown in the application one exemplary embodiment is hard Part structural representation;
Fig. 5 is a kind of structural representation accessing device storing resource shown in the application one exemplary embodiment.
Detailed description of the invention
Here will illustrate exemplary embodiment in detail, its example represents in the accompanying drawings.Explained below relates to During accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represents same or analogous key element.Following exemplary embodiment Described in embodiment do not represent all embodiments consistent with the application.On the contrary, they are only with the most appended The example of the apparatus and method that some aspects that described in detail in claims, the application are consistent.
It is only merely for describing the purpose of specific embodiment at term used in this application, and is not intended to be limiting the application. " a kind of ", " described " and " being somebody's turn to do " of singulative used in the application and appended claims is also intended to include majority Form, unless context clearly shows that other implications.It is also understood that term "and/or" used herein refers to and wraps Any or all containing one or more projects of listing being associated may combination.
Although should be appreciated that in the application possible employing term first, second, third, etc. to describe various information, but this A little information should not necessarily be limited by these terms.These terms are only used for same type of information is distinguished from each other out.Such as, without departing from In the case of the application scope, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as One information.Depend on linguistic context, word as used in this " if " can be construed to " ... time " or " when ... Time " or " in response to determining ".
The embodiment of the present application proposes a kind of access method storing resource, is applied to the memory node in many controlled storage systems In (i.e. storage device), the method has provided the user two copy LUN (primary copy LUN and from copy LUN, i.e. two entities LUN), the attribute of the two copy LUN is identical, data syn-chronization and belong to different memory nodes, therefore, when a storage During node failure, the copy LUN on another memory node can normally provide the user storage service, improve storage system can By property.
See Fig. 1, for the application many controlled storage systems schematic diagram.These many controlled storage systems include 3 memory nodes, respectively For NodeA, NodeB and NodeC, each memory node comprises 2 controllers (SP1 and SP2).Wherein, NodeA is upper creates Create LUN1, LUN0 and LUN1 on LUN0, NodeB and be entity LUN, and attribute identical (such as, storage size is identical). The embodiment of the present application using LUN0 and LUN1 as two copy LUN of same user.Each memory node is interconnected by private network, Such as, Ethernet, FC (Fibre Channel, optical-fibre channel), PCI-E (Peripheral Component Interconnection-Express, quick peripheral component interconnection) etc., it is achieved the data interaction between memory node.Application Run multi-path software in server S erverA, can access user's by a plurality of access path (A1, A2, B1, B2, C1, C2) Entity LUN (LUN0 or LUN1).
Seeing Fig. 2, store an embodiment flow chart of the access method of resource for the application, this embodiment is to storage money The access process in source is described.
Step 201, when memory node receives the write IO request that application server sends, takes according in described write IO request The LUN identification information of band obtains the primary copy LUN corresponding with described LUN identification information from the LUN relation list item of local record And the ownership memory node of primary copy LUN.
Before the many controlled storage systems of application server access, first complete the resource distribution work to many controlled storage systems Make.Specifically, manager logs in the administration interface of many controlled storage systems by management equipment, create two entity LUN for user, Hereinafter referred to as copy LUN, one of them is primary copy LUN, and another is from copy LUN, primary copy LUN with from copy LUN's Attribute is identical, but belongs to different memory nodes.Management equipment memory node in many controlled storage systems issues configuration life Order, carries the copy LUN information into user's distribution and routing information in configuration order.Memory node receives configuration order After, what on this memory node, establishment was corresponding with copy LUN acts on behalf of LUN, and it is identical with copy LUN attribute that this acts on behalf of LUN attribute, and Generate corresponding LUN relation list item, belong to the corresponding of memory node for what LUN and copy LUN and copy LUN acted on behalf of in record Relation.See table 1, for LUN relation list item example.
Table 1
Table 1 is the LUN relation list item on memory node NodeA, and this list item shows that creating one on NodeA acts on behalf of LUN (VA-LUN0), this VA-LUN0 is generation of copy LUN1 of copy LUN0 and NodeB of the sensing NodeA being arranged on NodeA Reason LUN.Then, distribute to application server by acting on behalf of LUN on memory node, i.e. configuration application server can be with access agent The path of LUN.
Features based on many controlled storage systems, manager can create LUN pair, the copy with user on different memory nodes That answers acts on behalf of LUN, to increase the path accessing copy LUN, improves the reliability of storage system.Certainly, this is not the application Emphasis, does not repeats them here.
After the resource distribution completing above-mentioned many controlled storage systems works, log in application server, set up application server The connection of each memory node in many controlled storage systems, in order to application server passes through multi-path software by difference memory node What attribute was identical act on behalf of LUN is identified as a copy LUN.
As application server access copy LUN, multi-path software select path according to network condition, path status etc. Issue I/O Request, and in I/O Request, carry the LUN identification information acting on behalf of LUN.The memory node being positioned on this path is according to answering Subsequent treatment is performed with the I/O Request of server.
In this step, memory node receives the write IO request that application server sends, and according to write IO request carries LUN identification information obtains the primary copy LUN corresponding with LUN identification information and major-minor from the LUN relation list item of local record The ownership memory node of this LUN.For example, it is assumed that LUN0 is primary copy LUN in table 1, LUN1 is from copy LUN, then by this step Suddenly would know that primary copy LUN is LUN0, this primary copy LUN is positioned on memory node NodeA.
Step 202, when the ownership memory node of described primary copy LUN is this memory node, carries described write IO request Hand over the primary copy LUN process on this memory node;And to from copy LUN synchronous write I/O data;Complete write IO request process with And after completing to write I/O data synchronization, result is returned described application server.
Step 203, when the ownership memory node of described primary copy LUN is not this memory node, to described primary copy LUN Ownership memory node forward described write IO request so that the ownership memory node of described primary copy LUN is by described write IO request Submit the primary copy LUN process of own node to, and to from copy LUN synchronous write I/O data;Receive the ownership of described primary copy LUN Memory node completes the result of transmission after write IO request processes and completes to write I/O data synchronization, by described result Return described application server.
From step 202 and step 203 it can be seen that the write IO request of application server transmission is all the time by primary copy LUN Reason, and to from copy LUN synchronous write I/O data, to ensure primary copy LUN and the data consistent from copy LUN.Meanwhile, primary copy Result is returned to application server along I/O Request path by the ownership memory node of LUN, completes the place of whole write IO request Reason process.
When memory node receives the read I O request that application server sends, according to the LUN mark carried in read I O request Knowledge information obtains the ownership memory node of the copy LUN corresponding with LUN identification information from the LUN relation list item of local record. Such as, when the LUN identification information carried in read I O request is VA-LUN0, inquiry table 1 understands, the copy corresponding with VA-LUN0 The ownership memory node of LUN is NodeA and NodeB.
When the ownership memory node of the copy LUN obtained comprises this memory node, submit read I O request to this storage Copy LUN process on node, and after read I O request has processed, result is returned application server.
When the ownership memory node of copy LUN does not comprises this memory node, from the ownership memory node of copy LUN The ownership memory node selecting a copy LUN forwards read I O request, so that the ownership memory node of the copy LUN selected will be read I/O Request submits the copy LUN process on own node to;The ownership memory node of the copy LUN that reception processes this read I O request is sent out The result sent, returns result to application server.
From the processing procedure of above-mentioned read I O request it can be seen that work as the pair having coupling the memory node receiving read I O request During this LUN, either primary copy LUN is still from copy LUN, all by the copy LUN process of this memory node, saves and deposits to other The bandwidth consumption that storage node forwards;When there is no the copy LUN of coupling on the memory node receiving read I O request, can be at primary copy LUN and arbitrarily select a copy LUN to process read I O request from copy LUN.Result returns along the path of read I O request.
So far, the I/O Request processing procedure of the many controlled storage systems of this LUN of two-pack is completed.
When the ownership memory node fault of copy LUN in many controlled storage systems, its processing procedure is as follows:
When the ownership memory node fault of primary copy LUN, and it is this memory node from the ownership memory node of copy LUN Time, obtain the differentiated identification that arbitration equipment (in many controlled storage systems, each memory node is all connected) records with arbitration equipment, This differentiated identification is the most consistent with the data from copy LUN for identifying primary copy LUN.
When difference be designated data inconsistent time, from copy LUN cannot replace primary copy LUN provide storage service, storage Service disconnection.
When difference is designated data consistent, primary copy LUN will be defined as from copy LUN on this memory node, and lead to Accuse to other memory node, so that other memory node redefines primary copy LUN.Meanwhile, this memory node (redefines The ownership memory node of primary copy LUN) start to record difference bitmap, this difference bitmap is used for recording in primary copy LUN newly-increased number According to position.And when there are newly-increased data, it is that data are inconsistent that notice arbitration equipment updates differentiated identification.When break down When the ownership memory node of primary copy LUN recovers normal, the primary copy LUN recovering normal memory node is defined as from copy LUN, and synchronize this memory node the newly-increased data of primary copy LUN to this from copy LUN according to the difference bitmap of record, to protect Data syn-chronization after card fault recovery, and after completing newly-increased data syn-chronization, it is data that notice arbitration equipment updates differentiated identification Unanimously.
When from the ownership memory node fault of copy LUN, and the ownership memory node of primary copy LUN is this memory node Time, record difference bitmap, and when there are newly-increased data, it is that data are inconsistent that notice arbitration equipment updates differentiated identification.When event Barrier when the ownership memory node of copy LUN recovers normal, synchronize major-minor to this from copy LUN according to the difference bitmap of record The newly-increased data of this LUN, to ensure data consistent, and after completing newly-increased data syn-chronization, notice arbitration equipment updates difference mark Know for data consistent.
It can be seen from the above description that the ownership memory node fault of a certain copy LUN, can returning by another copy LUN Belong to memory node and continue with business, improve the reliability of data storage.
So far, fault treating procedure is completed.
Additionally, when application server is no longer necessary to by certain storage node accesses copy LUN, can save in this storage On point, the LUN that acts on behalf of corresponding for copy LUN is cancelled distribution from application server, delete corresponding LUN relation list item, then delete Act on behalf of LUN, thus reduce the consumption of system resource.
It can be seen from the above description that the application has provided the user this LUN of two-pack, this LUN of this two-pack belongs to difference Memory node, therefore, the effect of data backup can be played, when a memory node fault, on another memory node Copy LUN can normally provide the user storage service, it is ensured that customer service is not interrupted, and improves the reliability of storage system.
The most still as a example by Fig. 1, storage resource access process is discussed in detail.
It is assumed that user's request is positioned on NodeA for creating two accessory body LUN LUN0 and LUN1, LUN0, capacity is 100GB;LUN1 is positioned on NodeB, and capacity is 100GB, and wherein, LUN0 is primary copy LUN, and LUN1 is from copy LUN, ServerA accesses copy LUN (LUN0/LUN1) by NodeA, NodeB, NodeC.
First, configuration resource.RAID (Redundant Array of Independent Disks, independence at NodeA Redundant array of inexpensive disk) on create LUN0, capacity is 100GB;Creating LUN1 on the RAID of NodeB, capacity is 100GB.
Then, resource is distributed to application server.Manager is issued to NodeA, NodeB, NodeC by management equipment Configuration order.As a example by NodeA, NodeA receives the configuration order that management equipment issues, and carries copy LUN in this configuration order Information (LUN1 of LUN0, NodeB of NodeA) and routing information;NodeA receives configuration order, creates with NodeA's What the LUN1 of LUN0, NodeB was corresponding acts on behalf of LUN VA-LUN0, inherits the attribute of LUN0 and LUN1, and NodeA adds LUN in this locality Relation list item, sees table 1.Then, NodeA is acquisition approach information from configuration order, and VA-LUN0 is distributed to application service Device, path incidence relation sees table 2.
Table 2
From table 2 it can be seen that on NodeA for VA-LUN0 be associated with the SP1 of two paths: ServerA → NodeA → SP2 → the VA-LUN0 of VA-LUN0, ServerA → NodeA, i.e. can access VA-by path A1 and A2 shown in Fig. 1 LUN0, and then access the LUN1 of LUN0 or NodeB of NodeA.
In like manner, complete to act on behalf of on all memory nodes establishment and the distribution of LUN according to user's request, see Fig. 3.Configure Cheng Hou, LUN relation list item and path incidence relation on each memory node are respectively as follows:
Table 3
Table 4
Table 3 is the LUN relation list item on NodeB;Table 4 is the path incidence relation on NodeB.
Table 5
Table 6
Table 5 is the LUN relation list item on NodeC;Table 6 is the path incidence relation on NodeC.
Manager logs in ServerA, sets up ServerA connection of controller to each memory node, then ServerA can lead to Crossing multi-path software selects either path to access the copy LUN distributing to user.
After completing above-mentioned configuration, ServerA can access copy LUN, below lifts the explanation storage of several specific embodiment The access process of resource.
Embodiment 1 (as a example by NodeA):
It is assumed that the multi-path software on ServerA selects path A1 to issue write IO request, write IO request is carried LUN Identification information VA-LUN0, after memory node NodeA receives this write IO request, inquires about table 1, obtains major-minor corresponding to VA-LUN0 This LUN be the ownership memory node of LUN0, LUN0 be NodeA, i.e. primary copy LUN is positioned on self memory node, therefore, NodeA submits to the LUN0 on own node to process write IO request.After write IO request has processed, NodeA passes through private network will Write I/O data and be sent to NodeB, on NodeB from copy LUN LUN1 synchronous write I/O data.Meanwhile, by result along A1 Path returns to ServerA.
It is assumed that the multi-path software on ServerA selects path A1 to issue read I O request, read I O request is carried LUN Identification information VA-LUN0, after memory node NodeA receives this write IO request, inquires about table 1, obtains copy corresponding to VA-LUN0 The ownership memory node of LUN is NodeA and NodeB, i.e. comprises in the ownership memory node obtained and currently receives read I O request Memory node NodeA, therefore, read I O request is submitted to the LUN0 process on this memory node by NodeA, after having processed, will Result returns ServerA by path A1.
When the ownership memory node NodeA fault of primary copy LUN, primary copy LUN LUN0 cannot perform to store industry again Business.
When from the ownership memory node NodeB fault of copy LUN, the ownership memory node NodeA of primary copy LUN starts Record difference bitmap, and when there are newly-increased data, it is that data differ that notice arbitration equipment (Fig. 1 is not shown) updates differentiated identification Cause.When NodeB recovers normal, NodeA synchronizes LUN0 newly-increased from copy LUN LUN1 according to difference bitmap on NodeB Data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
So far, the description of embodiment 1 is completed.
Embodiment 2 (as a example by NodeB):
It is assumed that the multi-path software on ServerA selects path B1 to issue write IO request, write IO request is carried LUN Identification information VB-LUN0, after memory node NodeB receives this write IO request, inquires about table 3, obtains major-minor corresponding to VB-LUN0 This LUN be the ownership memory node of LUN0, LUN0 be NodeA, i.e. the ownership memory node of primary copy LUN is not that current reception is write The memory node NodeB of I/O Request, then NodeB forwards write IO request to NodeA, NodeA submit to LUN0 to process write IO request, And after write IO request has processed, on NodeB from copy LUN LUN1 synchronous write I/O data.Meanwhile, NodeB receives The result that NodeA returns, and return to ServerA by path B1.
It is assumed that the multi-path software on ServerA selects path B1 to issue read I O request, read I O request is carried LUN Identification information VB-LUN0, after memory node NodeB receives this write IO request, inquires about table 3, obtains copy corresponding to VB-LUN0 The ownership memory node of LUN is NodeA and NodeB, i.e. comprises in the ownership memory node obtained and currently receives read I O request Memory node NodeB, therefore, read I O request is submitted to the LUN1 process on this memory node by NodeB, after having processed, will Result returns ServerA by path B1.
When from the ownership memory node NodeB fault of copy LUN, I/O Request cannot be performed again from copy LUN LUN1 Process.
When the ownership memory node NodeA fault of primary copy LUN, from the ownership memory node NodeB of copy LUN from secondary Sanction equipment obtains the differentiated identification of arbitration equipment record.When difference be designated data inconsistent time, on NodeB from copy LUN Cannot perform storage service, storage service interrupts;When difference is designated data consistent, LUN1 is defined as primary copy by NodeB LUN, and it is advertised to other memory node (NodeC), so that NodeC determines the new LUN1 that primary copy LUN is NodeB.Meanwhile, NodeB is as the ownership memory node record difference bitmap of new primary copy LUN, and when there are newly-increased data, notice arbitration sets The standby differentiated identification that updates is that data are inconsistent.When the ownership memory node NodeA of former primary copy LUN recovers normal, by NodeA LUN0 be defined as from copy LUN, and synchronize the newly-increased data of LUN1 to LUN0 according to difference bitmap, and complete newly-increased data After synchronization, it is data consistent that notice arbitration equipment updates differentiated identification.
So far, the description of embodiment 2 is completed.
Embodiment 3 (as a example by NodeC):
It is assumed that the multi-path software on ServerA selects path C1 to issue write IO request, write IO request is carried LUN Identification information VC-LUN0, after memory node NodeC receives this write IO request, inquires about table 5, obtains major-minor corresponding to VC-LUN0 This LUN be the ownership memory node of LUN0, LUN0 be NodeA, i.e. the ownership memory node of primary copy LUN is not that current reception is write The memory node NodeC of I/O Request, then NodeC forwards write IO request to NodeA, NodeA submit to LUN0 to process write IO request, And after write IO request has processed, on NodeB from copy LUN LUN1 synchronous write I/O data.Meanwhile, NodeC receives The result that NodeA returns, and return to ServerA by path C1.
It is assumed that the multi-path software on ServerA selects path C1 to issue read I O request, read I O request is carried LUN Identification information VC-LUN0, after memory node NodeC receives this write IO request, inquires about table 5, obtains copy corresponding to VC-LUN0 The ownership memory node of LUN is NodeA and NodeB, does not comprise and currently receive read I O request in the ownership memory node of copy LUN Memory node NodeC, therefore, NodeC selects copy LUN to process read I O request from LUN0 and LUN1, it is assumed that select LUN1, then read I O request is transmitted to the ownership memory node NodeB of LUN1 by NodeC, and NodeB submits read I O request to this storage LUN1 process on node, and after processing is completed, result is returned to NodeC, NodeC and is returned by path C1 again ServerA。
When the ownership memory node NodeA fault of primary copy LUN, wait the home node NodeB from copy LUN to be received Notice, the LUN1 redefining NodeB is primary copy LUN.
So far, the description of embodiment 3 is completed.
Corresponding with the embodiment of aforementioned storage resource access method, present invention also provides storage resource access device Embodiment.
The application stores the embodiment of resource access device and can apply on memory node.Device embodiment can be passed through Software realizes, it is also possible to realize by the way of hardware or software and hardware combining.As a example by implemented in software, as a logic meaning Device in justice, is to be formed by computer program instructions corresponding in the processor run memory of its place equipment.From For hardware view, as shown in Figure 4, store a kind of hardware structure diagram of resource access device place equipment for the application, except Outside processor shown in Fig. 4, network interface and memorizer, in embodiment, the equipment at device place is generally according to this equipment Actual functional capability, it is also possible to include other hardware, this repeated no more.
Refer to Fig. 5, for the structural representation of the storage resource access device in one embodiment of the application.This storage provides Source accesses device and includes: acquiring unit 501 and processing unit 502, wherein:
Acquiring unit 501, when memory node receives the write IO request that application server sends, writes IO according to described The LUN identification information carried in request obtains the master corresponding with described LUN identification information from the LUN relation list item of local record The ownership memory node of copy LUN and primary copy LUN, described LUN relation list item is used for recording LUN identification information, copy LUN And the corresponding relation of the ownership memory node of copy LUN, wherein, described copy LUN includes primary copy LUN and from copy LUN, Described primary copy LUN is identical with the described attribute from copy LUN, and belongs to different memory nodes;
Processing unit 502, for when the ownership memory node of described primary copy LUN is this memory node, writes described I/O Request submits the primary copy LUN process on this memory node to;And to from copy LUN synchronous write I/O data;Please completing to write IO After asking process and completing to write I/O data synchronization, result is returned described application server;When returning of described primary copy LUN When genus memory node is not this memory node, forward described write IO request to the ownership memory node of described primary copy LUN, so that Described write IO request is submitted to the primary copy LUN process of own node by the ownership memory node of described primary copy LUN, and to from pair This LUN synchronous write I/O data;The ownership memory node receiving described primary copy LUN completes write IO request process and is completing to write The result that I/O data sends after synchronizing, returns described result to described application server.
Further,
Described acquiring unit 501, when being additionally operable to the read I O request that memory node receives application server transmission, according to institute State the LUN identification information carried in read I O request to obtain and described LUN identification information pair from the LUN relation list item of local record The ownership memory node of the copy LUN answered;
Described processing unit 502, is additionally operable to when comprising this memory node in the ownership memory node of described copy LUN, Described read I O request is submitted to the copy LUN process on this memory node, and after read I O request has processed, by result Return described application server;When the ownership memory node of described copy LUN does not comprises this memory node, from copy LUN Ownership memory node in select a copy LUN ownership memory node forward described read I O request so that select copy The ownership memory node of LUN submits described read I O request to copy LUN process on own node;Reception processes described reading IO please The result that the ownership memory node of the copy LUN asked sends, returns described result to described application server.
Further, described device also includes:
Trouble unit, is used for when the ownership memory node fault of primary copy LUN, and from the ownership memory node of copy LUN When being this memory node, obtain the differentiated identification of arbitration equipment record, described differentiated identification be used for identifying described primary copy LUN and The described data from copy LUN are the most consistent;When described differentiated identification is data consistent, by this memory node from copy LUN is defined as primary copy LUN, and is advertised to other memory node, so that other memory node redefines primary copy LUN.
Further,
Described trouble unit, is additionally operable to this memory node record difference bitmap, and described difference bitmap is used for recording primary copy The position of newly-increased data in LUN, and when there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is data Inconsistent;When the ownership memory node of the primary copy LUN of fault recovers normal, the primary copy of normal memory node will be recovered LUN is defined as from copy LUN, and synchronizes this memory node primary copy LUN to this from copy LUN according to the difference bitmap of record Newly-increased data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
Further, described device also includes:
Trouble unit, for working as the ownership memory node fault from copy LUN, and the ownership memory node of primary copy LUN When being this memory node, recording difference bitmap, described difference bitmap increases the position of data newly for recording in primary copy LUN, and When there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is that data are inconsistent;When fault from copy When the ownership memory node of LUN recovers normal, synchronize primary copy LUN new to this from copy LUN according to the difference bitmap of record Increase data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
In said apparatus, the function of unit and the process that realizes of effect specifically refer to corresponding step in said method Realize process, do not repeat them here.
For device embodiment, owing to it corresponds essentially to embodiment of the method, so relevant part sees method in fact The part executing example illustrates.Device embodiment described above is only schematically, wherein said as separating component The unit illustrated can be or may not be physically separate, and the parts shown as unit can be or can also It not physical location, i.e. may be located at a place, or can also be distributed on multiple NE.Can be according to reality Need to select some or all of module therein to realize the purpose of the application scheme.Those of ordinary skill in the art are not paying In the case of going out creative work, i.e. it is appreciated that and implements.
The foregoing is only the preferred embodiment of the application, not in order to limit the application, all essences in the application Within god and principle, any modification, equivalent substitution and improvement etc. done, should be included within the scope of the application protection.

Claims (10)

1. the access method storing resource, it is characterised in that the storage joint that described method is applied in many controlled storage systems On point, described many controlled storage systems include multiple memory node, and the method includes:
When memory node receives the write IO request that application server sends, according to the LUN identification carried in described write IO request Information obtains the primary copy LUN corresponding with described LUN identification information and primary copy from the LUN relation list item of local record The ownership memory node of LUN, described LUN relation list item is for recording LUN identification information, copy LUN and the ownership of copy LUN The corresponding relation of memory node, wherein, described copy LUN includes primary copy LUN and from copy LUN, described primary copy LUN and institute State the attribute from copy LUN identical, and belong to different memory nodes;
When the ownership memory node of described primary copy LUN is this memory node, submit described write IO request to this memory node On primary copy LUN process;And to from copy LUN synchronous write I/O data;Process completing write IO request and complete to write IO number After synchronizing, result is returned described application server;
When the ownership memory node of described primary copy LUN is not this memory node, store joint to the ownership of described primary copy LUN Point forwards described write IO request, so that the ownership memory node of described primary copy LUN submits described write IO request to own node Primary copy LUN process, and to from copy LUN synchronous write I/O data;Receive the ownership memory node of described primary copy LUN complete Become write IO request to process and complete to write the result sent after I/O data synchronizes, described result is returned described application Server.
2. the method for claim 1, it is characterised in that described method also includes:
When memory node receives the read I O request that application server sends, according to the LUN identification carried in described read I O request Information obtains the ownership storage joint of the copy LUN corresponding with described LUN identification information from the LUN relation list item of local record Point;
When the ownership memory node of described copy LUN comprises this memory node, this storage is submitted to save described read I O request Copy LUN process on point, and after read I O request has processed, result is returned described application server;
When the ownership memory node of described copy LUN does not comprises this memory node, from the ownership memory node of copy LUN The ownership memory node selecting a copy LUN forwards described read I O request, so that the ownership memory node of the copy LUN selected Submit described read I O request to copy LUN process on own node;Reception processes the ownership of the copy LUN of described read I O request The result that memory node sends, returns described result to described application server.
3. method as claimed in claim 1 or 2, it is characterised in that described method also includes:
When the ownership memory node fault of primary copy LUN, and from the ownership memory node of copy LUN be this memory node time, obtain Taking the differentiated identification of arbitration equipment record, described differentiated identification is for identifying described primary copy LUN and the described number from copy LUN According to the most consistent;
When described differentiated identification is data consistent, primary copy LUN will be defined as from copy LUN on this memory node, and lead to Accuse to other memory node, so that other memory node redefines primary copy LUN.
4. method as claimed in claim 3, it is characterised in that described method also includes:
This memory node record difference bitmap, described difference bitmap for recording in primary copy LUN the position of newly-increased data, and When there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is that data are inconsistent;
When the ownership memory node of the primary copy LUN of fault recovers normal, the primary copy LUN of normal memory node will be recovered It is defined as from copy LUN, and synchronizes this memory node primary copy LUN to this from copy LUN according to the difference bitmap of record Newly-increased data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
5. method as claimed in claim 1 or 2, it is characterised in that described method also includes:
When from the ownership memory node fault of copy LUN, and when the ownership memory node of primary copy LUN is this memory node, note Record difference bitmap, described difference bitmap increases the position of data newly for recording in primary copy LUN, and when newly-increased data occur, Notifying that described arbitration equipment updates described differentiated identification is that data are inconsistent;
When recovering normal from the ownership memory node of copy LUN of fault, according to the difference bitmap recorded to this from copy LUN Synchronize the newly-increased data of primary copy LUN, and after completing newly-increased data syn-chronization, it is data that notice arbitration equipment updates differentiated identification Unanimously.
6. the access device storing resource, it is characterised in that the storage joint that described device is applied in many controlled storage systems On point, described many controlled storage systems include multiple memory node, and this device includes:
Acquiring unit, when memory node receives the write IO request that application server sends, according in described write IO request The LUN identification information carried obtains the primary copy corresponding with described LUN identification information from the LUN relation list item of local record The ownership memory node of LUN and primary copy LUN, described LUN relation list item be used for recording LUN identification information, copy LUN and The corresponding relation of the ownership memory node of copy LUN, wherein, described copy LUN includes primary copy LUN and from copy LUN, described Primary copy LUN is identical with the described attribute from copy LUN, and belongs to different memory nodes;
Processing unit, for when the ownership memory node of described primary copy LUN is this memory node, carries described write IO request Hand over the primary copy LUN process on this memory node;And to from copy LUN synchronous write I/O data;Complete write IO request process with And after completing to write I/O data synchronization, result is returned described application server;When the ownership of described primary copy LUN stores joint When point is not this memory node, forward described write IO request to the ownership memory node of described primary copy LUN, so that described major-minor Described write IO request is submitted to the primary copy LUN process of own node by the ownership memory node of this LUN, and to same from copy LUN Step writes I/O data;The ownership memory node receiving described primary copy LUN completes write IO request process and is completing to write I/O data The result sent after synchronization, returns described result to described application server.
7. device as claimed in claim 6, it is characterised in that:
Described acquiring unit, when being additionally operable to the read I O request that memory node receives application server transmission, according to described reading IO The LUN identification information carried in request obtains the pair corresponding with described LUN identification information from the LUN relation list item of local record The ownership memory node of this LUN;
Described processing unit, is additionally operable to when comprising this memory node in the ownership memory node of described copy LUN, by described reading I/O Request submits the copy LUN process on this memory node to, and after read I O request has processed, result is returned described Application server;When the ownership memory node of described copy LUN does not comprises this memory node, deposit from the ownership of copy LUN The ownership memory node selecting a copy LUN in storage node forwards described read I O request, so that the ownership of the copy LUN selected Memory node submits described read I O request to copy LUN process on own node;Reception processes the copy of described read I O request The result that the ownership memory node of LUN sends, returns described result to described application server.
Device the most as claimed in claims 6 or 7, it is characterised in that described device also includes:
Trouble unit, is used for when the ownership memory node fault of primary copy LUN, and is this from the ownership memory node of copy LUN During memory node, obtaining the differentiated identification of arbitration equipment record, described differentiated identification is used for identifying described primary copy LUN and described The most consistent from the data of copy LUN;When described differentiated identification is data consistent, by this memory node from copy LUN It is defined as primary copy LUN, and is advertised to other memory node, so that other memory node redefines primary copy LUN.
9. device as claimed in claim 8, it is characterised in that:
Described trouble unit, is additionally operable to this memory node record difference bitmap, and described difference bitmap is used for recording in primary copy LUN The position of newly-increased data, and when there are newly-increased data, notifying that described arbitration equipment updates described differentiated identification is that data differ Cause;When the ownership memory node of the primary copy LUN of fault recovers normal, the primary copy LUN of normal memory node will be recovered It is defined as from copy LUN, and synchronizes this memory node primary copy LUN to this from copy LUN according to the difference bitmap of record Newly-increased data, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
Device the most as claimed in claims 6 or 7, it is characterised in that described device also includes:
Trouble unit, for working as the ownership memory node fault from copy LUN, and the ownership memory node of primary copy LUN is this During memory node, recording difference bitmap, described difference bitmap is for recording the position of newly-increased data in primary copy LUN, and is going out When now increasing data newly, notifying that described arbitration equipment updates described differentiated identification is that data are inconsistent;When fault from copy LUN's When ownership memory node recovers normal, according to the difference bitmap recorded to this from the newly-increased number of copy LUN synchronization primary copy LUN According to, and after completing newly-increased data syn-chronization, it is data consistent that notice arbitration equipment updates differentiated identification.
CN201610886961.6A 2016-10-11 2016-10-11 A kind of access method and device of storage resource Active CN106331166B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610886961.6A CN106331166B (en) 2016-10-11 2016-10-11 A kind of access method and device of storage resource

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610886961.6A CN106331166B (en) 2016-10-11 2016-10-11 A kind of access method and device of storage resource

Publications (2)

Publication Number Publication Date
CN106331166A true CN106331166A (en) 2017-01-11
CN106331166B CN106331166B (en) 2019-09-06

Family

ID=57821429

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610886961.6A Active CN106331166B (en) 2016-10-11 2016-10-11 A kind of access method and device of storage resource

Country Status (1)

Country Link
CN (1) CN106331166B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107171865A (en) * 2017-06-30 2017-09-15 郑州云海信息技术有限公司 A kind of multinode storage system and LUN ownernode switching methods
CN107329859A (en) * 2017-06-29 2017-11-07 杭州宏杉科技股份有限公司 A kind of data guard method and storage device
CN108153491A (en) * 2017-12-22 2018-06-12 深圳市瑞驰信息技术有限公司 A kind of storage method and framework for closing part server
CN109684860A (en) * 2018-12-29 2019-04-26 杭州宏杉科技股份有限公司 A kind of data ciphering method and device based on business relations
CN114201117A (en) * 2021-12-22 2022-03-18 苏州浪潮智能科技有限公司 Cache data processing method and device, computer equipment and storage medium
WO2022127762A1 (en) * 2020-12-14 2022-06-23 华为云计算技术有限公司 Cloud platform and bucket management method for object storage service provided thereby

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103885717A (en) * 2012-12-19 2014-06-25 杭州宏杉科技有限公司 Data replication method and data replication device
CN104461768A (en) * 2013-09-22 2015-03-25 华为技术有限公司 Copy storing device and copy storing method
CN105635310A (en) * 2016-01-20 2016-06-01 杭州宏杉科技有限公司 Access method and device for storage resource

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103885717A (en) * 2012-12-19 2014-06-25 杭州宏杉科技有限公司 Data replication method and data replication device
CN104461768A (en) * 2013-09-22 2015-03-25 华为技术有限公司 Copy storing device and copy storing method
CN105635310A (en) * 2016-01-20 2016-06-01 杭州宏杉科技有限公司 Access method and device for storage resource

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107329859A (en) * 2017-06-29 2017-11-07 杭州宏杉科技股份有限公司 A kind of data guard method and storage device
CN107171865A (en) * 2017-06-30 2017-09-15 郑州云海信息技术有限公司 A kind of multinode storage system and LUN ownernode switching methods
CN108153491A (en) * 2017-12-22 2018-06-12 深圳市瑞驰信息技术有限公司 A kind of storage method and framework for closing part server
CN109684860A (en) * 2018-12-29 2019-04-26 杭州宏杉科技股份有限公司 A kind of data ciphering method and device based on business relations
CN109684860B (en) * 2018-12-29 2020-08-14 杭州宏杉科技股份有限公司 Data encryption method and device based on business relation
WO2022127762A1 (en) * 2020-12-14 2022-06-23 华为云计算技术有限公司 Cloud platform and bucket management method for object storage service provided thereby
CN114201117A (en) * 2021-12-22 2022-03-18 苏州浪潮智能科技有限公司 Cache data processing method and device, computer equipment and storage medium
CN114201117B (en) * 2021-12-22 2023-09-01 苏州浪潮智能科技有限公司 Processing method and device for cache data, computer equipment and storage medium

Also Published As

Publication number Publication date
CN106331166B (en) 2019-09-06

Similar Documents

Publication Publication Date Title
CN106331166B (en) A kind of access method and device of storage resource
JP5102901B2 (en) Method and system for maintaining data integrity between multiple data servers across a data center
US7536586B2 (en) System and method for the management of failure recovery in multiple-node shared-storage environments
JP4791051B2 (en) Method, system, and computer program for system architecture for any number of backup components
US9256605B1 (en) Reading and writing to an unexposed device
US7668876B1 (en) Snapshot-based replication infrastructure for efficient logging with minimal performance effect
US6571354B1 (en) Method and apparatus for storage unit replacement according to array priority
US6598174B1 (en) Method and apparatus for storage unit replacement in non-redundant array
US7793148B2 (en) Using virtual copies in a failover and failback environment
US6553389B1 (en) Resource availability determination mechanism for distributed data storage system
US6782416B2 (en) Distributed and geographically dispersed quorum resource disks
CN103793271B (en) Methods and systems for switching between mirrored volumes
CN103840961A (en) Dual-computer hot backup system
US7761431B2 (en) Consolidating session information for a cluster of sessions in a coupled session environment
JP2005196683A (en) Information processing system, information processor and control method of information processing system
US7146526B2 (en) Data I/O system using a plurality of mirror volumes
CN104994168A (en) distributed storage method and distributed storage system
CN103593266A (en) ot standby method based on arbitration disk mechanism
CN105635310B (en) A kind of access method and device of storage resource
JP2012508925A (en) Active-active failover for direct attached storage systems
US7353285B2 (en) Apparatus, system, and method for maintaining task prioritization and load balancing
CN106502822A (en) Data read-write method and device
CN103970486A (en) Method and system for data storage
CN106528338A (en) Remote data replication method, storage equipment and storage system
Vallath Oracle real application clusters

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: Hangzhou City, Zhejiang Province, Binjiang District West Street 310053 Qianmo Road No. 482 building eleventh layer to the twelfth layer A

Applicant after: Hangzhou Sequoia Polytron Technologies Inc

Address before: Hangzhou City, Zhejiang Province, Binjiang District West Street 310053 Qianmo Road No. 482 building eleventh layer to the twelfth layer A

Applicant before: Hangzhou MacroSAN Technologies Co., Ltd.

COR Change of bibliographic data
GR01 Patent grant
GR01 Patent grant