CN107220124B - Path selection method and device - Google Patents

Path selection method and device Download PDF

Info

Publication number
CN107220124B
CN107220124B CN201710385460.4A CN201710385460A CN107220124B CN 107220124 B CN107220124 B CN 107220124B CN 201710385460 A CN201710385460 A CN 201710385460A CN 107220124 B CN107220124 B CN 107220124B
Authority
CN
China
Prior art keywords
target port
request
block write
write request
storage server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710385460.4A
Other languages
Chinese (zh)
Other versions
CN107220124A (en
Inventor
耿芳忠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN201710385460.4A priority Critical patent/CN107220124B/en
Publication of CN107220124A publication Critical patent/CN107220124A/en
Application granted granted Critical
Publication of CN107220124B publication Critical patent/CN107220124B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/10Program control for peripheral devices
    • G06F13/12Program control for peripheral devices using hardware independent of the central processor, e.g. channel or peripheral processor
    • G06F13/124Program control for peripheral devices using hardware independent of the central processor, e.g. channel or peripheral processor where hardware is a sequential transfer control unit, e.g. microprocessor, peripheral processor or state-machine
    • G06F13/128Program control for peripheral devices using hardware independent of the central processor, e.g. channel or peripheral processor where hardware is a sequential transfer control unit, e.g. microprocessor, peripheral processor or state-machine for dedicated transfers to a network

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Computer And Data Communications (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The embodiment of the invention discloses a path selection method and a device, which realize a new path selection strategy in an MPIO driver, namely distinguish read or write operations of IO requests, send all read operations and small block write operations through a target port group with higher performance, send large block write operations through a target port group with lower performance, and further realize a load balanced path selection strategy in the selected target port group. The new path selection strategy takes the performance requirements of the application server into consideration, the maximum utilization of cache resources on the storage server side and the minimum consumption of CPU resources.

Description

Path selection method and device
Technical Field
The invention relates to the technical field of data processing, in particular to a path selection method and a path selection device.
Background
In an enterprise-class application system, an application server for processing a service request and a Storage system are connected to each other through a Storage Area Network (SAN). In order to avoid a single point of failure and improve system reliability and IO (Input/Output) throughput, MPIO (Multiple Path IO) access is generally adopted between an application server and a storage device. As shown in fig. 1, the application server a is connected to a storage device with dual controllers through two SAN switches, and a total of 4 physical paths from the application server a to the storage device are implemented. With the help of the multi-path disk driver, the application server a may initiate an IO request to the storage device through any one path.
Multiple physical paths between the application server and the storage device are identified by an initial port-target port connection (I _ T nexus), where the target ports may be grouped according to access status, referred to as TPG (target port group). As indicated in FIG. 2, the storage server (storage) has two controllers (controller-0, controller-1), two target ports (t-port 0, t-port 1) on controller-0, and two target ports (t-port 2, t-port 3) on controller-1. The storage server is usually designed to have the same consistent target port access status for the target ports on the same controller, so that the target ports on the same controller can be divided into a target port group. In fig. 2, two target port groups, TPG0 and TPG1, may have different performance when accessing LUN0 (logical Unit Number, also commonly used to refer to a logical Unit) through different target port groups, for example, the performance is higher when accessing LUN0 through TPG0, and the performance is lower when accessing LUN0 through TPG1, which is ALUA (Asymmetric logical Unit Access). The multipath driver (MPIO driver) may discover the definitions of the storage server side with respect to the set of target ports, create corresponding objects, and refer to these objects in path selection.
In order to realize the advantage of multi-path access, the multi-path driver may select an appropriate path to send the IO request through a variety of path selection policies. The current main path selection strategies are divided into the following 3 types:
failover. Normally, only one path is used to send an IO request, and only when the path in use fails, the other path is selected.
round robin. The path is used in a round-robin manner, that is, a certain amount of IO requests are sent through the path a first, and then a certain amount of IO requests are sent through the path B.
load balance load balancing. Through a path IO load calculation method, a path with the minimum current load is selected for sending an IO request.
However, the above path selection measurement cannot satisfy both the performance requirements of the application server, the maximum utilization of cache resources on the storage server side, and the minimum consumption of CPU resources.
Disclosure of Invention
In view of this, embodiments of the present invention provide a path selection method and apparatus, which can meet the performance requirement of an application server, maximize the utilization of cache resources on the storage server side, and minimize the consumption of CPU resources.
In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:
a method of path selection, comprising:
receiving an IO request sent by an application server side;
resolving the IO request into a read request, a small block write request and a large block write request;
and selecting a target port group with higher performance to send the read request and the small block write request to a storage server, and selecting a target port group with lower performance to send the large block write request to the storage server.
Preferably, the parsing the IO request into a read request, a small block write request, and a large block write request includes:
and resolving the OPERATION CODE field and the TRANSFER LENGTH field of the SCSI command, and dividing the IO request into a read request, a small block write request and a large block write request.
Preferably, the selecting a target port group with higher performance to send the read request and the small block write request to a storage server includes:
and selecting one target port from the target port group with higher performance according to a load balancing strategy or a round robin strategy to send the read request and the small block write request to a storage server.
Preferably, the selecting a target port group with lower performance to send the chunk write request to a storage server includes:
and selecting one target port from the target port group with lower performance according to a load balancing strategy or a round robin strategy to send the large block write request to a storage server.
A path selection device comprising:
the receiving module is used for receiving an IO request sent by an application server side;
the analysis module is used for analyzing the IO request into a read request, a small block write request and a large block write request;
and the path selection module is used for selecting a target port group with higher performance to send the read request and the small block write request to a storage server, and selecting a target port group with lower performance to send the large block write request to the storage server.
Preferably, the parsing module is specifically configured to:
and resolving the OPERATION CODE field and the TRANSFER LENGTH field of the SCSI command, and dividing the IO request into a read request, a small block write request and a large block write request.
Preferably, the path selection module is specifically configured to:
and selecting one target port from the target port group with higher performance according to a load balancing strategy or a round robin strategy to send the read request and the small block write request to a storage server.
Preferably, the path selection module is specifically configured to:
and selecting one target port from the target port group with lower performance according to a load balancing strategy or a round robin strategy to send the large block write request to a storage server.
Based on the technical scheme, the embodiment of the invention discloses a path selection method and a path selection device, which are used for receiving an IO request sent by an application server side; resolving the IO request into a read request, a small block write request and a large block write request; and selecting a target port group with lower performance to send the large block write request to the storage server, thereby meeting the performance requirements of the application server, the maximum utilization of cache resources on the side of the storage server and the minimum consumption of CPU resources.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic architecture diagram between an application server and a storage device according to the background art of the present invention;
FIG. 2 is a diagram illustrating multiple physical paths between an application server and a storage device according to the background art of the present invention;
fig. 3 is a schematic flow chart of a path selection method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a command description of write (10) according to an embodiment of the present invention;
fig. 5 is a diagram illustrating a path selection method according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a path selection device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The MPIO driver program operated on the application server side is responsible for selecting one path from the multiple paths and sending the IO request to the storage server. The invention realizes a new path selection strategy through an improved MPIO driver, and the specific implementation mode is described as follows:
referring to fig. 3, fig. 3 is a schematic flow chart of a path selection method according to an embodiment of the present invention, where an execution subject of the method is an MPIO driver executed by an application server side, and the method specifically includes the following steps:
step S100, receiving an IO request sent by the application server side.
Step S110, the IO request is analyzed into a read request, a small block write request and a large block write request.
It should be noted that the parsing the IO request into a read request, a small block write request, and a large block write request includes: and resolving the OPERATION CODE field and the TRANSFER LENGTH field of the SCSI command, and dividing the IO request into a read request, a small block write request and a large block write request.
Each SCSI command has a defined command operator (OPERATION CODE). The specification defines a set of read and write operation commands, where the command operator for read (10) is 0x28 and the command operator for write (10) is 0x2 a. The present invention will use SCSI command operators and parameters to implement the defined features. FIG. 4 is a schematic diagram of command description of write (10), byte0(OPERATION CODE) is an operator, byte2-byte5(LOGIC BLOCK ADDRESS) is a start BLOCK ADDRESS, and byte7-byte8(TRANSFER LENGTH) is a BLOCK number.
Step S120, selecting a target port group with higher performance to send the read request and the small block write request to a storage server, and selecting a target port group with lower performance to send the large block write request to the storage server.
It should be noted that, the selecting a target port group with higher performance to send the read request and the small block write request to a storage server includes: and selecting one target port from the target port group with higher performance according to a load balancing strategy or a round robin strategy to send the read request and the small block write request to a storage server. The selecting the target port group with lower performance to send the large block write request to a storage server comprises the following steps: and selecting one target port from the target port group with lower performance according to a load balancing strategy or a round robin strategy to send the large block write request to a storage server.
The path selection method provided in the embodiment of the present invention implements a new path selection policy in the MPIO driver, that is, distinguishes read or write operations of an IO request, sends all read operations and small block write operations through a target port group with higher performance, sends large block write operations through a target port group with lower performance, and implements a load balancing path selection policy in the selected target port group. The new path selection strategy takes the performance requirements of the application server into consideration, the maximum utilization of cache resources on the storage server side and the minimum consumption of CPU resources.
Referring to fig. 5, fig. 5 is a diagram illustrating a path selection method according to an embodiment of the present invention, as shown in fig. 5, MPIO drivers may access LUN0 through TPG0 and TPG1, where TPG0 provides higher IO performance and TPG1 provides lower IO performance.
Step 1, analyzing an OPERATION CODE field and an TRANSFER LENGTH field of an SCSI command by a dispatch module for a block io request of a disk drive, and sorting the following 3 types:
in the read request, the OPERATION CODE field indicates that the OPERATION is read ().
OPERATION CODE indicates that the OPERATION is write (), and the TRANSFER LENGTH field is less than or equal to a specified value (e.g., TRANSFER LENGTH ═ 8 in the case where block size is 512).
The OPERATION CODE indicates that the OPERATION is write (), and the TRANSFER LENGTH field is greater than a specified value (e.g., TRANSFER LENGTH >8 in the case of a block size of 512).
And 2, selecting one path from paths in TPG0 through LB (load balance) or RR (round robin) strategies, and sending a read request and a small block write request. One path is selected among the paths within TPG1 through LB or RR policies to send a bulk write request.
In the MPIO driver, the IO requests are divided into 3 types (read requests, small block write requests and large block write requests), and the different types of IO requests are sent to different control nodes of the storage server. The nodes responsible for handling large block write operations need not maintain the cache of LUNs, only the overhead of writing buffer. The node responsible for the read operation and the small block write operation is responsible for maintaining the cache of the LUN and executing the operation between the cache and the medium. Therefore, the cache resources of the storage server can be utilized to the maximum extent, cache synchronization operation among nodes is reduced, time spent by the storage server on node synchronization operation is saved, and the application server can obtain the optimal IO performance.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a path selection device according to an embodiment of the present invention, where the path selection device specifically includes:
a receiving module 100, configured to receive an IO request sent by an application server side;
an analysis module 110, configured to analyze the IO request into a read request, a small block write request, and a large block write request;
the path selection module 120 is configured to select a target port group with higher performance to send the read request and the small block write request to a storage server, and select a target port group with lower performance to send the large block write request to the storage server.
Preferably, the parsing module is specifically configured to:
and resolving the OPERATION CODE field and the TRANSFER LENGTH field of the SCSI command, and dividing the IO request into a read request, a small block write request and a large block write request.
Preferably, the path selection module is specifically configured to:
and selecting one target port from the target port group with higher performance according to a load balancing strategy or a round robin strategy to send the read request and the small block write request to a storage server.
Preferably, the path selection module is specifically configured to:
and selecting one target port from the target port group with lower performance according to a load balancing strategy or a round robin strategy to send the large block write request to a storage server.
In summary, the following steps:
the path selection method and the path selection device provided by the embodiment of the invention realize a new path selection strategy in the MPIO driver, namely distinguish read or write operations of IO requests, send all read operations and small block write operations through a target port group with higher performance, send large block write operations through a target port group with lower performance, and further realize a load balanced path selection strategy in the selected target port group. The new path selection strategy takes the performance requirements of the application server into consideration, the maximum utilization of cache resources on the storage server side and the minimum consumption of CPU resources.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (6)

1. A method for path selection, comprising:
receiving an IO request sent by an application server side, wherein the application server side runs an MPIO driver;
analyzing an OPERATION CODE field and an TRANSFER LENGTH field of the SCSI command, dividing the IO request into a read request, a small block write request and a large block write request, and sending different types of IO requests to different control nodes of the storage server;
and selecting a target port group with higher performance to send the read request and the small block write request to a storage server, and selecting a target port group with lower performance to send the large block write request to the storage server.
2. The method of claim 1, wherein the selecting the higher-performance target port group to send the read request and the small block write request to a storage server comprises:
and selecting one target port from the target port group with higher performance according to a load balancing strategy or a round robin strategy to send the read request and the small block write request to a storage server.
3. The method of claim 1, wherein selecting the lower performing target port group to send the chunk write request to a storage server comprises:
and selecting one target port from the target port group with lower performance according to a load balancing strategy or a round robin strategy to send the large block write request to a storage server.
4. A path selection device, comprising:
the receiving module is used for receiving an IO request sent by an application server side, and the application server side runs an MPIO driver;
the analysis module is used for analyzing the OPERATION CODE field and the TRANSFER LENGTH field of the SCSI command, dividing the IO request into a read request, a small block write request and a large block write request, sending different types of IO requests to different control nodes of the storage server;
and the path selection module is used for selecting a target port group with higher performance to send the read request and the small block write request to a storage server, and selecting a target port group with lower performance to send the large block write request to the storage server.
5. The apparatus of claim 4, wherein the path selection module is specifically configured to:
and selecting one target port from the target port group with higher performance according to a load balancing strategy or a round robin strategy to send the read request and the small block write request to a storage server.
6. The apparatus of claim 4, wherein the path selection module is specifically configured to:
and selecting one target port from the target port group with lower performance according to a load balancing strategy or a round robin strategy to send the large block write request to a storage server.
CN201710385460.4A 2017-05-26 2017-05-26 Path selection method and device Active CN107220124B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710385460.4A CN107220124B (en) 2017-05-26 2017-05-26 Path selection method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710385460.4A CN107220124B (en) 2017-05-26 2017-05-26 Path selection method and device

Publications (2)

Publication Number Publication Date
CN107220124A CN107220124A (en) 2017-09-29
CN107220124B true CN107220124B (en) 2021-01-12

Family

ID=59947060

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710385460.4A Active CN107220124B (en) 2017-05-26 2017-05-26 Path selection method and device

Country Status (1)

Country Link
CN (1) CN107220124B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109918027B (en) * 2019-05-16 2019-08-09 上海燧原科技有限公司 Store access control method, device, equipment and storage medium
CN111190546B (en) * 2019-12-30 2023-10-20 深圳创新科软件技术有限公司 Distributed block storage performance optimization method based on ALUA and local cache

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103236985A (en) * 2013-04-02 2013-08-07 浪潮电子信息产业股份有限公司 Multipath load balancing system for accessing storage
CN103294786A (en) * 2013-05-17 2013-09-11 华中科技大学 Metadata organization and management method and system of distributed file system
CN103530387A (en) * 2013-10-22 2014-01-22 浪潮电子信息产业股份有限公司 Improved method aimed at small files of HDFS
CN104090847A (en) * 2014-06-25 2014-10-08 华中科技大学 Address distribution method of solid-state storage device
CN105099912A (en) * 2015-08-07 2015-11-25 浪潮电子信息产业股份有限公司 Multipath data scheduling method and device
CN106126600A (en) * 2016-06-21 2016-11-16 浪潮电子信息产业股份有限公司 A kind of perform the method for read and write abruption, Apparatus and system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103236985A (en) * 2013-04-02 2013-08-07 浪潮电子信息产业股份有限公司 Multipath load balancing system for accessing storage
CN103294786A (en) * 2013-05-17 2013-09-11 华中科技大学 Metadata organization and management method and system of distributed file system
CN103530387A (en) * 2013-10-22 2014-01-22 浪潮电子信息产业股份有限公司 Improved method aimed at small files of HDFS
CN104090847A (en) * 2014-06-25 2014-10-08 华中科技大学 Address distribution method of solid-state storage device
CN105099912A (en) * 2015-08-07 2015-11-25 浪潮电子信息产业股份有限公司 Multipath data scheduling method and device
CN106126600A (en) * 2016-06-21 2016-11-16 浪潮电子信息产业股份有限公司 A kind of perform the method for read and write abruption, Apparatus and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ALUA,AA,多路径;Pierson-Wu;《CSDN论坛》;20151230;第1页 *

Also Published As

Publication number Publication date
CN107220124A (en) 2017-09-29

Similar Documents

Publication Publication Date Title
US9916275B2 (en) Preventing input/output (I/O) traffic overloading of an interconnect channel in a distributed data storage system
US10296223B2 (en) Methods and apparatus for controlling data reading from a storage system
US8893146B2 (en) Method and system of an I/O stack for controlling flows of workload specific I/O requests
US9015519B2 (en) Method and system for cluster wide adaptive I/O scheduling by a multipathing driver
US8902736B2 (en) Selecting an alternative path for an input/output request
US20160092136A1 (en) Method and apparatus for cost-based load balancing for port selection
US11366590B2 (en) Host device with multi-path layer providing dynamic control of one or more path selection algorithms
US7827251B2 (en) Fast write operations to a mirrored volume in a volume manager
US20080235448A1 (en) Storage apparatus and storage area arrangement method
US20160217049A1 (en) Fibre Channel Failover Based on Fabric Connectivity
US20160077996A1 (en) Fibre Channel Storage Array Having Standby Controller With ALUA Standby Mode for Forwarding SCSI Commands
US9158714B2 (en) Method and system for multi-layer differential load balancing in tightly coupled clusters
US20190155922A1 (en) Server for torus network-based distributed file system and method using the same
US8650576B2 (en) Storage apparatus and storage apparatus management method
US10063425B1 (en) Event-based in-band host registration
CN107220124B (en) Path selection method and device
US9300530B2 (en) Management device, management method, and medium
US11934679B2 (en) System and method for segmenting volumes across a multi-node storage system
US8473643B2 (en) Method and apparatus of tier storage management awareness networking
US10521380B1 (en) Using storage controllers to respond to multipath input/output requests
CN116909953A (en) Multipath planning method, device and medium
CN102546652B (en) System and method for server load balancing
US11693577B2 (en) Storage operation processing during data migration using migrated indicator from source storage
US11704056B2 (en) Independent set data lanes for IOD SSD
US20140324951A1 (en) Network selecting apparatus and operating method thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20201202

Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province

Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd.

Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601

Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd.

TA01 Transfer of patent application right
GR01 Patent grant
GR01 Patent grant