CN113448516B - Data processing method, system, medium and equipment based on RAID card - Google Patents

Data processing method, system, medium and equipment based on RAID card Download PDF

Info

Publication number
CN113448516B
CN113448516B CN202110626707.3A CN202110626707A CN113448516B CN 113448516 B CN113448516 B CN 113448516B CN 202110626707 A CN202110626707 A CN 202110626707A CN 113448516 B CN113448516 B CN 113448516B
Authority
CN
China
Prior art keywords
processing
data
processing core
preset
raid card
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110626707.3A
Other languages
Chinese (zh)
Other versions
CN113448516A (en
Inventor
邢科钰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Yingxin Computer Technology Co Ltd
Original Assignee
Shandong Yingxin Computer Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Yingxin Computer Technology Co Ltd filed Critical Shandong Yingxin Computer Technology Co Ltd
Priority to CN202110626707.3A priority Critical patent/CN113448516B/en
Publication of CN113448516A publication Critical patent/CN113448516A/en
Application granted granted Critical
Publication of CN113448516B publication Critical patent/CN113448516B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0613—Improving I/O performance in relation to throughput
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0689—Disk arrays, e.g. RAID, JBOD
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a data processing method, a system, a medium and equipment based on a RAID card, wherein the method comprises the following steps: interconnecting the processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core; responding to the IOPS load distribution of each processing core to meet a first preset condition, and carrying out task dynamic allocation among the processing cores through a preset channel; and responding to the IOPS load distribution of each processing core to meet a second preset condition, classifying the data to be processed and respectively distributing the classified data to different processing cores for processing. The invention realizes the task load balance of each processing core through the task dynamic allocation; the data to be processed is classified and distributed to the corresponding processing cores for processing, so that the data processing is smoother and smoother, the data processing efficiency and the data processing capacity can be improved, and the I/O read-write performance and the data processing stability are improved.

Description

Data processing method, system, medium and equipment based on RAID card
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method, system, medium, and device based on a RAID card.
Background
RAID cards are increasingly being used in critical big data facilities such as front-end servers, mass storage servers, etc. due to their outstanding characteristics of security, stability, high redundancy, etc. The current mainstream RAID card can basically meet the performance requirements in the aspects of physical bandwidth, interface protocol, data transmission, security mechanism and the like based on hardware design, and process and store basic data. The existing RAID card controllers have fixed processing units, typically 2 processing units, each processing unit controlling a fixed number of interface devices, such as 9361-16I RAID cards, are dual-core controllers, each processing core controlling 8 SAS ports, respectively. The core task processing mechanism belongs to a segmentation independent mechanism and respectively executes own tasks.
The existing RAID card has poor data stability under high I/O (data input/output) load, and is mainly characterized in that data can be dropped in a large-area I/O read-write process, so that a processor cannot complete effective read-write processing. Moreover, the lack of balanced load among the processing cores affects the data reading and writing and the performance of the RAID card, and is mainly characterized by high I/O delay and lower data QoS. QoS (Quality of Service ) refers to a network that can utilize various basic technologies to provide better service capability for specified network communications, and is a security mechanism of the network, and is a technology for solving the problems of network delay and congestion. Therefore, the data processing capability of the current RAID card is still to be improved.
Disclosure of Invention
In view of the above, the present invention is directed to a data processing method, system, medium and device based on a RAID card, so as to solve the problem in the prior art that the data processing capability of the RAID card is low.
Based on the above object, the present invention provides a data processing method based on a RAID card, comprising the following steps:
interconnecting the processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core;
responding to the IOPS load distribution of each processing core to meet a first preset condition, and carrying out task dynamic allocation among the processing cores through a preset channel;
and responding to the IOPS load distribution of each processing core to meet a second preset condition, classifying the data to be processed and distributing the classified data to different processing cores for processing.
In some embodiments, interconnecting the processing cores of the RAID card controller through a preset channel, and monitoring the IOPS load amounts of the processing cores respectively includes: and interconnecting the two processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core.
In some embodiments, in response to the IOPS load distribution of each processing core meeting a first preset condition, dynamically allocating tasks among the processing cores through a preset channel comprises: and in response to the IOPS load of the first processing core exceeding a first preset threshold and the IOPS load of the second processing core being less than a second preset threshold, performing task dynamic allocation on the first processing core and the second processing core through a preset channel so as to balance the task amount processed by each processing core.
In some embodiments, in response to the IOPS load distribution of each processing core meeting the second preset condition, classifying the data to be processed and distributing the classified data to different processing cores for processing respectively includes: and responding to the IOPS load capacity of each processing core exceeding a third preset threshold value, classifying the data to be processed according to the size of the data block, and respectively distributing the classified data to different processing cores for processing.
In some embodiments, classifying the data to be processed by the data block size includes: the data to be processed is classified according to the data block size by a filter in firmware of the RAID card.
In some embodiments, the task dynamic allocation includes: and allocating an identification number for the task to be processed, and allocating the task to be processed to a corresponding processing core for processing based on the identification number.
In some embodiments, interconnecting the processing cores of the RAID card controller through a preset channel further comprises: and the processing cores of the RAID card controller are interconnected through a UPI channel.
In another aspect of the present invention, there is also provided a data processing system based on a RAID card, including:
the monitoring module is configured to interconnect the processing cores of the RAID card controller through a preset channel and monitor the IOPS load capacity of each processing core respectively;
the task dynamic allocation module is configured to respond to the condition that the IOPS load capacity distribution of each processing core meets a first preset condition, and perform task dynamic allocation among the processing cores through a preset channel; and
the data classification processing module is configured to respond to the condition that the IOPS load capacity distribution of each processing core meets a second preset condition, classify the data to be processed and respectively distribute the classified data to different processing cores for processing.
In yet another aspect of the present invention, there is also provided a computer readable storage medium storing computer program instructions which, when executed, implement any of the methods described above.
In yet another aspect of the present invention, there is also provided a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, performs any of the methods described above.
The invention has at least the following beneficial technical effects:
1. the invention provides the interconnection channel for each processing core of the RAID card controller, thereby being beneficial to the dynamic allocation of tasks among the processing cores, realizing the task load balance of each processing core, reducing the processing tasks for the processing cores with large loads and being beneficial to the continuous and stable data processing;
2. by classifying the data to be processed and distributing the classified data to the corresponding processing cores for processing, the processing cores can process certain types of data in full time, compared with the processing of scattered data, the processing of the data can be smoother and smoother, the bandwidth can be utilized more efficiently, and the efficiency of the data processing can be improved;
3. the invention improves the data processing capability of the processing core of the RAID card and improves the I/O read-write performance and the stability of data processing.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data processing method based on a RAID card according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a RAID card based data processing system provided in accordance with an embodiment of the present invention;
FIG. 3 is a schematic diagram of a computer readable storage medium implementing a RAID card based data processing method according to an embodiment of the present invention;
fig. 4 is a schematic hardware structure of a computer device for executing a data processing method based on a RAID card according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two non-identical entities with the same name or non-identical parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention. Furthermore, the terms "comprise" and "have," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, article, or other step or unit that comprises a list of steps or units.
Based on the above object, a first aspect of the embodiments of the present invention proposes an embodiment of a data processing method based on a RAID card. Fig. 1 is a schematic diagram of an embodiment of a data processing method based on a RAID card according to the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
step S10, interconnection is carried out among the processing cores of the RAID card controller through a preset channel, and the IOPS load capacity of each processing core is monitored respectively;
step S20, responding to the IOPS load distribution of each processing core to meet a first preset condition, and carrying out task dynamic allocation among the processing cores through a preset channel;
and step S30, responding to the IOPS load distribution of each processing core to meet a second preset condition, classifying the data to be processed and respectively distributing the classified data to different processing cores for processing.
RAID (Redundant Arrays of Independent Disks) it is a magnetic disk array, which is a magnetic disk array with huge capacity formed by combining a plurality of independent magnetic disks, and improves the performance of the whole magnetic disk system by using the addition effect generated by providing data by individual magnetic disks. With this technique, data is cut into a number of sections, which are stored on individual hard disks. By placing data on multiple hard disks, input and output operations can overlap in a balanced manner, improving performance.
IOPS (Input/Output Per Second), i.e., the data Input/output (or read/write times) per second, is one of the main indicators for measuring disk performance. IOPS refers to the number of I/O requests that a system can handle per unit time, typically in units of I/O requests per second, where an I/O request is typically a read or write data operation request. For applications with frequent random read-write, such as doclet storage (pictures), OLTP databases, mail servers, attention is paid to random read-write performance, so IOPS is a key measure. For applications with frequent sequential reading and writing, a large amount of continuous data needs to be transmitted, such as video editing of a television station, video on demand VOD (Video On Demand), and continuous reading and writing performance is concerned, so data throughput is a key measure.
IOPS load: the IOPS of the processing core of each RAID card controller is limited, and a core task load percentage value, that is, the current workload, is obtained by the ratio of the processing core's current task's IOPS throughput to the processing core's total IO processing throughput.
According to the embodiment of the invention, the interconnection channels are provided for the processing cores of the RAID card controller, so that the task dynamic allocation among the processing cores is facilitated, the task load balance of the processing cores is realized, the processing tasks are lightened for the processing cores with large loads, and the continuous and stable data processing is facilitated; by classifying the data to be processed and distributing the classified data to the corresponding processing cores for processing, the processing cores can process certain types of data in full time, compared with the processing of scattered data, the processing of the data can be smoother and smoother, the credit bandwidth can be utilized more efficiently, and the efficiency of the data processing can be improved; the invention improves the data processing capability of the processing core of the RAID card and improves the I/O read-write performance and the stability of data processing.
In some embodiments, interconnecting the processing cores of the RAID card controller through a preset channel, and monitoring the IOPS load amounts of the processing cores respectively includes: and interconnecting the two processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core.
In some embodiments, in response to the IOPS load distribution of each processing core meeting a first preset condition, dynamically allocating tasks among the processing cores through a preset channel comprises: and in response to the IOPS load of the first processing core exceeding a first preset threshold and the IOPS load of the second processing core being less than a second preset threshold, performing task dynamic allocation on the first processing core and the second processing core through a preset channel so as to balance the task amount processed by each processing core.
In this embodiment, when the RAID card controller is a dual processing core, the first preset threshold may be set to 70% and the second preset threshold may be set to 50%, that is, when the IOPS load of the first processing core is greater than 70% and the IOPS load of the second processing core is less than 50%, the tasks of the first processing core and the second processing core are dynamically allocated through the preset channel so as to balance the task amount processed by each processing core, thereby solving the problem of the I/O processing performance degradation caused by the high load of a certain processing core. The values of the first preset threshold and the second preset threshold are not limited to this, and may be set according to actual conditions, but a condition that the first preset threshold is larger than the second preset threshold needs to be satisfied.
In some embodiments, in response to the IOPS load distribution of each processing core meeting the second preset condition, classifying the data to be processed and distributing the classified data to different processing cores for processing respectively includes: and responding to the IOPS load capacity of each processing core exceeding a third preset threshold value, classifying the data to be processed according to the size of the data block, and respectively distributing the classified data to different processing cores for processing.
In some embodiments, classifying the data to be processed by the data block size includes: the data to be processed is classified according to the data block size by a filter in firmware of the RAID card.
In this embodiment, the third preset threshold may also be 70%, and when the IOPS load capacity of each processing core exceeds 70%, the data to be processed is classified according to the size of the data block by a filter in the firmware of the RAID card, and the classified data is distributed to the corresponding processing core for processing. If the RAID card controller is a dual processing core, the data to be processed may be classified according to small data blocks and large data blocks, for example, small data block ranges including 4K, 8K, 16K, and 32K, and large data block ranges including 64K, 128K, 256K, 512K, and 1024K, and then it may be configured that the small data blocks are processed by the first processing core and the large data blocks are processed by the second processing core. The sorting of the large and small data blocks realizes special core, and has the advantages that when the data bandwidth and the data processing task amount of the processor are fixed, the bandwidth can be more efficiently utilized by sorting and then processing the relatively scattered data blocks, the rapid processing of the data is realized, the time delay is reduced, the overall stability of the data IO is finally improved, and the phenomenon that the performance in the data processing drops greatly is avoided. The setting of the third preset threshold is not limited thereto, and may be set according to actual circumstances.
In some embodiments, the task dynamic allocation includes: and allocating an identification number for the task to be processed, and allocating the task to be processed to a corresponding processing core for processing based on the identification number.
In this embodiment, each task is assigned a unique identifier in an OS (operating system) kernel before being executed, so that the processor can perform task allocation and task execution.
In some embodiments, interconnecting the processing cores of the RAID card controller through a preset channel further comprises: and the processing cores of the RAID card controller are interconnected through a UPI channel.
In this embodiment, UPI (Ultra Path interconnect) represents a hyperlink, and the UPI bus can implement direct interconnection between chips, has the characteristics of high bandwidth and high transmission, and provides assistance for dynamic task allocation by using the UPI for interconnection between processing cores.
In other examples, the processed stable data is continuously and effectively transferred and read/written by means of RAID card buffer capacity, SAS bandwidth (channel connecting RAID card and disk), PCIE bandwidth (interconnection channel between CPU and RAID card) and disk bandwidth (data channel of each physical disk).
In a second aspect of the embodiment of the present invention, a data processing system based on a RAID card is also provided. FIG. 2 is a schematic diagram illustrating one embodiment of a RAID card based data processing system provided by the present invention. As shown in fig. 2, a data processing system based on a RAID card includes: the monitoring module 10 is configured to interconnect the processing cores of the RAID card controller through a preset channel, and monitor the IOPS load of each processing core; the task dynamic allocation module 20 is configured to perform task dynamic allocation between the processing cores through a preset channel in response to the IOPS load distribution of the processing cores meeting a first preset condition; and a data classification processing module 30 configured to classify the data to be processed and distribute the classified data to different processing cores respectively for processing in response to the IOPS load distribution of each processing core satisfying a second preset condition.
According to the data processing system based on the RAID card, the interconnection channels are provided for the processing cores of the RAID card controller, so that the task dynamic allocation among the processing cores is facilitated, the task load balance of the processing cores is realized, the processing tasks are lightened for the processing cores with large loads, and the continuous and stable data processing is facilitated; by classifying the data to be processed and distributing the classified data to the corresponding processing cores for processing, the processing cores can process certain types of data in full time, compared with the processing of scattered data, the processing of the data can be smoother and smoother, the bandwidth can be utilized more efficiently, and the efficiency of the data processing can be improved; the system of the invention improves the data processing capability of the processing core of the RAID card and improves the I/O read-write performance and the stability of data processing.
In some embodiments, the monitoring module 10 is further configured to interconnect two processing cores of the RAID card controller through a preset channel, and monitor the IOPS load of each processing core separately.
In some embodiments, the task dynamic allocation module 20 is further configured to dynamically allocate tasks to the first processing core and the second processing core through the preset channel to balance the amount of tasks processed by each processing core in response to the IOPS load of the first processing core exceeding a first preset threshold and the IOPS load of the second processing core being less than a second preset threshold.
In this embodiment, when the RAID card controller is a dual processing core, the first preset threshold may be set to 70% and the second preset threshold may be set to 50%, that is, when the IOPS load of the first processing core is greater than 70% and the IOPS load of the second processing core is less than 50%, the tasks of the first processing core and the second processing core are dynamically allocated through the preset channel so as to balance the task amount processed by each processing core, thereby solving the problem of the I/O processing performance degradation caused by the high load of a certain processing core.
In some embodiments, the data classification processing module 30 is further configured to classify the data to be processed according to the data block size in response to the IOPS load amounts of the processing cores exceeding the third preset threshold, and distribute the classified data to different processing cores for processing, respectively.
In some embodiments, the data classification processing module 30 includes a classification module configured to classify the data to be processed by a filter in firmware of the RAID card according to a data block size.
In this embodiment, the third preset threshold may also be 70%, and when the IOPS load capacity of each processing core exceeds 70%, the data to be processed is classified according to the size of the data block by a filter in the firmware of the RAID card, and the classified data is distributed to the corresponding processing core for processing.
In some embodiments, the task dynamic allocation module 20 includes an allocation module configured to allocate an identification number to a task to be processed, and allocate the task to be processed to a corresponding processing core for processing based on the identification number.
In some embodiments, the monitoring module 10 includes an interconnect module configured to interconnect the processing cores of the RAID card controller via a UPI channel.
In a third aspect of the embodiments of the present invention, a computer readable storage medium is provided, and fig. 3 shows a schematic diagram of a computer readable storage medium implementing a data processing method based on a RAID card according to an embodiment of the present invention. As shown in fig. 3, the computer-readable storage medium 3 stores computer program instructions 31, which computer program instructions 31 when executed implement the steps of:
interconnecting the processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core;
responding to the IOPS load distribution of each processing core to meet a first preset condition, and carrying out task dynamic allocation among the processing cores through a preset channel;
and responding to the IOPS load distribution of each processing core to meet a second preset condition, classifying the data to be processed and respectively distributing the classified data to different processing cores for processing.
In some embodiments, interconnecting the processing cores of the RAID card controller through a preset channel, and monitoring the IOPS load amounts of the processing cores respectively includes: and interconnecting the two processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core.
In some embodiments, in response to the IOPS load distribution of each processing core meeting a first preset condition, dynamically allocating tasks among the processing cores through a preset channel comprises: and in response to the IOPS load of the first processing core exceeding a first preset threshold and the IOPS load of the second processing core being less than a second preset threshold, performing task dynamic allocation on the first processing core and the second processing core through a preset channel so as to balance the task amount processed by each processing core.
In some embodiments, in response to the IOPS load distribution of each processing core meeting the second preset condition, classifying the data to be processed and distributing the classified data to different processing cores for processing respectively includes: and responding to the IOPS load capacity of each processing core exceeding a third preset threshold value, classifying the data to be processed according to the size of the data block, and respectively distributing the classified data to different processing cores for processing.
In some embodiments, classifying the data to be processed by the data block size includes: the data to be processed is classified according to the data block size by a filter in firmware of the RAID card.
In some embodiments, the task dynamic allocation includes: and allocating an identification number for the task to be processed, and allocating the task to be processed to a corresponding processing core for processing based on the identification number.
In some embodiments, interconnecting the processing cores of the RAID card controller through a preset channel further comprises: and the processing cores of the RAID card controller are interconnected through a UPI channel.
It should be appreciated that all of the embodiments, features and advantages set forth above with respect to a RAID card-based data processing method according to the present invention apply equally to a RAID card-based data processing system and storage medium according to the present invention without conflicting therewith.
In a fourth aspect of the embodiments of the present invention, there is also provided a computer device comprising a memory 402 and a processor 401, the memory storing a computer program which, when executed by the processor, implements the method of any of the embodiments described above.
Referring to fig. 4, a schematic hardware structure of an embodiment of a computer device for performing a data processing method based on a RAID card according to the present invention is shown. Taking the example of a computer device as shown in fig. 4, a processor 401 and a memory 402 are included in the computer device, and may further include: an input device 403 and an output device 404. The processor 401, memory 402, input device 403, and output device 404 may be connected by a bus or otherwise, for example in fig. 4. The input device 403 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the RAID card based data processing system. The output 404 may include a display device such as a display screen.
The memory 402 is used as a non-volatile computer readable storage medium, and may be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions/modules corresponding to the RAID card-based data processing method in the embodiments of the present application. Memory 402 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created by use of a data processing method based on a RAID card, or the like. In addition, memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, which may be connected to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The processor 401 executes various functional applications of the server and data processing, that is, implements the RAID card-based data processing method of the above-described method embodiment by running nonvolatile software programs, instructions, and modules stored in the memory 402.
Finally, it should be noted that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, RAM may be available in a variety of forms such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP and/or any other such configuration.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items. The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.

Claims (8)

1. A data processing method based on a RAID card, comprising the steps of:
interconnecting the processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core;
responding to the IOPS load capacity distribution of each processing core to meet a first preset condition, and carrying out task dynamic allocation among the processing cores through the preset channel;
responding to the IOPS load distribution of each processing core to meet a second preset condition, classifying the data to be processed and respectively distributing the classified data to different processing cores for processing;
the RAID card controller is provided with a first processing core and a second processing core, wherein the RAID card controller is provided with the first processing core and the second processing core;
responding to the IOPS load capacity distribution of each processing core to meet a first preset condition, and carrying out task dynamic allocation among the processing cores through the preset channel comprises the following steps: in response to the IOPS load of a first processing core exceeding a first preset threshold and the IOPS load of a second processing core being not greater than a second preset threshold, performing task dynamic allocation on the first processing core and the second processing core through the preset channel to balance the task amount processed by each processing core, wherein the first preset threshold is greater than the second preset threshold;
responding to the IOPS load distribution of each processing core meeting a second preset condition, classifying the data to be processed and respectively distributing the classified data to different processing cores for processing comprises the following steps: and responding to the IOPS load capacity of each processing core exceeding a third preset threshold, classifying the data to be processed according to the size of the data block, and respectively distributing the classified data to different processing cores for processing, wherein the third preset threshold is larger than or equal to the first preset threshold.
2. The method of claim 1, wherein interconnecting the processing cores of the RAID card controller through a preset channel, and monitoring the IOPS load of each processing core comprises:
and interconnecting the two processing cores of the RAID card controller through a preset channel, and respectively monitoring the IOPS load capacity of each processing core.
3. The method of claim 1, wherein classifying the data to be processed according to the data block size comprises:
and classifying the data to be processed according to the size of the data block by a filter in the firmware of the RAID card.
4. The method of claim 1, wherein the task dynamic allocation comprises:
and allocating an identification number for the task to be processed, and allocating the task to be processed to a corresponding processing core for processing based on the identification number.
5. The method of claim 1, wherein interconnecting processing cores of the RAID card controller through a predetermined channel further comprises:
and the processing cores of the RAID card controller are interconnected through a UPI channel.
6. A RAID card-based data processing system, comprising:
the monitoring module is configured to interconnect the processing cores of the RAID card controller through a preset channel and monitor the IOPS load capacity of each processing core respectively;
the task dynamic allocation module is configured to respond to the condition that the IOPS load capacity distribution of each processing core meets a first preset condition, and perform task dynamic allocation among the processing cores through the preset channel; and
the data classification processing module is configured to respond to the condition that the IOPS load capacity distribution of each processing core meets a second preset condition, classify the data to be processed and respectively distribute the classified data to different processing cores for processing;
the RAID card controller is provided with a first processing core and a second processing core, wherein the RAID card controller is provided with the first processing core and the second processing core;
the task dynamic allocation module is configured to respond to the condition that the IOPS load capacity distribution of each processing core meets a first preset condition, and the task dynamic allocation between each processing core through the preset channel comprises the following steps: in response to the IOPS load of a first processing core exceeding a first preset threshold and the IOPS load of a second processing core being not greater than a second preset threshold, performing task dynamic allocation on the first processing core and the second processing core through the preset channel to balance the task amount processed by each processing core, wherein the first preset threshold is greater than the second preset threshold;
the data classification processing module is configured to respond to the condition that the IOPS load capacity distribution of each processing core meets a second preset condition, classify the data to be processed and respectively distribute the classified data to different processing cores for processing, and the data classification processing module comprises: and responding to the IOPS load capacity of each processing core exceeding a third preset threshold, classifying the data to be processed according to the size of the data block, and respectively distributing the classified data to different processing cores for processing, wherein the third preset threshold is larger than or equal to the first preset threshold.
7. A computer readable storage medium, characterized in that computer program instructions are stored, which when executed implement the method of any of claims 1-5.
8. A computer device comprising a memory and a processor, wherein the memory has stored therein a computer program which, when executed by the processor, performs the method of any of claims 1-5.
CN202110626707.3A 2021-06-04 2021-06-04 Data processing method, system, medium and equipment based on RAID card Active CN113448516B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110626707.3A CN113448516B (en) 2021-06-04 2021-06-04 Data processing method, system, medium and equipment based on RAID card

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110626707.3A CN113448516B (en) 2021-06-04 2021-06-04 Data processing method, system, medium and equipment based on RAID card

Publications (2)

Publication Number Publication Date
CN113448516A CN113448516A (en) 2021-09-28
CN113448516B true CN113448516B (en) 2023-07-21

Family

ID=77810834

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110626707.3A Active CN113448516B (en) 2021-06-04 2021-06-04 Data processing method, system, medium and equipment based on RAID card

Country Status (1)

Country Link
CN (1) CN113448516B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114281531B (en) * 2021-12-10 2023-11-03 苏州浪潮智能科技有限公司 Method, system, storage medium and equipment for distributing CPU cores

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102184093A (en) * 2011-06-14 2011-09-14 复旦大学 Parallel computing array framework constructed by multi CELL processors
CN105528330B (en) * 2014-09-30 2019-05-28 杭州华为数字技术有限公司 The method, apparatus of load balancing is gathered together and many-core processor
CN106980533B (en) * 2016-01-18 2020-04-28 杭州海康威视数字技术股份有限公司 Task scheduling method and device based on heterogeneous processor and electronic equipment
CN111984407B (en) * 2020-08-07 2022-12-23 苏州浪潮智能科技有限公司 Data block read-write performance optimization method, system, terminal and storage medium

Also Published As

Publication number Publication date
CN113448516A (en) 2021-09-28

Similar Documents

Publication Publication Date Title
DE102019122363A1 (en) PROGRAMMABLE DOUBLE-ROW WORK MEMORY MODULE ACCELERATOR CARD (DIMM ACCELERATOR CARD)
CN104750557B (en) A kind of EMS memory management process and memory management device
US9454407B2 (en) Service resource allocation
US20160132541A1 (en) Efficient implementations for mapreduce systems
US20140379722A1 (en) System and method to maximize server resource utilization and performance of metadata operations
US20130219405A1 (en) Apparatus and method for managing data stream distributed parallel processing service
CN109981702B (en) File storage method and system
US11856054B2 (en) Quality of service (QOS) setting recommendations for volumes across a cluster
CN106027595A (en) Access log processing method and system for CDN node
CN111338579B (en) Read-write cache optimization method, system, terminal and storage medium based on storage pool
DE112021004473T5 (en) STORAGE-LEVEL LOAD BALANCING
CN114840140B (en) Cloud data caching method, device, equipment and storage medium
CN113448516B (en) Data processing method, system, medium and equipment based on RAID card
CN114595043A (en) IO (input/output) scheduling method and device
CN114138178B (en) IO processing method and system
CN110032469A (en) Data processing system and its operating method
DE102020132767A1 (en) Quality of Service (QoS) management of a storage system
CN111949681A (en) Data aggregation processing device and method and storage medium
CN109960662A (en) A kind of method for recovering internal storage and equipment
CN110990148A (en) Method, device and medium for optimizing storage performance
CN107273188B (en) Virtual machine Central Processing Unit (CPU) binding method and device
DE112021005833T5 (en) DISAGGREGATION OF CONTROL PATH AND DATA PATH
CN114968073A (en) Data prefetching method, equipment and system
EP3764618A1 (en) Method for efficient optimisation of storage occupancy
CN115033390B (en) Load balancing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant