CN112835853B - Data processing type determining method and device - Google Patents
Data processing type determining method and device Download PDFInfo
- Publication number
- CN112835853B CN112835853B CN202011634046.0A CN202011634046A CN112835853B CN 112835853 B CN112835853 B CN 112835853B CN 202011634046 A CN202011634046 A CN 202011634046A CN 112835853 B CN112835853 B CN 112835853B
- Authority
- CN
- China
- Prior art keywords
- information
- data processing
- historical
- determining
- processing type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 211
- 238000000034 method Methods 0.000 title claims abstract description 138
- 230000008569 process Effects 0.000 claims abstract description 96
- 238000013500 data storage Methods 0.000 claims abstract description 54
- 238000012549 training Methods 0.000 claims description 31
- 238000004891 communication Methods 0.000 claims description 18
- 238000013136 deep learning model Methods 0.000 claims description 18
- 238000003062 neural network model Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 230000003993 interaction Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013145 classification model Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012847 principal component analysis method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
Abstract
The embodiment of the invention provides a method and a device for determining a data processing type, and relates to the technical field of data processing, wherein the method comprises the following steps: acquiring first I/O information of a first process for providing data access service for a client in a data storage system within a preset time period; the first I/O information is compared with the historical I/O information in characteristics, and second I/O information similar to the first I/O information is determined from the historical I/O information. And determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client. By applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for determining a data processing type.
Background
The data storage system is a commonly used system for providing services such as data storage, data reading and writing, and the like, and is used for providing a data interface, providing data access service for a client through the data interface and performing corresponding data processing. The number, kind, etc. of data required by the client in the process of performing data processing of different data processing types are different. In order to provide different data access services to clients when they perform different data processing, the data storage system needs to determine the data processing type of the data processing performed by the client.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing type determining method and device for determining the data processing type of data processing performed by a client. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a method for determining a data processing type, where the method includes:
acquiring first I/O information of a first process for providing data access service for a client in a data storage system within a preset time period;
comparing the first I/O information with historical I/O information in characteristics, and determining second I/O information similar to the first I/O information from the historical I/O information;
and determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client.
In one embodiment of the present invention, the comparing the first I/O information with the historical I/O information to determine second I/O information similar to the first I/O information from the historical I/O information includes:
extracting characteristics of the first I/O information;
calculating the similarity between the extracted features and historical features, wherein the historical features are as follows: features of historical I/O information corresponding to the training type of the deep learning model;
if the similarity is higher than the preset similarity, determining that the historical I/O information corresponding to the historical features is second I/O information.
In one embodiment of the present invention, the comparing the first I/O information with the historical I/O information to determine second I/O information similar to the first I/O information from the historical I/O information includes:
inputting the first I/O information into a pre-trained processing type determining model, determining second I/O information similar to the first I/O information, and outputting a data processing type corresponding to the second I/O information, wherein the processing type determining model is as follows: and taking the sample historical I/O information as input of a neural network model, taking the data processing type corresponding to the sample historical I/O information as training supervision, and training the neural network model to obtain a model which is used for determining the data processing type corresponding to the I/O information.
In one embodiment of the present invention, after the determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client, the method further includes:
switching a process for providing a data access service for the client into a second process, wherein the second process is: a process for providing data access services for clients performing the determined type of data processing.
In one embodiment of the present invention, the first I/O information includes at least one of the following information:
the number of times the file is opened, the number of times the file is closed, the number of times the file is read.
In a second aspect, an embodiment of the present invention provides a data processing type determining apparatus, including:
the information acquisition module is used for acquiring first I/O information of a first process for providing data access service for the client in the data storage system within a preset time period;
the characteristic comparison module is used for carrying out characteristic comparison on the first I/O information and the historical I/O information, and determining second I/O information similar to the first I/O information from the historical I/O information;
and the type determining module is used for determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client.
In one embodiment of the present invention, the feature comparison module is specifically configured to:
extracting characteristics of the first I/O information;
calculating the similarity between the extracted features and historical features, wherein the historical features are as follows: features of historical I/O information corresponding to the training type of the deep learning model;
if the similarity is higher than the preset similarity, determining that the historical I/O information corresponding to the historical features is second I/O information.
In one embodiment of the present invention, the feature comparison module is specifically configured to:
inputting the first I/O information into a pre-trained processing type determining model, determining second I/O information similar to the first I/O information, and outputting a data processing type corresponding to the second I/O information, wherein the processing type determining model is as follows: and taking the sample historical I/O information as input of a neural network model, taking the data processing type corresponding to the sample historical I/O information as training supervision, and training the neural network model to obtain a model which is used for determining the data processing type corresponding to the I/O information.
In one embodiment of the invention, the apparatus further comprises:
the process switching module is configured to switch a process for providing a data access service for the client to a second process, where the second process is: a process for providing data access services for clients performing the determined type of data processing.
In one embodiment of the present invention, the first I/O information includes at least one of the following information:
the number of times the file is opened, the number of times the file is closed, the number of times the file is read.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of the above first aspects when executing a program stored on a memory.
In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored therein, which when executed by a processor, implements the method steps of any of the first aspects described above.
In a fifth aspect, embodiments of the present invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of any of the first aspects described above.
The embodiment of the invention has the beneficial effects that:
when the scheme provided by the embodiment of the invention is applied to determining the data processing type, the first I/O information of the first process for providing the data access service for the client in the data storage system within the preset time length can be compared with the historical I/O information in characteristics, the second I/O information similar to the first I/O information is determined from the historical I/O information, and the data processing type corresponding to the second I/O information is determined as the data processing type of the data processing performed by the client.
From the above, since the client performs the data processing, the client may perform I/O data interaction with the data storage system to obtain data. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, by applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a first method for determining a data processing type according to an embodiment of the present invention;
FIG. 2 is a flowchart of a second method for determining a data processing type according to an embodiment of the present invention;
FIG. 3 is a flowchart of a third method for determining a data processing type according to an embodiment of the present invention;
FIG. 4 is a flowchart of a fourth method for determining a data processing type according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a first data processing type determining apparatus according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a second data processing type determining apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order to determine a data processing type of data processing performed by a client, the embodiment of the invention provides a data processing type determining method and device.
In one embodiment of the present invention, there is provided a data processing type determining method, including:
and obtaining first I/O information of a first process for providing data access service for the client in the data storage system within a preset time period.
And comparing the first I/O information with the historical I/O information in characteristics, and determining second I/O information similar to the first I/O information from the historical I/O information.
And determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client.
From the above, since the client performs the data processing, the client may perform I/O data interaction with the data storage system to obtain data. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, according to the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
First, an execution body of an embodiment of the present invention will be described. Specifically, the execution body may be an electronic device included in the data storage system, or may be another electronic device other than the data storage system. If the execution body of the embodiment of the invention is the other electronic device, after determining the data processing type of the data processing performed by the client, the other electronic device may send the determined data processing type to the data storage system, so that the data storage system can determine the data processing type of the data processing performed by the client.
The method and the device for determining the data processing type provided by the embodiment of the invention are explained by a specific embodiment.
Referring to fig. 1, an embodiment of the present invention provides a flowchart of a first data processing type determining method, which may be implemented through the following steps S101 to S103.
S101: and obtaining first I/O information of a first process for providing data access service for the client in the data storage system within a preset time period.
Specifically, the data storage system can access to a plurality of clients, and the data storage system can provide a plurality of services for each client, wherein the services comprise data access services, and the data access services comprise data read-write services and the like. In the process of data reading and writing, the client and the data storage system need to perform Input/Output (I/O) data interaction.
Each service may be implemented by a process, and after each client accesses the data storage system to request the data access service, the data storage system may create at least one process for the client. The first process for providing the data access service for the client may generate first I/O information during operation, where the first I/O information may include at least one of a number of times a file is opened, a number of times a file is closed, and a number of times a file is read, where the file is a file stored in the data storage system. Wherein the specific content included in the first I/O information is different depending on the specific data access service requested by the user.
In addition, the preset duration may be 30s, 60s, etc., and the first I/O information may be first I/O information of a first process within the preset duration from when the client accesses data from the data storage system through the first process for the first time.
In this case, the first I/O information may be obtained by statistics of the client, and in this case, the client obtains the first I/O information that is sent to the electronic device after the first I/O information is obtained, that is, the electronic device may obtain the first I/O information by receiving information sent by the client. The client may send the first I/O information to an electronic device executing an embodiment of the present invention according to HTTP (HyperText Transfer Protocol ) or other protocols.
In another case, if the electronic device is one device in the data storage system, the first I/O information may be obtained by statistics of the electronic device itself.
S102: and comparing the first I/O information with the historical I/O information in characteristics, and determining second I/O information similar to the first I/O information from the historical I/O information.
Specifically, the historical I/O information may include I/O information corresponding to different data processing types. After the client accesses and obtains data from the data storage system through the process provided by the data storage system, the client uses the obtained data to perform data processing, so that the process can correspond to the data processing performed by the client. The historical I/O information of the process corresponds to data processing performed by the client, the data processing may be classified into different data processing types, and the historical I/O information corresponds to the data processing types.
For example, the data processing types may include a deep learning model training type, an image processing type, a character processing type, and the like.
In one embodiment of the present invention, since the data processing types corresponding to the historical I/O information are different, the features of the historical I/O information are different, for example, if the historical I/O information corresponds to the image processing type, the features of the historical I/O information indicate that the client obtains more image data, and if the historical I/O information corresponds to the character processing type, the features of the historical I/O information indicate that the client obtains more character data.
Therefore, the first I/O information can be subjected to feature extraction, the extracted features of the first I/O information are compared with the features of the historical I/O information, the similarity between the features of the first I/O information and the features of the historical I/O information is determined, the historical I/O information with the similarity higher than the preset historical similarity is determined to be second I/O information similar to the first I/O information.
Specifically, the characteristics of the historical I/O information may be obtained by extracting the characteristics of the historical I/O information in the process of executing the scheme provided by the embodiment of the present invention. In addition, since the historical I/O information is obtained in advance, the features of the historical I/O information may also be obtained in advance before executing the scheme provided by the embodiment of the present invention.
In one embodiment of the present invention, the feature of the first I/O information may be extracted by an algorithm such as a principal component analysis method or a linear discriminant analysis method, which is not limited in the embodiment of the present invention.
S103: and determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client.
Specifically, the historical I/O information is obtained in advance, so that the information of the corresponding data processing type can be obtained in advance. The data processing type corresponding to the second I/O information in the historical I/O information is known and accurate.
Since the first I/O information is similar to the second I/O information, the data processing type corresponding to the first I/O information can be regarded as the same as the data processing type corresponding to the second I/O information. Since the data processing type corresponding to the second I/O information is accurate, the determined data processing type corresponding to the first I/O information can be considered to be accurate. Because the first I/O information is the first I/O information of the first process in the process that the client accesses and obtains data from the data storage system through the first process for data processing. Therefore, the data processing type corresponding to the second I/O information can be determined as the data processing type of the data processing performed by the client.
From the above, since the client performs the data processing, the client may perform I/O data interaction with the data storage system to obtain data. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, by applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
Referring to fig. 2, a flowchart of a second method for determining a data processing type according to an embodiment of the present invention is shown. In contrast to the previous embodiment shown in fig. 1, the above step S102 may be implemented by the following steps S102A-S102C.
S102A: and extracting the characteristics of the first I/O information.
Specifically, the content represented by the extracted features is different according to the difference of the data contained in the first I/O information.
For example, if the first I/O information includes the data amount of the data of the client that obtains the different types of data through the first process within the preset duration, the extracted feature may represent that the client that obtains the data with the highest data amount, or obtains the data with the lowest data amount, etc. through the first process within the preset duration.
If the first I/O information includes the number of times that the client obtains the data through the first process within the preset duration, the extracted feature may represent the number of times that the client obtains the data through the first process within the preset duration.
S102B: similarity between the extracted features and the historical features is calculated.
Wherein, the history features are: features of historical I/O information corresponding to the deep learning model training type.
Specifically, a large amount of data is required to be used in the training process of the deep learning model, so that the training process of the deep learning model is completed. Compared with other data processing, the client needs to acquire a large amount of data from the data storage system, and the times of opening the file, closing the file and reading the file are higher in the process of acquiring the data. Therefore, the historical characteristics are different from the characteristics of the historical I/O information corresponding to other data processing types, and the historical I/O information corresponding to the training type of the deep learning model can be distinguished from other historical I/O information through the historical characteristics.
In one embodiment of the present invention, the similarity between the extracted features and the history features may be calculated by a common calculation method in the prior art, such as a cosine similarity algorithm, a manhattan distance algorithm, a pearson correlation coefficient algorithm, etc., which is not limited in this embodiment of the present invention.
S102C: if the similarity is higher than the preset similarity, determining the historical I/O information corresponding to the historical characteristics as second I/O information.
For example, the preset similarity may be 70%, 80%, or the like.
If the similarity is higher than a preset similarity, the first I/O information may be considered to be I/O information of the same category as the historical I/O information corresponding to the historical feature, and the historical I/O information corresponding to the historical feature may be determined to be second I/O information.
From the above, the similarity between the features of the first I/O information and the historical features of the historical I/O information corresponding to the training type of the deep learning model is calculated, and if the similarity is higher than a preset similarity, the historical I/O information corresponding to the historical features is determined to be the second I/O information. Therefore, the data processing type of the data processing performed by the client can be accurately determined to be the deep learning model training type.
Referring to fig. 3, a flowchart of a third data processing type determining method according to an embodiment of the present invention is shown, and compared with the embodiment shown in fig. 1, the above step S102 may be implemented by the following step S102D.
S102D: inputting the first I/O information into a pre-trained processing type determining model, determining second I/O information similar to the first I/O information, and outputting a data processing type corresponding to the second I/O information.
Wherein, the processing type determining model is as follows: and taking the sample historical I/O information as input of a neural network model, taking the data processing type corresponding to the sample historical I/O information as training supervision, and training the neural network model to obtain a model which is used for determining the data processing type corresponding to the I/O information.
Specifically, the sample history I/O information may include information included in the history I/O information, or may include other I/O information obtained in advance.
The sample history I/O information is information of known corresponding data processing type, the sample history I/O information is input into a neural network model, the neural network model is trained by taking the data processing type corresponding to the sample history I/O information as training supervision, and the processing type determining model learns the characteristics of a large amount of sample history I/O information. Thus, after the first I/O information is input into the above-described process type determination model, the trained process type determination model is able to identify the characteristics of the first I/O information. Thereby outputting the data processing type corresponding to the first I/O information.
The processing type determining model may output a data processing type corresponding to the first I/O information, for example, a deep learning model training type, an image processing type, a character processing type, or the like, that is, the first I/O information is classified into one of the deep learning model training type, the image processing type, and the character processing type. The above-described process type determination model can be considered as a classification model.
The process type determination model may output whether the data process type corresponding to the first I/O information is a certain type, for example, whether the data process type is a deep learning model training type, or the like. That is, the output result of the process type determination model is yes or no, and the process type determination model may be regarded as a classification model.
From the above, it can be seen that the scheme provided by the embodiment of the present invention uses the pre-trained processing type determination model to determine the data processing type corresponding to the first I/O information. Since the processing type determination model is trained using a large number of sample history I/O information, the processing type determination model learns the characteristics of the large number of sample history I/O information, so that the data processing type corresponding to the first I/O information can be accurately determined.
Referring to fig. 4, a flowchart of a fourth data processing type determining method according to an embodiment of the present invention is shown, and compared with the embodiment shown in fig. 1, the method further includes the following step S104 after the step S103.
S104: and switching the process for providing the data access service for the client into a second process.
Wherein, the second process is as follows: a process for providing data access services for clients performing the determined type of data processing.
Specifically, since the data processing types of the data processing performed by the client are different, the amount, the kind, and the like of the data accessed and obtained by the client from the data storage system are different, the process for providing the data access service for the client can be switched to the second process matched with the determined data processing type, so that the data access service can be better provided for the client.
For example, if the determined data processing type is an image processing type, the above-described second process may be a process suitable for transmitting an image, such as a process using a protocol suitable for transmitting an image.
If the determined data processing type is the deep learning model training type, the client needs to frequently acquire data from the data storage system, so the second process can be suitable for a process of frequently transmitting data.
From the above, after determining the data processing type of the data processing performed by the client, the process of providing the data access service by the client may be switched to the second process matched with the determined data processing type, so that the data storage system may better provide the data access service for the client.
Corresponding to the foregoing data processing type determining method, referring to fig. 5, an embodiment of the present invention further provides a schematic structural diagram of a first data processing type determining apparatus, where the apparatus includes:
the information obtaining module 501 is configured to obtain first I/O information of a first process in a data storage system for providing a data access service for a client within a preset duration;
the feature comparison module 502 is configured to perform feature comparison on the first I/O information and historical I/O information, and determine second I/O information similar to the first I/O information from the historical I/O information;
and a type determining module 503, configured to determine a data processing type corresponding to the second I/O information as a data processing type of the data processing performed by the client.
From the above, since the client performs the data processing, the client may perform I/O data interaction with the data storage system to obtain data. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, by applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
In one embodiment of the present invention, the feature comparison module 502 is specifically configured to:
extracting characteristics of the first I/O information;
calculating the similarity between the extracted features and historical features, wherein the historical features are as follows: features of historical I/O information corresponding to the training type of the deep learning model;
if the similarity is higher than the preset similarity, determining that the historical I/O information corresponding to the historical features is second I/O information.
From the above, the similarity between the features of the first I/O information and the historical features of the historical I/O information corresponding to the training type of the deep learning model is calculated, and if the similarity is higher than a preset similarity, the historical I/O information corresponding to the historical features is determined to be the second I/O information. Therefore, the data processing type of the data processing performed by the client can be accurately determined to be the deep learning model training type.
In one embodiment of the present invention, the feature comparison module 502 is specifically configured to:
inputting the first I/O information into a pre-trained processing type determining model, determining second I/O information similar to the first I/O information, and outputting a data processing type corresponding to the second I/O information, wherein the processing type determining model is as follows: and taking the sample historical I/O information as input of a neural network model, taking the data processing type corresponding to the sample historical I/O information as training supervision, and training the neural network model to obtain a model which is used for determining the data processing type corresponding to the I/O information.
From the above, it can be seen that the scheme provided by the embodiment of the present invention uses the pre-trained processing type determination model to determine the data processing type corresponding to the first I/O information. Since the processing type determination model is trained using a large number of sample history I/O information, the processing type determination model learns the characteristics of the large number of sample history I/O information, so that the data processing type corresponding to the first I/O information can be accurately determined.
Referring to fig. 6, a schematic structural diagram of a second data processing type determining apparatus according to an embodiment of the present invention, compared with the embodiment shown in fig. 5, the apparatus further includes:
a process switching module 504, configured to switch a process for providing a data access service for the client to a second process, where the second process is: a process for providing data access services for clients performing the determined type of data processing.
From the above, after determining the data processing type of the data processing performed by the client, the process of providing the data access service by the client may be switched to the second process matched with the determined data processing type, so that the data storage system may better provide the data access service for the client.
In one embodiment of the present invention, the first I/O information includes at least one of the following information:
the number of times the file is opened, the number of times the file is closed, the number of times the file is read.
The embodiment of the present invention further provides an electronic device, as shown in fig. 7, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, where the processor 701, the communication interface 702, and the memory 703 perform communication with each other through the communication bus 704,
a memory 703 for storing a computer program;
the processor 701 is configured to implement any of the method steps described in the above-described data processing type determination method when executing the program stored in the memory 703.
When the electronic equipment provided by the embodiment of the invention determines the data processing type, the client can perform I/O data interaction with the data storage system in the process of data processing so as to acquire data. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, by applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
The communication bus mentioned above for the electronic devices may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, there is also provided a computer readable storage medium having stored therein a computer program which when executed by a processor implements the steps of any of the above-described data processing type determining methods.
When the computer program stored in the computer readable storage medium provided by the embodiment determines the data processing type, the client can perform I/O data interaction with the data storage system to obtain data during the data processing process. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, by applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the data processing type determining methods of the above embodiments.
When the computer program product provided in this embodiment is executed to determine the data processing type, the client may perform I/O data interaction with the data storage system during the data processing process, so as to obtain data. In the case of different data processing by the client, the characteristics of the first I/O information of the first process, in which the data storage system provides the data access service for the client, are different in the process of performing I/O data interaction with the data storage system by the client. Therefore, by applying the scheme provided by the embodiment of the invention, the data processing type of the data processing performed by the client can be determined through the first I/O information.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the apparatus, the electronic device, the computer-readable storage medium and the computer program product, the description is relatively simple, as it is substantially similar to the method embodiments, and relevant points are found in the partial description of the method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (12)
1. A method of determining a type of data processing, the method comprising:
acquiring first I/O information of a first process for providing data access service for a client in a data storage system within a preset time period;
comparing the first I/O information with historical I/O information in characteristics, and determining second I/O information similar to the first I/O information from the historical I/O information;
and determining the data processing type which corresponds to the second I/O information and is known in advance as the data processing type of the data processing performed by the client.
2. The method of claim 1, wherein the comparing the first I/O information with historical I/O information, determining second I/O information from the historical I/O information that is similar to the first I/O information, comprises:
extracting characteristics of the first I/O information;
calculating the similarity between the extracted features and historical features, wherein the historical features are as follows: features of historical I/O information corresponding to the training type of the deep learning model;
if the similarity is higher than the preset similarity, determining that the historical I/O information corresponding to the historical features is second I/O information.
3. The method of claim 1, wherein the comparing the first I/O information with historical I/O information, determining second I/O information from the historical I/O information that is similar to the first I/O information, comprises:
inputting the first I/O information into a pre-trained processing type determining model, determining second I/O information similar to the first I/O information, and outputting a data processing type corresponding to the second I/O information, wherein the processing type determining model is as follows: and taking the sample historical I/O information as input of a neural network model, taking the data processing type corresponding to the sample historical I/O information as training supervision, and training the neural network model to obtain a model which is used for determining the data processing type corresponding to the I/O information.
4. The method according to claim 1, further comprising, after the determining the data processing type corresponding to the second I/O information as the data processing type of the data processing performed by the client:
switching a process for providing a data access service for the client into a second process, wherein the second process is: a process for providing data access services for clients performing the determined type of data processing.
5. The method of any of claims 1-4, wherein the first I/O information comprises at least one of:
the number of times the file is opened, the number of times the file is closed, the number of times the file is read.
6. A data processing type determining apparatus, characterized in that the apparatus comprises:
the information acquisition module is used for acquiring first I/O information of a first process for providing data access service for the client in the data storage system within a preset time period;
the characteristic comparison module is used for carrying out characteristic comparison on the first I/O information and the historical I/O information, and determining second I/O information similar to the first I/O information from the historical I/O information;
and the type determining module is used for determining the data processing type which corresponds to the second I/O information and is known in advance as the data processing type of the data processing performed by the client.
7. The apparatus of claim 6, wherein the feature comparison module is specifically configured to:
extracting characteristics of the first I/O information;
calculating the similarity between the extracted features and historical features, wherein the historical features are as follows: features of historical I/O information corresponding to the training type of the deep learning model;
if the similarity is higher than the preset similarity, determining that the historical I/O information corresponding to the historical features is second I/O information.
8. The apparatus of claim 6, wherein the feature comparison module is specifically configured to:
inputting the first I/O information into a pre-trained processing type determining model, determining second I/O information similar to the first I/O information, and outputting a data processing type corresponding to the second I/O information, wherein the processing type determining model is as follows: and taking the sample historical I/O information as input of a neural network model, taking the data processing type corresponding to the sample historical I/O information as training supervision, and training the neural network model to obtain a model which is used for determining the data processing type corresponding to the I/O information.
9. The apparatus of claim 6, wherein the apparatus further comprises:
the process switching module is configured to switch a process for providing a data access service for the client to a second process, where the second process is: a process for providing data access services for clients performing the determined type of data processing.
10. The apparatus according to any of claims 6-9, wherein the first I/O information comprises at least one of:
the number of times the file is opened, the number of times the file is closed, the number of times the file is read.
11. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for carrying out the method steps of any one of claims 1-5 when executing a program stored on a memory.
12. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein a computer program which, when executed by a processor, implements the method steps of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011634046.0A CN112835853B (en) | 2020-12-31 | 2020-12-31 | Data processing type determining method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011634046.0A CN112835853B (en) | 2020-12-31 | 2020-12-31 | Data processing type determining method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112835853A CN112835853A (en) | 2021-05-25 |
| CN112835853B true CN112835853B (en) | 2024-03-22 |
Family
ID=75926184
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011634046.0A Active CN112835853B (en) | 2020-12-31 | 2020-12-31 | Data processing type determining method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112835853B (en) |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002140211A (en) * | 2000-11-02 | 2002-05-17 | Ricoh Co Ltd | Information processor and its inter-process communication history acquiring method |
| JP2003167715A (en) * | 2001-11-29 | 2003-06-13 | Ricoh Co Ltd | Method for displaying communication history among processes, program for making computer execute the same, image formation device, and image formation system |
| CN101013427A (en) * | 2006-02-03 | 2007-08-08 | 国际商业机器公司 | Method and system for managing data |
| US7797284B1 (en) * | 2007-04-25 | 2010-09-14 | Netapp, Inc. | Dedicated software thread for communicating backup history during backup operations |
| JP2016038721A (en) * | 2014-08-07 | 2016-03-22 | 日本電気株式会社 | Malware estimation device, malware estimation method, and malware estimation program |
| CN108076157A (en) * | 2017-12-29 | 2018-05-25 | 北京奇虎科技有限公司 | Message content push control method, system and computer equipment |
| US10078337B1 (en) * | 2017-07-14 | 2018-09-18 | Uber Technologies, Inc. | Generation of trip estimates using real-time data and historical data |
| CN109343789A (en) * | 2018-06-05 | 2019-02-15 | 深圳市木浪云数据有限公司 | A kind of reading accelerated method, device and electronic equipment based on IO scene Recognition |
| US10474496B1 (en) * | 2018-10-17 | 2019-11-12 | EMC IP Holding Company LLC | Dynamic multitasking for distributed storage systems by detecting events for triggering a context switch |
| CN110472011A (en) * | 2019-07-19 | 2019-11-19 | 平安科技（深圳）有限公司 | A kind of cost of litigation prediction technique, device and terminal device |
| CN110704390A (en) * | 2019-09-17 | 2020-01-17 | 平安科技（深圳）有限公司 | Method, device, electronic equipment and medium for obtaining server maintenance script |
| CN110782129A (en) * | 2019-09-29 | 2020-02-11 | 中国银联股份有限公司 | Business progress monitoring method, device and system and computer readable storage medium |
| CN111176565A (en) * | 2019-12-25 | 2020-05-19 | 三星（中国）半导体有限公司 | Method and device for determining storage load of application |
| CN111368864A (en) * | 2018-12-26 | 2020-07-03 | 阿里巴巴集团控股有限公司 | Identification method, availability evaluation method and device, electronic equipment and storage medium |
| CN111752918A (en) * | 2020-05-15 | 2020-10-09 | 南京国电南自维美德自动化有限公司 | Historical data interaction system and configuration method thereof |
| CN111930974A (en) * | 2020-08-10 | 2020-11-13 | 北京字节跳动网络技术有限公司 | Audio and video type recommendation method, device, equipment and storage medium |
| CN111931049A (en) * | 2020-08-02 | 2020-11-13 | 吕维东 | Business processing method based on big data and artificial intelligence and block chain financial platform |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008134705A (en) * | 2006-11-27 | 2008-06-12 | Hitachi Ltd | Data processing method and data analysis device |
| US20120102568A1 (en) * | 2010-10-26 | 2012-04-26 | Mcafee, Inc. | System and method for malware alerting based on analysis of historical network and process activity |
| CN110457577B (en) * | 2015-04-01 | 2023-04-14 | 创新先进技术有限公司 | Data processing method, device, equipment and computer storage medium |
| CN107978311B (en) * | 2017-11-24 | 2020-08-25 | 腾讯科技（深圳）有限公司 | Voice data processing method and device and voice interaction equipment |
-
2020
- 2020-12-31 CN CN202011634046.0A patent/CN112835853B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002140211A (en) * | 2000-11-02 | 2002-05-17 | Ricoh Co Ltd | Information processor and its inter-process communication history acquiring method |
| JP2003167715A (en) * | 2001-11-29 | 2003-06-13 | Ricoh Co Ltd | Method for displaying communication history among processes, program for making computer execute the same, image formation device, and image formation system |
| CN101013427A (en) * | 2006-02-03 | 2007-08-08 | 国际商业机器公司 | Method and system for managing data |
| US7797284B1 (en) * | 2007-04-25 | 2010-09-14 | Netapp, Inc. | Dedicated software thread for communicating backup history during backup operations |
| JP2016038721A (en) * | 2014-08-07 | 2016-03-22 | 日本電気株式会社 | Malware estimation device, malware estimation method, and malware estimation program |
| US10078337B1 (en) * | 2017-07-14 | 2018-09-18 | Uber Technologies, Inc. | Generation of trip estimates using real-time data and historical data |
| CN108076157A (en) * | 2017-12-29 | 2018-05-25 | 北京奇虎科技有限公司 | Message content push control method, system and computer equipment |
| CN109343789A (en) * | 2018-06-05 | 2019-02-15 | 深圳市木浪云数据有限公司 | A kind of reading accelerated method, device and electronic equipment based on IO scene Recognition |
| US10474496B1 (en) * | 2018-10-17 | 2019-11-12 | EMC IP Holding Company LLC | Dynamic multitasking for distributed storage systems by detecting events for triggering a context switch |
| CN111368864A (en) * | 2018-12-26 | 2020-07-03 | 阿里巴巴集团控股有限公司 | Identification method, availability evaluation method and device, electronic equipment and storage medium |
| CN110472011A (en) * | 2019-07-19 | 2019-11-19 | 平安科技（深圳）有限公司 | A kind of cost of litigation prediction technique, device and terminal device |
| CN110704390A (en) * | 2019-09-17 | 2020-01-17 | 平安科技（深圳）有限公司 | Method, device, electronic equipment and medium for obtaining server maintenance script |
| CN110782129A (en) * | 2019-09-29 | 2020-02-11 | 中国银联股份有限公司 | Business progress monitoring method, device and system and computer readable storage medium |
| CN111176565A (en) * | 2019-12-25 | 2020-05-19 | 三星（中国）半导体有限公司 | Method and device for determining storage load of application |
| CN111752918A (en) * | 2020-05-15 | 2020-10-09 | 南京国电南自维美德自动化有限公司 | Historical data interaction system and configuration method thereof |
| CN111931049A (en) * | 2020-08-02 | 2020-11-13 | 吕维东 | Business processing method based on big data and artificial intelligence and block chain financial platform |
| CN111930974A (en) * | 2020-08-10 | 2020-11-13 | 北京字节跳动网络技术有限公司 | Audio and video type recommendation method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| EAST即时通信系统;王兰;肖炳甲;袁旗平;;计算机系统应用;20180815(第08期);112-117 * |
| 基于霍克斯过程的社交网络用户关系强度模型;于岩;陈鸿昶;于洪涛;;电子学报;20160615(第06期);101-107 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112835853A (en) | 2021-05-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110069709B (en) | Intention recognition method, device, computer readable medium and electronic equipment | |
| CN107040397B (en) | Service parameter acquisition method and device | |
| CN111047429A (en) | Probability prediction method and device | |
| CN112966102A (en) | Classification model construction and text sentence classification method, equipment and storage medium | |
| WO2023011470A1 (en) | Machine learning system and model training method | |
| CN112529663A (en) | Commodity recommendation method and device, terminal equipment and storage medium | |
| CN112017777B (en) | Method and device for predicting similar pair problem and electronic equipment | |
| CN111222051B (en) | Training method and device for trend prediction model | |
| CN112835853B (en) | Data processing type determining method and device | |
| CN114693011A (en) | Policy matching method, device, equipment and medium | |
| CN116016365B (en) | Webpage identification method based on data packet length information under encrypted flow | |
| CN111159464A (en) | Audio clip detection method and related equipment | |
| CN115660060A (en) | Model training method, detection method, device, equipment and storage medium | |
| CN115795207A (en) | Method and system for automatically identifying fraud-related websites | |
| CN113255319B (en) | Model training method, text segmentation method, abstract extraction method and device | |
| CN115984734A (en) | Model training method, video recall method, model training device, video recall device, electronic equipment and storage medium | |
| CN111984867B (en) | Network resource determining method and device | |
| CN114065858A (en) | Model training method and device, electronic equipment and storage medium | |
| CN111080349B (en) | Method, device, server and medium for identifying multiple devices of same user | |
| CN114398228A (en) | Method and device for predicting equipment resource use condition and electronic equipment | |
| CN112580917B (en) | Customer characteristic evaluation method and device | |
| CN114048392B (en) | Multimedia resource pushing method and device, electronic equipment and storage medium | |
| CN116737939B (en) | Meta learning method, text classification device, electronic equipment and storage medium | |
| CN116383029B (en) | User behavior label generation method and device based on small program | |
| CN110750979B (en) | Method for determining continuity of chapters and detection device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |