CN116594566A - Disk scanning method, device, equipment and storage medium - Google Patents
Disk scanning method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116594566A CN116594566A CN202310607304.3A CN202310607304A CN116594566A CN 116594566 A CN116594566 A CN 116594566A CN 202310607304 A CN202310607304 A CN 202310607304A CN 116594566 A CN116594566 A CN 116594566A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- information
- disk
- machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000012545 processing Methods 0.000 claims abstract description 34
- 238000007781 pre-processing Methods 0.000 claims abstract description 28
- 238000012546 transfer Methods 0.000 claims description 6
- 230000001105 regulatory effect Effects 0.000 claims description 4
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 18
- 230000006870 function Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000011160 research Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a disk scanning method, a device, equipment and a storage medium, which are characterized in that the region where a target machine currently corresponding to a disk to be scanned is located and machine configuration information are obtained by responding to a disk scanning instruction, data definition configuration information corresponding to the target machine is determined according to the information, and then the data definition configuration information is analyzed to generate a plurality of data definitions and links corresponding to each data definition; preprocessing all acquired disk information based on each data definition to obtain target data of each category; and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into a target machine, thereby completing the scanning of the magnetic disk. According to the scheme, the disk information can be selectively scanned into the target machine, so that the scanning of the disk can be independently and flexibly realized, and the scanned data can be conveniently researched and analyzed.
Description
Technical Field
The present application relates to the field of data scanning technologies, and in particular, to a method, an apparatus, a device, and a storage medium for scanning a disk.
Background
Data collection and accumulation are required in many industries and are used as the basis for training and analysis of a system or a model. Particularly, in the technical field of automatic driving, in order to facilitate engineers to perfect control of an automatic driving vehicle, driving data of the automatic driving vehicle needs to be collected frequently, and analysis and integration are performed on the collected data to optimize the control of the automatic driving vehicle, wherein the most commonly used method is to collect various data of the automatic driving vehicle by using a magnetic disk, take the magnetic disk off the vehicle after the vehicle runs for one day, and scan the data in the magnetic disk into a machine or a server for subsequent analysis and research on the data.
However, the existing disk scanning method cannot perform autonomous configuration according to the scanning uploading destination corresponding to the disk, and cannot flexibly scan out the data in the disk for different types of data and different business contents, so that analysis and research on the data by the industry personnel are inconvenient.
Disclosure of Invention
In view of the above, the present application provides a method, apparatus, device and storage medium for scanning a disk, which are used for solving the problems that the existing method for scanning a disk cannot perform autonomous configuration according to a scanning uploading destination corresponding to the disk, cannot flexibly scan out data in the disk for different types of data and different service contents, and is inconvenient for the industry personnel to analyze and study the data.
In order to achieve the above object, the following schemes are proposed:
in a first aspect, a disk scanning method includes:
responding to a disk scanning instruction, acquiring the region and machine configuration information of a target machine corresponding to a disk to be scanned, and determining data definition configuration information corresponding to the target machine according to the region and machine configuration information of the target machine;
analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category;
and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
Preferably, the determining the data definition configuration information corresponding to the target machine according to the region where the target machine is located and the machine configuration information includes:
Determining a current operating period of the target machine;
and sending the current operation period, the region where the target machine is located and the machine configuration information to a cloud end of the target machine, so that the cloud end can determine data definition configuration information corresponding to the region where the target machine is located and the machine configuration information in the current operation period from a cloud end database.
Preferably, after the parsing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition, the method further includes:
acquiring the current environment of the target machine;
if the current environment of the target machine does not belong to the preset full bandwidth environment, performing secondary analysis on the data definition configuration information to generate bandwidth limitation information;
and limiting the target machine by the bandwidth limiting information.
Preferably, the preprocessing is performed on all the disk information based on each data definition to obtain target data of each category, including:
marking each piece of information in all the disc information by utilizing each preset data type to obtain each piece of marked information;
From the marked information, taking the marked information corresponding to the preset type to be scanned as the information to be scanned;
comparing each piece of information to be scanned with each data definition to determine the data definition corresponding to the piece of information to be scanned;
and combining and dividing each piece of information to be scanned to obtain target data of each category.
Preferably, the preprocessing is performed on all the disk information based on each data definition to obtain target data of each category, including:
in all the disk information, each piece of data defined by each piece of data is taken as each piece of first data;
for each piece of first data, acquiring a scanning path, a category, a data size and creation time of the piece of first data;
combining the scanning path, the category, the data size and the creation time of the first data to obtain catalog data corresponding to the first data, and integrating the first data and the catalog data corresponding to the first data to obtain target data corresponding to the first data;
and taking the target data corresponding to other first data with the same definition as the data to which the first data belongs and the target data corresponding to the first data as the target data of the same category, thereby obtaining the target data of each category.
Preferably, the linear processing of the target data of each category according to the link corresponding to each data definition is performed to transfer the target data of each category into the target machine, so as to complete the scanning of the disk, including:
aiming at each data definition corresponding link, acquiring the data scanning priority corresponding to the link;
sorting the target data of each category according to the data scanning priority corresponding to each link;
and carrying out linear processing on the target data of each category in turn according to the sequencing order so as to transmit the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
Preferably, after the link corresponding to each data definition performs linear processing on the target data of each category to transfer the target data of each category into the target machine, the method further includes:
judging whether the current memory space of the target machine remains;
if yes, calculating the residual memory space of the target machine, and adjusting the link corresponding to each data definition according to the residual space;
If not, the bandwidth of the place where the target machine is located is up-regulated.
In a second aspect, a disk scanning apparatus includes:
the data definition configuration information determining module is used for responding to a disk scanning instruction, acquiring the region where a target machine corresponding to a disk to be scanned is located and machine configuration information, and determining the data definition configuration information corresponding to the target machine according to the region where the target machine is located and the machine configuration information;
the analysis module is used for analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
the preprocessing module is used for acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category;
and the linear processing module is used for carrying out linear processing on the target data of each category according to the links corresponding to each data definition so as to transmit the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
In a third aspect, a disk scanning apparatus includes a memory and a processor;
The memory is used for storing programs;
the processor is configured to execute the program to implement the steps of the disk scanning method according to the first aspect.
In a fourth aspect, a storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the disk scanning method according to the first aspect.
According to the technical scheme, the method and the device acquire the region and the machine configuration information of the target machine corresponding to the magnetic disk to be scanned currently by responding to the magnetic disk scanning instruction, and determine the data definition configuration information corresponding to the target machine according to the region and the machine configuration information of the target machine; analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition; acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category; and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk. According to the scheme, the data definition configuration information is acquired by utilizing the region and self configuration of the target machine with the predetermined corresponding relation, and the data definition and the corresponding link can be obtained after analysis, so that the preprocessing mode and the linear processing mode for the disk information are determined according to the data definition and the corresponding link, the disk information can be selectively scanned into the target machine, the scanning of the disk is automatically and flexibly realized, and the subsequent research and analysis of the scanned data by the personnel in the industry are facilitated.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is an alternative flow chart of a disk scanning method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a configuration parsing module according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data identification module according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a magnetic disk scanning device according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a disk scanning apparatus according to an embodiment of the present application;
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Data collection and accumulation are required in many industries and are used as the basis for training and analysis of a system or a model. Especially in the technical field of automatic driving, the stability and reliability of an automatic driving model are realized by training various data, such as real vehicle video data, laser radar data, data collected by other sensors and the like, so that the data need to be collected and accumulated, besides, in order to facilitate the improvement of the control of the automatic driving vehicle by engineers, when the automatic driving vehicle has problems or errors, the corresponding various sensors and system log data can be analyzed and developed on the automatic driving vehicle, so that after the necessary data on which the automatic driving model is trained are collected, redundant data are additionally collected for analysis and error positioning of the system, and the collected data are analyzed and integrated to optimize the control of the automatic driving vehicle, wherein the most commonly used data of the automatic driving vehicle are collected by using a magnetic disk, the magnetic disk is taken off from the vehicle after the vehicle runs for one day, and the data in the magnetic disk are scanned into a machine or a server for subsequent analysis and research on the data.
However, the existing disk scanning method cannot perform autonomous configuration according to the scanning uploading destination corresponding to the disk, and cannot flexibly scan out the data in the disk for different types of data and different business contents, so that analysis and research on the data by the industry personnel are inconvenient.
The embodiment of the application provides a disk scanning method, which can be applied to various computer terminals or intelligent terminals, wherein an execution subject of the method can be a processor or a server of the computer terminal or the intelligent terminal, and a flow chart of the method is shown in fig. 1, and specifically comprises the following steps:
s1: and responding to a disk scanning instruction, acquiring the region and the machine configuration information of a target machine corresponding to a disk to be scanned, and determining data definition configuration information corresponding to the target machine according to the region and the machine configuration information of the target machine.
In the application, the region where the target machine corresponding to the magnetic disk to be scanned is located and the machine configuration information are obtained in response to the instruction initiated by the user or the target machine to scan the magnetic disk. It will be appreciated that the target machine itself has a certain storage space, if all the disc information cannot be scanned to the target machine at one time, after the disc is reinserted to the operation end (such as an autopilot vehicle), the remaining disc information will be covered by the information newly produced by the autopilot vehicle, which may cause information deletion and discarding, so that the machine configuration information of the target machine needs to be obtained, and the storage space of the target machine is determined therefrom, so as to avoid the above-mentioned process.
S2: and analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition.
The process of analyzing the data definition configuration information can be regarded as an autonomous configuration process, and the process is divided into a plurality of data definitions or a plurality of different data definitions according to the types of all disk data, and meanwhile, each data definition needs to be defined with a corresponding link, so that the self-definition of the data and the uploading link is realized, and the user can conveniently scan and control the disk information.
S3: and acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category.
In this step, all disc information in the disc is obtained, and all disc information is preprocessed according to the data definition determined in the previous step, so as to obtain target data of each category, specifically, the preprocessing may be to divide all disc information according to a plurality of data definitions, for example, the data definition is divided according to the type of data, then all disc information will be divided into a plurality of types, and each type corresponds to a plurality of pieces of information; or the data definition is divided according to the alarm level degree of the data, and the disk information divided into the same data definition belongs to one alarm level, which is not limited in the embodiment.
S4: and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
In many cases, when data communication is performed, a plurality of links are often connected in series to form a path between two ports, so in the application, different links are correspondingly arranged for different data definitions, and different disk information can be distinguished. In one example, for a magnetic disk in an autonomous vehicle, alarm information of the autonomous vehicle is divided into data 1 by data definition, normal running information of the autonomous vehicle is divided into data 2 by data definition, a link corresponding to the data 1 is a link 1, a link corresponding to the data 2 is a link 2, and in addition, information transmission modes of the link 1 and the link 2 are different, transmission speeds are different, so that scanning efficiency is also different. Therefore, the linear processing is carried out on the target data of each category according to the link corresponding to each data definition, so that the efficiency of disk scanning can be improved, and important disk information can be screened from the target data for preferential uploading.
According to the technical scheme, the method and the device acquire the region and the machine configuration information of the target machine corresponding to the magnetic disk to be scanned currently by responding to the magnetic disk scanning instruction, and determine the data definition configuration information corresponding to the target machine according to the region and the machine configuration information of the target machine; analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition; acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category; and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk. According to the scheme, the data definition configuration information is acquired by utilizing the region and self configuration of the target machine with the predetermined corresponding relation, and the data definition and the corresponding link can be obtained after analysis, so that the preprocessing mode and the linear processing mode for the disk information are determined according to the data definition and the corresponding link, the disk information can be selectively scanned into the target machine, the scanning of the disk is automatically and flexibly realized, and the subsequent research and analysis of the scanned data by the personnel in the industry are facilitated.
In the method provided by the embodiment of the invention, according to the region where the target machine is located and the machine configuration information, a process of determining data definition configuration information corresponding to the target machine is determined, and the specific description is as follows:
determining a current operating period of the target machine;
and sending the current operation period, the region where the target machine is located and the machine configuration information to a cloud end of the target machine, so that the cloud end can determine data definition configuration information corresponding to the region where the target machine is located and the machine configuration information in the current operation period from a cloud end database.
Specifically, in one scenario, the target machine is currently located in an office, and a plurality of data storage centers near the office exist, and considering that the uploading destination of the disk information can affect the scanning process, therefore, aiming at the influence of factors of the target machine, according to the location of the data storage centers near the area where the target machine is located, a new data storage center needs to be selected, so that cost saving is realized; in addition, in the process of determining the data definition configuration information, the configuration of the target machine is changed due to the change of the operation period, so that the current operation period of the target machine is required to be determined, the current operation period and the region where the target machine is and the machine configuration information acquired before are sent to the cloud, and after the cloud receives the information, the data definition configuration information corresponding to the information is returned and stored in the data storage center for the subsequent analysis process.
The above embodiment describes the process of determining the data definition configuration information corresponding to the target machine according to the location of the target machine and the machine configuration information, and after parsing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition, the present application further includes the following steps:
acquiring the current environment of the target machine;
if the current environment of the target machine does not belong to the preset full bandwidth environment, performing secondary analysis on the data definition configuration information to generate bandwidth limitation information;
and limiting the target machine by the bandwidth limiting information.
In particular, a full bandwidth environment refers to a location where the target machine can use the full bandwidth. Different enterprises or units have different placement positions for the target machine, some place the target machine in an office, some place the target machine in a machine room, and different placement positions can influence the bandwidth use condition of the target machine, for example, the bandwidth of the machine room is limited little and the speed is high, so that the target machine is placed in the machine room, and the full use of the bandwidth can be realized. Therefore, in the application, if the current environment of the target machine does not belong to the preset full bandwidth environment, the data definition configuration is analyzed to generate the data definition and the corresponding link, and secondary analysis is needed to obtain the bandwidth limitation information, and the target machine is limited according to the bandwidth limitation information, so that the aim of reducing the disk information scanning is fulfilled, and the availability of the disk information is ensured.
The following describes a process of preprocessing all disk information based on each data definition to obtain target data of each class in the present application. The following list two ways:
marking all pieces of information in all the disc information by utilizing preset data types to obtain marked information;
from the marked information, taking the marked information corresponding to the preset type to be scanned as the information to be scanned;
comparing each piece of information to be scanned with each data definition to determine the data definition corresponding to the piece of information to be scanned;
and combining and dividing each piece of information to be scanned to obtain target data of each category.
In the above process, for the autopilot field, various information generated by the autopilot vehicle will be stored in the vehicle disk indiscriminately, but in particular, the importance degree of each information is different, if all the disk data are scanned to the target machine, the target machine will be jammed and wasted, and the efficiency is low, and more important information cannot be effectively screened out for use. Therefore, the type to be scanned is set, and the type to be scanned is the disc information which really needs to be scanned into the target machine. In addition, a plurality of data types are preset, each data type is used as a label, each piece of information in all disc information is marked to obtain marked information, the marked information is in one-to-one correspondence according to the preset type to be scanned, so that the information to be scanned is screened out from the marked information, and then the combination and division of the information to be scanned can be realized according to the data definition, so that target data of each category is obtained.
In all the magnetic disk information, each piece of data defined by each piece of data is taken as each piece of first data;
for each piece of first data, acquiring a scanning path, a category, a data size and creation time of the piece of first data;
combining the scanning path, the category, the data size and the creation time of the first data to obtain catalog data corresponding to the first data, and integrating the first data and the catalog data corresponding to the first data to obtain target data corresponding to the first data;
and taking the target data corresponding to other first data with the same definition as the data to which the first data belongs and the target data corresponding to the first data as the target data of the same category, thereby obtaining the target data of each category.
Specifically, each data definition is compared with each piece of data in all disk information, and each piece of data belonging to each data definition is taken as first data. It can be understood that each piece of first data has a corresponding scan path, a corresponding category, a corresponding data size and a corresponding creation time, and each piece of target data can be obtained after integrating the information with the own data. Then, the target data of each category is obtained by dividing, classifying and combining.
The above embodiment describes a process of preprocessing all disk information based on each data definition to obtain target data of each class, and the following embodiment describes a process of linearly processing the target data of each class according to a link corresponding to each data definition to transfer the target data of each class into the target machine, thereby completing a process of scanning a disk, specifically as follows:
aiming at each data definition corresponding link, acquiring the data scanning priority corresponding to the link;
sorting the target data of each category according to the data scanning priority corresponding to each link;
and carrying out linear processing on the target data of each category in turn according to the sequencing order so as to transmit the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
Specifically, in one example, the bandwidth of an office in a certain area is only 100MB, the target machine is located in the office, the storage space of the target machine is only 500GB, even if each kind of target data in the disk is continuously scanned and uploaded in one day, the memory space of the target machine is not left for two or three days, so that the target data is ordered by setting the data scanning priority in the link, more important data is arranged in front, or the data occupying smaller bandwidth is arranged in front, so that the data can be ensured to be scanned into the target machine preferentially, the high utilization rate and scanning efficiency of the data are ensured, and other unimportant data can be deleted directly.
It will be appreciated that after the scanning process has been screened, the method may further comprise: judging whether the current memory space of the target machine remains; if yes, calculating the residual memory space of the target machine, and adjusting the link corresponding to each data definition according to the residual space; if not, the bandwidth of the place where the target machine is located is up-regulated. Along with the above example, the office bandwidth application is up-regulated to 1000M, and then all disk information can be completely scanned and uploaded.
Optionally, the embodiment of the application further provides a disk scanning device, which comprises a configuration analysis module, a data identification module and a data link processing module, wherein:
the structure diagram of the configuration analysis module is shown in fig. 2, wherein a SyncConfig function is arranged in the configuration analysis module and is used for responding to a disk scanning instruction, acquiring the region where a target machine corresponding to a disk to be scanned is currently located and machine configuration information, determining data definition configuration information corresponding to the target machine according to the region where the target machine is located and the machine configuration information, and storing the data definition configuration information into a device memory; the ConfigParser function is also arranged for analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition, and possibly some extra bandwidth limitation information and the like; in addition, a StoreConfig function is also arranged and used for storing various information analyzed by the ConfigParser function in the memory of the device for subsequent access and calling by other modules.
The structure diagram of the data identification module is shown in fig. 3, in which a GetDataDefinition function is set for obtaining various parsed information from the StoreConfig function; the system is also provided with a DistinguishData function, which is used for preprocessing all disk information according to each data definition in the information acquired by the GetDataDefinition function to obtain target data of each category, or is responsible for traversing each piece of first data stored in a specific data catalog, integrating the scanning path, the category, the data size and the creation time of the first data, and regenerating the target data of each category into the memory of the device; a store datainfo function is also provided for storing the target data for each category in a local database.
The data link processing module is used for carrying out linear processing on the target data of each category according to the links corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk; and the method can also be used for monitoring the abnormal state of data uploading. The module is mainly responsible for notifying other corresponding modules of the links to execute.
The data link processing module, the configuration analysis module and the data identification module together form a complete set of disk scanning device (disk scanner) which can be configured by user definition and has expansibility, and has the functions of monitoring the abnormal state of data uploading and monitoring the abnormal data uploading condition.
Corresponding to the method shown in fig. 1, the embodiment of the present invention further provides another disk scanning device, which is used for implementing the method shown in fig. 1, where the disk scanning device provided in the embodiment of the present invention may be described in conjunction with fig. 4 in a computer terminal or various mobile devices, and as shown in fig. 4, the device may include:
the data definition configuration information determining module 10 is configured to obtain, in response to a disk scanning instruction, a region where a target machine currently corresponding to a disk to be scanned is located and machine configuration information, and determine data definition configuration information corresponding to the target machine according to the region where the target machine is located and the machine configuration information;
the parsing module 20 is configured to parse the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
The preprocessing module 30 is configured to obtain all disc information in the disc, and perform preprocessing on all disc information based on each data definition, so as to obtain target data of each category;
and the linear processing module 40 is configured to perform linear processing on the target data of each class according to the link corresponding to each data definition, so as to transfer the target data of each class into the target machine, thereby completing the scanning of the disk.
According to the technical scheme, the method and the device acquire the region and the machine configuration information of the target machine corresponding to the magnetic disk to be scanned currently by responding to the magnetic disk scanning instruction, and determine the data definition configuration information corresponding to the target machine according to the region and the machine configuration information of the target machine; analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition; acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category; and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk. According to the scheme, the data definition configuration information is acquired by utilizing the region and self configuration of the target machine with the predetermined corresponding relation, and the data definition and the corresponding link can be obtained after analysis, so that the preprocessing mode and the linear processing mode for the disk information are determined according to the data definition and the corresponding link, the disk information can be selectively scanned into the target machine, the scanning of the disk is automatically and flexibly realized, and the subsequent research and analysis of the scanned data by the personnel in the industry are facilitated.
In one example, the data definition configuration information determination module 10 may include:
a module for determining a current operating cycle of the target machine;
the module is used for sending the current operation period, the region where the target machine is located and the machine configuration information to the cloud end of the target machine, so that the cloud end can determine data definition configuration information corresponding to the region where the target machine is located and the machine configuration information in the current operation period from a cloud end database.
In one example, the apparatus may include:
the environment acquisition module is used for acquiring the current environment of the target machine;
the secondary analysis module is used for carrying out secondary analysis on the data definition configuration information to generate bandwidth limitation information if the current environment of the target machine does not belong to a preset full bandwidth environment;
and the limiting module is used for limiting the target machine by the bandwidth limiting information.
In one example, the preprocessing module 30 may include:
the marking module is used for marking each piece of information in all the disc information by utilizing each preset data type to obtain each piece of marked information;
The screening module is used for taking all pieces of marked information corresponding to preset all types to be scanned as all pieces of information to be scanned from all pieces of marked information;
the comparison module is used for comparing each piece of information to be scanned with each data definition so as to determine the data definition corresponding to the piece of information to be scanned;
and the combination dividing module is used for combining and dividing each piece of information to be scanned so as to obtain target data of each category.
In one example, the preprocessing module 30 may further include:
the comparison module is used for taking each piece of data defined by each piece of data as each piece of first data in all disc information;
the basic information acquisition module is used for acquiring a scanning path, a category, a data size and creation time of each piece of first data aiming at each piece of first data;
the integration module is used for combining the scanning path, the category, the data size and the creation time of the first data to obtain catalog data corresponding to the first data, and integrating the first data with the catalog data corresponding to the first data to obtain target data corresponding to the first data;
The classification module is used for taking the target data corresponding to other first data with the same definition as the data to which the first data belongs and the target data corresponding to the first data as the target data of the same category, thereby obtaining the target data of each category.
In one example, the linear processing module 40 may include:
the priority acquisition module is used for acquiring the data scanning priority corresponding to each link according to the link corresponding to each data definition;
the sorting module is used for sorting the target data of each category according to the data scanning priority corresponding to each link;
and the input module is used for sequentially carrying out linear processing on the target data of each category according to the sequencing order so as to transmit the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
In one example, the apparatus may further include:
the judging module is used for judging whether the current memory space of the target machine remains;
the link adjustment module is used for calculating the residual memory space of the target machine if yes, and adjusting the link corresponding to each data definition according to the residual space;
And the bandwidth up-regulating module is used for up-regulating the bandwidth of the place where the target machine is located if not.
Still further, an embodiment of the present application provides a disk scanning apparatus. Alternatively, fig. 5 shows a block diagram of a hardware structure of the disk scanning device, and referring to fig. 5, the hardware structure of the disk scanning device may include: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.
In the embodiment of the present application, the number of the processor 01, the communication interface 02, the memory 03 and the communication bus 04 is at least one, and the processor 01, the communication interface 02 and the memory 03 complete communication with each other through the communication bus 04.
The processor 01 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present application, or the like.
The memory 03 may include a high-speed RAM memory, and may further include a nonvolatile memory (non-volatile memory) or the like, such as at least one magnetic disk memory.
The memory stores a program, and the processor can call the program stored in the memory, and the program is used for executing the following disk scanning method, which comprises the following steps:
Responding to a disk scanning instruction, acquiring the region and machine configuration information of a target machine corresponding to a disk to be scanned, and determining data definition configuration information corresponding to the target machine according to the region and machine configuration information of the target machine;
analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category;
and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk. Alternatively, the refinement function and the extension function of the program may refer to the description of the disk scanning method in the method embodiment.
The embodiment of the application also provides a storage medium, which can store a program suitable for being executed by a processor, and when the program runs, the device where the storage medium is controlled to execute the following disk scanning method, comprising the following steps:
Responding to a disk scanning instruction, acquiring the region and machine configuration information of a target machine corresponding to a disk to be scanned, and determining data definition configuration information corresponding to the target machine according to the region and machine configuration information of the target machine;
analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category;
and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
In particular, the storage medium may be a computer-readable storage medium, which may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM.
Alternatively, the refinement function and the extension function of the program may refer to the description of the disk scanning method in the method embodiment.
In addition, functional modules in various embodiments of the present disclosure may be integrated together to form a single portion, or each module may exist alone, or two or more modules may be integrated to form a single portion. The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored on a computer readable storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a live device, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present disclosure.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of scanning a disk, comprising:
responding to a disk scanning instruction, acquiring the region and machine configuration information of a target machine corresponding to a disk to be scanned, and determining data definition configuration information corresponding to the target machine according to the region and machine configuration information of the target machine;
analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
Acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category;
and carrying out linear processing on the target data of each category according to the link corresponding to each data definition so as to store the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
2. The method of claim 1, wherein determining data defining configuration information corresponding to the target machine based on the location of the target machine and machine configuration information comprises:
determining a current operating period of the target machine;
and sending the current operation period, the region where the target machine is located and the machine configuration information to a cloud end of the target machine, so that the cloud end can determine data definition configuration information corresponding to the region where the target machine is located and the machine configuration information in the current operation period from a cloud end database.
3. The method of claim 1, further comprising, after said parsing the data definition configuration information to generate a plurality of data definitions and links to which each of the data definitions corresponds:
Acquiring the current environment of the target machine;
if the current environment of the target machine does not belong to the preset full bandwidth environment, performing secondary analysis on the data definition configuration information to generate bandwidth limitation information;
and limiting the target machine by the bandwidth limiting information.
4. The method of claim 1, wherein preprocessing all disk information based on each of the data definitions to obtain each class of target data comprises:
marking each piece of information in all the disc information by utilizing each preset data type to obtain each piece of marked information;
from the marked information, taking the marked information corresponding to the preset type to be scanned as the information to be scanned;
comparing each piece of information to be scanned with each data definition to determine the data definition corresponding to the piece of information to be scanned;
and combining and dividing each piece of information to be scanned to obtain target data of each category.
5. The method of claim 1, wherein preprocessing all disk information based on each of the data definitions to obtain each class of target data comprises:
In all the disk information, each piece of data defined by each piece of data is taken as each piece of first data;
for each piece of first data, acquiring a scanning path, a category, a data size and creation time of the piece of first data;
combining the scanning path, the category, the data size and the creation time of the first data to obtain catalog data corresponding to the first data, and integrating the first data and the catalog data corresponding to the first data to obtain target data corresponding to the first data;
and taking the target data corresponding to other first data with the same definition as the data to which the first data belongs and the target data corresponding to the first data as the target data of the same category, thereby obtaining the target data of each category.
6. The method according to claim 1, wherein the performing linear processing on the target data of each category according to the link corresponding to each data definition to transfer the target data of each category into the target machine, thereby completing scanning of the disk, includes:
aiming at each data definition corresponding link, acquiring the data scanning priority corresponding to the link;
Sorting the target data of each category according to the data scanning priority corresponding to each link;
and carrying out linear processing on the target data of each category in turn according to the sequencing order so as to transmit the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
7. The method of claim 1, further comprising, after the linearly processing the target data of each class according to the link corresponding to each of the data definitions to transfer the target data of each class into the target machine, completing the scanning of the disk:
judging whether the current memory space of the target machine remains;
if yes, calculating the residual memory space of the target machine, and adjusting the link corresponding to each data definition according to the residual space;
if not, the bandwidth of the place where the target machine is located is up-regulated.
8. A disk scanning device, comprising:
the data definition configuration information determining module is used for responding to a disk scanning instruction, acquiring the region where a target machine corresponding to a disk to be scanned is located and machine configuration information, and determining the data definition configuration information corresponding to the target machine according to the region where the target machine is located and the machine configuration information;
The analysis module is used for analyzing the data definition configuration information to generate a plurality of data definitions and links corresponding to each data definition;
the preprocessing module is used for acquiring all disc information in the disc, and preprocessing all disc information based on each data definition to obtain target data of each category;
and the linear processing module is used for carrying out linear processing on the target data of each category according to the links corresponding to each data definition so as to transmit the target data of each category into the target machine, thereby completing the scanning of the magnetic disk.
9. A disk scanning device comprising a memory and a processor;
the memory is used for storing programs;
the processor is configured to execute the program to implement the steps of the disk scanning method according to any one of claims 1 to 7.
10. A storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the disk scanning method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310607304.3A CN116594566A (en) | 2023-05-25 | 2023-05-25 | Disk scanning method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310607304.3A CN116594566A (en) | 2023-05-25 | 2023-05-25 | Disk scanning method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116594566A true CN116594566A (en) | 2023-08-15 |
Family
ID=87593569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310607304.3A Pending CN116594566A (en) | 2023-05-25 | 2023-05-25 | Disk scanning method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116594566A (en) |
-
2023
- 2023-05-25 CN CN202310607304.3A patent/CN116594566A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11657309B2 (en) | Behavior analysis and visualization for a computer infrastructure | |
| CN108491305A (en) | A kind of detection method and system of server failure | |
| CN108259268B (en) | Network monitoring data processing method and device, computer equipment and storage medium | |
| CN110650052B (en) | Customer reason fault identification processing method and system based on intelligent algorithm | |
| CN107957940B (en) | Test log processing method, system and terminal | |
| CN111274095A (en) | Log data processing method, device, equipment and computer readable storage medium | |
| US11461037B2 (en) | Data collection system and data collection method | |
| US11115288B2 (en) | Parameter setting method, data analysis device and data analysis system | |
| CN112612680A (en) | Message warning method, system, computer equipment and storage medium | |
| CN111736571A (en) | Fault diagnosis system and method, and storage medium | |
| US20120072589A1 (en) | Information Processing Apparatus and Method of Operating the Same | |
| CN116319777A (en) | Intelligent gateway service processing method based on edge calculation | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| CN104461847B (en) | Data processor detection method and device | |
| CN108476151A (en) | For capturing and being shown in local control network network（LCN）In grouping and other message system and method | |
| CN116594566A (en) | Disk scanning method, device, equipment and storage medium | |
| CN116071902B (en) | Method, equipment and medium for monitoring power equipment of machine room | |
| CN112527610A (en) | Method and device for monitoring operation state of equipment | |
| CN116708488A (en) | Water conservancy Internet of things equipment state synchronization method, device, equipment and storage medium | |
| CN115883330A (en) | Alarm event processing method, system, device, storage medium and program product | |
| CN113472881B (en) | Statistical method and device for online terminal equipment | |
| CN115499514A (en) | Data storage service access method, computing device and computer storage medium | |
| CN113568811A (en) | Distributed safety monitoring data processing method | |
| CN113381907A (en) | Log collection method and device, electronic equipment and storage medium | |
| CN105897503A (en) | Hadoop cluster bottleneck detection algorithm based on resource information gain |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |