CN112905127A - Data processing method and data processing system - Google Patents
Data processing method and data processing system Download PDFInfo
- Publication number
- CN112905127A CN112905127A CN202110320495.6A CN202110320495A CN112905127A CN 112905127 A CN112905127 A CN 112905127A CN 202110320495 A CN202110320495 A CN 202110320495A CN 112905127 A CN112905127 A CN 112905127A
- Authority
- CN
- China
- Prior art keywords
- data
- compression
- historical data
- data processing
- component
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
Abstract
The present disclosure provides a data processing method and a data processing system. The data processing method is applied to a data processing system, and the data processing system comprises a data compression component and a service coordination component; the data processing method comprises the following steps: compressing, by the data compression component, historical data that satisfies data compression conditions, and prohibiting, by the service coordination component, writing and/or reading of the historical data that is being compressed; in response to completion of the compression of the historical data, allowing, by the service orchestration component, writing and/or reading of the compressed historical data.
Description
Technical Field
The disclosure relates to the field of data processing and the field of new energy power generation, in particular to a data processing method and a data processing system.
Background
In recent years, the new energy industry is developed vigorously, and wind power generation and photovoltaic power generation are very important clean energy (also called new energy) power generation technologies in recent years. Safe and stable operation of the new energy power generation equipment is particularly important for ensuring stable power grid and new energy power generation capacity. With the rapid layout of new energy power generation in recent years, the operating data of power generation equipment is more and more, and the purposes are more and more. This places very high demands on providing real-time data storage queries.
Many data processing methods exist, but it is difficult to support mass data processing in the new energy industry well, and rapid data retrieval cannot be provided while data storage or compression is performed. Moreover, the data processing method often ignores the use characteristics and application scenes of the new energy equipment data.
Disclosure of Invention
Data processing of new energy power generation equipment requires high compression rate, reduces occupied storage space, and provides efficient data retrieval capability and data writing capability. The method is a great problem in the field of new energy big data processing. According to the data use characteristics of the new energy power generation equipment, the data can be compressed, and meanwhile, the usability (for example, the data read-write performance) of the data can be ensured.
According to an embodiment of the present disclosure, there is provided a data processing method applied to a data processing system including a data compression component and a service coordination component; the data processing method comprises the following steps: compressing, by the data compression component, historical data that satisfies data compression conditions, and prohibiting, by the service coordination component, writing and/or reading of the historical data that is being compressed; in response to completion of the compression of the historical data, allowing, by the service orchestration component, writing and/or reading of the compressed historical data.
According to an embodiment of the present disclosure, there is provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the data processing method as described above.
According to an embodiment of the present disclosure, there is provided a computing device including: a processor; a memory storing a computer program which, when executed by the processor, implements the data processing method as described above.
According to an embodiment of the present disclosure, there is provided a data processing system including: a data compression component configured to compress historical data satisfying a data compression condition; a service coordination component configured to inhibit writing and/or reading of historical data being compressed; in response to completion of compression of the historical data, writing and/or reading of the compressed historical data is allowed.
By adopting the data processing method and the data processing system according to the embodiment of the disclosure, at least the following technical effects can be realized: according to the data use characteristics of the new energy equipment (for example, due to the fact that long-time network disconnection is needed for power grid safety inspection, field maintenance and the like, data delay and supplement are caused), the data are selectively compressed, and meanwhile, the usability of the data is guaranteed, for example, massive historical data can be reasonably compressed to reduce occupied storage space, and meanwhile, the reading and writing usability of the historical data in non-compression is guaranteed; it may also allow writing and reading of compressed data, e.g. the compressed data may be subject to being.
Drawings
The above and other objects and features of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings.
FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present disclosure.
Fig. 2 is a flow chart of a data processing method according to an embodiment of the present disclosure.
Fig. 3 is a flow diagram of a data compression process according to an embodiment of the present disclosure.
Fig. 4 is a flow chart of a data processing method according to another embodiment of the present disclosure.
Fig. 5 is a flowchart of a data processing method according to another embodiment of the present disclosure.
Fig. 6 is a flow chart of a data write process according to an embodiment of the present disclosure.
Fig. 7 is a flow chart of a data read process according to an embodiment of the present disclosure.
Fig. 8 is a schematic diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
For processing mass data (for example, mass data of a new energy device), the conventional data processing technology has low efficiency in retrieving compressed historical data, and it is difficult to realize data compression while allowing efficient reading and writing of data that is not in compression. In addition, the conventional data processing technology often neglects the condition that the historical data needs to be added, and the retrieval efficiency of the compressed historical data is low.
The invention provides a data processing method and a data processing system, which can compress data and ensure the usability of the data according to the data use characteristics of new energy equipment (for example, the data delay and supplement and the like are caused by long-time network disconnection of power grid safety inspection, field maintenance and the like), for example, mass historical data can be reasonably compressed to reduce the occupied storage space, and the reading and writing usability of the uncompressed historical data is also ensured. In addition, the data processing method and the data processing system according to the embodiments of the present disclosure may also allow writing and reading of compressed data, for example, the compressed data may be subjected to padding and retrieval.
FIG. 1 is a schematic diagram of a data processing system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the data processing system 1 may include a data compression component 11 and a service coordination component 12. According to an embodiment of the present disclosure, the service coordination component 12 may be a component that performs coordination control on distributed services in a distributed storage system in communication with the new energy power generation device. Optionally, the data processing system 1 may further comprise a data writing component 13. Optionally, the data processing system 1 may also include a data reading component 14. The data processing system 1 and its various components described above may be implemented by software, hardware or firmware configured to perform the respective functions, or any combination thereof. The above components may be separate from or integrated with each other. The above examples are for illustration only, but the present disclosure is not limited thereto.
According to an embodiment of the present disclosure, the data processing system 1 may be configured to process data of the new energy power generation device. For example, but not limited to, the data processing system 1 may be installed in a central controller (e.g. a remote controller of a wind farm, a remote controller of a photovoltaic farm) of a new energy power plant (e.g. a wind park, a photovoltaic park). The data processing system 1 and the above components thereof can communicate with each new energy power generation device through wired connection or wireless connection to process data of the new energy power generation device. The above examples are for illustration only, but the present disclosure is not limited thereto.
According to embodiments of the present disclosure, the data compression component 11 may be configured to compress historical data that satisfies data compression conditions. The service orchestration component 12 may be configured to disable writing and/or reading of historical data in compression; in response to completion of compression of the historical data, writing and/or reading of the compressed historical data is allowed. The data writing component 13 may be configured to receive a write event, which may include, but is not limited to, at least one of the following information: the data writing method comprises the steps of writing a request, data to be written, a database identifier corresponding to the data to be written, writing state information and the like. The data writing component 13 may write the historical data with the data to be written included in the write event according to the indication of the service coordination component 12. The data writing component 13 may receive a write event from the data storage of the new energy generation device. Alternatively, the data writing component 13 may receive a writing event input by a worker or a user through any input device.
In accordance with embodiments of the present disclosure, the service coordination component 12 may be configured to monitor write events and/or read events to historical data being compressed. The data writing component 13 may be configured to, in response to the service coordination component 12 monitoring the writing event, cache data to be written included in the writing event; and responding to the completion of the compression of the historical data, and writing the compressed historical data by using the cached data to be written.
According to embodiments of the present disclosure, the data reading component 14 may be configured to receive a read event, which may include, but is not limited to, a read request, a database identification corresponding to the read request, read status information, and the like. The read request may include, but is not limited to, a query request, a retrieve request, and the like. The data reading component 14 may read (e.g., query, retrieve) historical data based on the read events as directed by the service orchestration component 12. The data reading component 14 may receive a read event from a controller of the new energy power plant or the like. Alternatively, the data writing component 13 may receive a reading event input by a worker or a user through any reading and writing device.
A data processing method according to an embodiment of the present disclosure will be described below with reference to fig. 2 to 7. The data processing method according to the embodiment of the present disclosure may be applied to a data processing system, for example, the data processing system 1. The operations performed by the various components in the data processing system 1 may be understood with reference to the data processing methods described below.
Fig. 2 is a flow chart of a data processing method according to an embodiment of the present disclosure. The data processing method according to an embodiment of the present disclosure may include: compressing, by the data compression component, the historical data that satisfies the data compression condition (e.g., performing, by the data compression component, steps S21 and/or S22), and inhibiting, by the service orchestration component, writing and/or reading of the historical data under compression (e.g., performing, by the service orchestration component, step S23); in response to completion of the compression of the historical data, writing and/or reading of the compressed historical data is allowed by the service orchestration component (e.g., steps S24 and S25 are performed by the service orchestration component).
As shown in fig. 2, in step S21, it may be determined whether the history data satisfies the data compression condition. If it is determined that the data compression condition is satisfied, step S22 is executed, otherwise, the determination is continued. According to the data compression condition, the historical data can be selectively compressed. According to an embodiment of the present disclosure, the data compression condition may include at least one of: a preconfigured time condition, a preconfigured read-write traffic condition, a preconfigured status condition, a preconfigured data table name condition. However, the present disclosure is not limited thereto. Data compression conditions can be set according to the data use characteristics of the new energy equipment.
According to an embodiment of the present disclosure, the preconfigured time condition may comprise a time point of the historical data being before a predetermined time period of the current time. For example, the predetermined period of time may be set according to a network outage duration of the new energy device. The new energy device may be disconnected for a long time due to security inspection or field maintenance of the power grid, for example, the network disconnection time may be as long as one to three months or longer. Whereas historical data during network outages may be frequently read and/or written after the logging-in. Thus, historical data prior to the network outage duration may be stored in compressed form (e.g., compressed using a data encoding algorithm and/or a distributed storage algorithm), while historical data subsequent to the network outage duration may be stored in plain form (i.e., uncompressed form).
According to the embodiment of the disclosure, the commonly stored historical data can be quickly read and/or written, so that efficient use of the commonly stored historical data can be ensured; and the compressed stored historical data can effectively reduce the occupied storage space and can also be written and/or read in response to the actual read-write request. Therefore, the storage space occupied by the data processing system can be effectively reduced, and meanwhile, the historical data can be reasonably used.
Optionally, the preconfigured read-write traffic condition may include the read-write traffic of the historical data being less than or equal to a predetermined traffic threshold. For example, the predetermined flow rate threshold value may be set according to a statistical value (e.g., an average value, a maximum value, a median value, a minimum value, etc.) of the read-write flow rate of the history data. In this way, the historical data with the read-write flow rate less than or equal to the predetermined flow rate threshold value can be compressed and stored (for example, compressed and stored by using a data encoding algorithm and/or a distributed storage algorithm), and the historical data with the read-write flow rate greater than the predetermined flow rate threshold value can be stored in a common way.
According to the embodiment of the disclosure, in order to avoid the frequently read and written historical data (e.g., the historical data table), it is necessary to determine whether the read and write flow of the historical data is less than or equal to the predetermined flow threshold value according to the read and write flow provided by the read and write monitor (e.g., the read and write monitor table). For example, historical data read and write traffic may be obtained by accessing an Application Program Interface (API) for read and write traffic monitoring provided by a cluster monitor (cloudera manager) of the data processing system.
According to the embodiment of the disclosure, the utilization rate of the commonly stored historical data is high (for example, the read-write flow is high), and since the commonly stored historical data is commonly stored, the commonly stored historical data can be quickly read and/or written, so that the high-efficiency use of the commonly stored historical data can be ensured; and the usage rate of the history data of the compressed storage is low (for example, the read-write flow is low), the occupied storage space can be effectively reduced because the history data is compressed and stored, and the history data can also be written and/or read in response to the actual read-write request. Therefore, the storage space occupied by the data processing system can be effectively reduced, and meanwhile, the historical data can be reasonably used.
Alternatively, the preconfigured state condition may include the historical data being in a non-read-write state. According to the pre-configured state condition, the historical data in the non-read-write state can be compressed, so that the conflict between data compression and data read-write is prevented.
Alternatively, the preconfigured data table name condition may include a data table name of the historical data being included in the data table name specified to be compressed. According to the pre-configured data table name condition, the historical data to be compressed can be accurately identified. For example, historical data to be compressed can be identified by configuring a regular expression corresponding to a data table name specified to be compressed.
In step S22, the history data satisfying the data compression condition may be compressed. According to the embodiment of the disclosure, the historical data meeting the data compression condition can be compressed by the data compression component by using a data encoding algorithm and/or a distributed storage algorithm. Further, the historical data that does not meet the data compression conditions may be stored in common (i.e., uncompressed storage) using a distributed storage algorithm.
According to embodiments of the present disclosure, the data encoding algorithm may include one or more of the following algorithms: erasure Code (EC) algorithm, huffman coding algorithm, run length coding algorithm. Optionally, the distributed storage algorithm may comprise one or more of the following algorithms: snapshot storage algorithm, multi-copy storage algorithm. The multi-copy storage algorithm may include a double-copy storage algorithm, a triple-copy storage algorithm. For example, the erasure coding algorithm may be a Hadoop Distributed File System (HDFS) based erasure coding algorithm. The erasure Code algorithm may include an array erasure Code algorithm, a Reed-Solomon (RS) erasure Code algorithm, and a Low Density Parity Check (LDPC) erasure Code algorithm.
In one embodiment of the present disclosure, historical data satisfying the data compression condition may be compression-stored using an erasure code algorithm and/or a snapshot storage algorithm, and historical data not satisfying the data compression condition may be general-stored using a multi-copy storage algorithm (e.g., a double-copy storage algorithm and/or a triple-copy storage algorithm). In another embodiment of the present disclosure, the history data satisfying the data compression condition may be compressed and stored using a two-copy storage algorithm, and the history data not satisfying the data compression condition may be commonly stored using a three-copy storage algorithm.
Therefore, the occupied storage space can be effectively reduced through compressed storage, and meanwhile, high-efficiency historical data use efficiency can be provided through common storage. The service coordination component can be used for carrying out coordination control on a plurality of services (such as data writing service provided by the data writing component and data reading service provided by the data reading component), so that the historical data meeting the data compression condition can be compressed under the condition that the reading and writing of the historical data which is not in compression are not influenced.
While step S22 is being performed, step S23 may be performed to inhibit writing and/or reading of the history data being compressed. At step S24, it may be determined whether compression of the history data is complete. For example, it may be periodically determined by the service coordination component whether compression of the historical data is complete. If the compression of the history data is completed, executing step S25, allowing writing and/or reading of the compressed history data; otherwise, whether the compression of the historical data is finished is continuously judged.
The data compression process in the data processing method will be described below with reference to fig. 3. Fig. 3 is a flow diagram of a data compression process according to an embodiment of the present disclosure. Here, the example of compressing the history data satisfying the data compression condition by using the erasure code algorithm is described, but the present disclosure is not limited thereto, and the history data satisfying the data compression condition may be compressed by another algorithm capable of reducing the occupied storage space.
As shown in FIG. 3, historical data that satisfies data compression conditions may be compressed by the data compression component using an erasure coding algorithm.
At step S31, a directory based on an erasure coding algorithm may be created. For example, for historical data satisfying data compression conditions, a corresponding directory can be established by using an erasure code strategy for data archiving. According to an embodiment of the present disclosure, an erasure code algorithm-based directory may be established on an HDFS in a controller of a new energy power plant, and the directory may be configured to store data using an erasure code policy.
In step S32, the history data satisfying the data compression condition may be migrated to a directory based on an erasure code algorithm to compress the history data. For example, historical data satisfying the data compression condition may be copied to a corresponding directory, so that the storage format of the historical data is converted into a storage format based on an erasure code policy, thereby implementing the compression of the historical data. According to the embodiment of the present disclosure, if the history data satisfying the data compression condition is initially stored in a format based on the three-copy storage policy, after the history data is migrated into the directory based on the erasure coding algorithm, the format of the history data is automatically converted into the storage format based on the erasure coding policy, so that the storage space occupied by the history data is significantly reduced, thereby achieving the compression of the history data.
In step S33, the name of the directory based on the erasure coding algorithm may be replaced with the original directory name of the history data. According to the embodiment of the present disclosure, the original directory name of the history data may be replaced with the name of the directory based on the erasure code algorithm, so that the compressed history data may be retrieved by identifying the original directory name, and the readability of the compressed history data may be ensured.
According to an embodiment of the present disclosure, the data processing method may further include: before compressing the historical data meeting the data compression condition, the historical data meeting the data compression condition can be backed up through the data compression component; and in response to the compression failure or timeout of the historical data, rolling back by using the backup historical data to restore the historical data.
Fig. 4 is a flow chart of a data processing method according to another embodiment of the present disclosure.
As shown in fig. 4, in step S41, the historical data satisfying the data compression condition may be backed up. Step S42 may be performed after the backup, compressing the historical data. The backup of the historical data can prevent the files of the historical data from being damaged and unable to be recovered when errors occur in the data compression process.
At step S43, it may be determined whether compression of the historical data failed or timed out. If it is determined to fail or time out, performing step S44, rolling back using the backed-up history data to restore the history data; otherwise, the step S42 is continuously executed or the step S43 is circularly executed. By using the backup historical data for rollback, the historical data which fails to be compressed or is compressed overtime can be restored to the original state, the storage error caused by the compression failure or the compression overtime is avoided, and the compressed historical data can be cleared.
During data compression, there may be a possibility that a user or an external device wants to write (e.g., patch) and/or read the history data being compressed, but data compression should be performed independently of data writing/reading. Therefore, during data compression, there is a need to monitor write events and/or read events. The following is described by way of example in connection with fig. 5, but the disclosure is not limited thereto.
Fig. 5 is a flowchart of a data processing method according to another embodiment of the present disclosure. According to embodiments of the present disclosure, write events and/or read events to historical data being compressed may be monitored by a service coordination component. Further, any historical data that is not in compression may also be allowed to be read and/or written (e.g., allowed by the service coordination component).
As shown in FIG. 5, at step S51, write events and/or read events to the historical data being compressed may be monitored.
For example, but not limiting of, a compression event may be received from a data compression component by a service coordination component. The data compression component can actively send compression events to the service orchestration component in response to an indication by the service orchestration component or upon starting data compression. The compression event may include, but is not limited to, at least one of the following information: identification related to historical data being compressed, compression status information, etc. The compression status information may include, but is not limited to, at least one of the following information: a notification that historical data begins to compress or is being compressed (e.g., may be used to notify that reads and/or writes are prohibited), a notification that historical data ends to compress (e.g., may be used to notify that reads and/or writes are allowed).
The service coordination component may send all or a portion of the information in the compression event (e.g., an identification related to the historical data being compressed, a notification that the historical data started or was being compressed, a notification that the historical data ended compression, etc.) to the data write component and/or the data read component to notify the data write component whether writing is allowed and/or to notify the data read component whether reading is allowed.
Further, a write event can be received from the data write component by the service coordination component. The data write component may actively send a write event to the service coordination component in response to an indication by the service coordination component or when a data write is requested. For example, a write event may be registered in the service coordination component by the data write component. The write event may include, but is not limited to, at least one of the following information: the data writing method comprises the steps of writing a request, data to be written, a database identifier corresponding to the data to be written, writing state information and the like. The write status information may include, but is not limited to, at least one of the following information: the notification of starting writing or writing the historical data (for example, may be used to notify that the compression of the data to be written is prohibited or that the historical data in the database corresponding to the data to be written is prohibited), and the notification of ending writing the historical data (for example, may be used to notify that the compression of the data to be written is permitted and/or that the compression of the historical data in the database corresponding to the data to be written is permitted).
The service coordination component may send all or a portion of the information in the write event (e.g., a write request for historical data being compressed) to the data compression component to inform the data compression component whether to allow compression of the data to be written and/or to allow compression of historical data in a database corresponding to the data to be written.
Further, a read event can be received from the data reading component by the service orchestration component. The data reading component can actively send a read event to the service orchestration component in response to an indication by the service orchestration component or when a data read is requested. For example, a read event can be registered in the service orchestration component by the data read component. The read event may include, but is not limited to, at least one of the following information: a read request, a database identification corresponding to the read request, read status information, and the like. The read status information may include, but is not limited to, at least one of the following information: a notification to start reading or being read of history data (e.g., may be used to notify that compression of history data in a database corresponding to a read request is prohibited), and a notification to end reading of history data (e.g., may be used to notify that compression of history data in a database corresponding to a read request is permitted).
The service coordination component may send all or a portion of the information in the read event (e.g., a write request for historical data in compression) to the data compression component to inform the data compression component whether to allow compression of the historical data in the database corresponding to the read request.
At step S52, it may be determined whether a write event is monitored. For example, it may be determined whether a write event is received from the data write component by the service coordination component. In this embodiment, the write event includes a write request for the historical data being compressed and the data to be written. Therefore, if the write event is monitored, step S53 is performed, otherwise step S55 is performed.
In step S53, in response to the monitoring of the write event by the service coordination component, the data to be written included in the write event may be cached by the data writing component. Further, a temporary write inhibit notification for historical data being compressed may also be output by the service orchestration component. This may inhibit writing to the historical data being compressed. However, for any history data that is not in compression, a write operation corresponding to the write request may be allowed to be performed. Since the historical data in compression cannot be written, the data to be written can be cached for subsequent writing. The cached data to be written may be stored in a data compression component or a service coordination component.
In step S54, in response to the completion of the compression of the history data, the compressed history data may be written by the data writing component using the cached data to be written.
According to the embodiment of the disclosure, the cached data to be written can be used for writing the compressed historical data based on the data encoding algorithm and/or the distributed storage algorithm described above. For example, the cached data to be written may be compressed and stored in the corresponding database together with the corresponding compressed historical data based on an erasure code algorithm. Optionally, the compressed historical data corresponding to the data to be written may also be restored (i.e., to a storage format before compression) based on a snapshot storage algorithm; then, storing the data to be written in and the corresponding compressed historical data into a corresponding database (for example, storing by using a multi-copy storage algorithm); and then, compressing and storing the written historical data based on a snapshot storage algorithm. Compared with a snapshot storage algorithm, the erasure code algorithm is more convenient for writing the cached data to be written into the compressed historical data.
Thus, the compressed history data can be additionally recorded. For example, when the historical data cannot be uploaded to the remote control end in time due to a long-time network disconnection of the new energy power generation equipment (due to a power grid check and the like), when the network connection is recovered and the historical data during the network disconnection (for example, the network disconnection is 3 months), the historical data before the network disconnection is compressed and stored, so the historical data during the network disconnection can be acquired through the data writing component, and the historical data during the network disconnection is added to the compressed historical data. Alternatively, if historical data prior to network outage is being compressed, historical data during network outage may be cached for additional logging after compression is complete.
At step S55, it may be determined whether a read event is monitored. For example, it may be determined whether a read event is received from the data reading component by the service coordination component. In this embodiment, the read event includes a read request for the history data being compressed, and thus, if the read event is monitored, step S56 is performed, otherwise, step S51 is continuously performed.
At step S56, a read inhibit notification for historical data being compressed may be output by the service orchestration component in response to the monitoring of the read event. This may inhibit reading of the historical data being compressed. However, for any history data that is not in compression, a read operation corresponding to the read request may be allowed to be performed.
The communication and respective operations between the data writing component, the data reading component, the data compression component, and the service coordination component are described above, and the data writing process and the data reading process will be described below in conjunction with fig. 6 and 7, but the present disclosure is not limited thereto.
Fig. 6 is a flow chart of a data write process according to an embodiment of the present disclosure. Data to be written may be received externally by the data writing component and processed to generate a write event.
As shown in fig. 6, data may be received by the data writing component (step S61). For example, data is received from an external device connected to the data writing component and/or data input to the data writing component by a user.
After the data is received, the data may be cleaned (step S62). For example, data that is ambiguous, duplicative, incomplete, violating business or logic rules, etc. may be subject to corresponding cleansing operations. Optionally, the data may also be filtered (step S63). For example, the data may be filtered according to data storage rules or the like of the new energy power generation device.
Optionally, the data may also be converted (step S64). For example, the format of the data may be converted to a storable format, such as an integer type, a real number type, a Boolean type, a character type, and a pointer type, or a combination of two or more of the foregoing types.
Finally, the data may be stored (step S65). For example, the data may be stored in a memory of the data writing component or a memory connected to the data writing component. According to the embodiments of the present disclosure, data may be stored in a normal storage (i.e., uncompressed storage) manner, or data satisfying a data compression condition may be stored in a compressed storage manner.
In addition, data reading can also be performed by the data reading component. Fig. 7 is a flow chart of a data read process according to an embodiment of the present disclosure. A read request may be received externally by the data reading component and processed to generate a read event.
As shown in FIG. 7, a read request may be received by the data reading component (step S71). For example, various query requests, retrieval requests, and the like can be received by the data reading component.
After receiving the read request, the read request may be split (step S72). For example, the read request may be split according to an identification of the new energy generation device, a database identification, a read request time, a data write time corresponding to the read request, a communication protocol associated with the read request, and so on. Thus, the database corresponding to the reading request is convenient to be accurately identified.
After the read request is split, the database may be searched (step S73). For example, a database corresponding to the read request may be searched according to the split read request. Data corresponding to the read request may be retrieved by searching a database to generate a read result. Then, the read result may be output (step S74).
The data processing system and the data processing method according to the embodiment of the present disclosure are described above with reference to fig. 1 to 7.
According to an embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed, implements a data processing method according to an embodiment of the present disclosure.
In an embodiment of the disclosure, the computer readable storage medium may carry one or more programs which, when executed, implement the following steps described with reference to fig. 2-7: the data processing method is applied to a data processing system, and the data processing system comprises a data compression component and a service coordination component; the data processing method comprises the following steps: compressing, by the data compression component, historical data that satisfies data compression conditions, and prohibiting, by the service coordination component, writing and/or reading of the historical data that is being compressed; in response to completion of the compression of the historical data, allowing, by the service orchestration component, writing and/or reading of the compressed historical data.
A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing. The computer readable storage medium may be embodied in any device; it may also be present separately and not assembled into the device.
A data processing method and a data processing system according to an embodiment of the present disclosure have been described above with reference to fig. 1 to 7. Next, a computing device according to an embodiment of the present disclosure is described with reference to fig. 8.
Fig. 8 is a schematic diagram of a computing device according to an embodiment of the present disclosure.
Referring to fig. 8, the computing device 8 according to an embodiment of the present disclosure may include a memory 81 and a processor 82, a computer program 83 is stored on the memory 81, and when the computer program 83 is executed by the processor 82, a data processing method according to an embodiment of the present disclosure is implemented.
In an embodiment of the present disclosure, when the computer program 83 is executed by the processor 82, the operations of the data processing method described with reference to fig. 2 to 7 may be implemented: the data processing method is applied to a data processing system, and the data processing system comprises a data compression component and a service coordination component; the data processing method comprises the following steps: compressing, by the data compression component, historical data that satisfies data compression conditions, and prohibiting, by the service coordination component, writing and/or reading of the historical data that is being compressed; in response to completion of the compression of the historical data, allowing, by the service orchestration component, writing and/or reading of the compressed historical data.
The computing device illustrated in fig. 8 is only one example and should not impose any limitations on the functionality or scope of use of embodiments of the disclosure.
The data processing method, the data processing system and the computing device according to the embodiment of the present disclosure have been described above with reference to fig. 1 to 8. However, it should be understood that: the data processing system shown in fig. 1 and its various components may each be configured as software, hardware, firmware, or any combination thereof to perform a particular function, the computing device shown in fig. 8 is not limited to including the components shown above, some components may be added or deleted as desired, and the above components may also be combined.
By adopting the data processing method and the data processing system according to the embodiment of the disclosure, at least the following technical effects can be realized: according to the data use characteristics of the new energy equipment (for example, due to the fact that long-time network disconnection is needed for power grid safety inspection, field maintenance and the like, data delay and supplement are caused), the data are selectively compressed, and meanwhile, the usability of the data is guaranteed, for example, massive historical data can be reasonably compressed to reduce occupied storage space, and meanwhile, the reading and writing usability of the historical data in non-compression is guaranteed; it may also allow writing and reading of compressed data, e.g. the compressed data may be subject to being.
The control logic or functions performed by the various components or controllers in the control system may be represented by flowcharts or the like in one or more of the figures. These figures provide representative control strategies and/or logic that may be implemented using one or more processing strategies (e.g., event-driven, interrupt-driven, multi-tasking, multi-threading, and so forth). As such, various steps or functions illustrated may be performed in the sequence illustrated, in parallel, or in some cases omitted. Although not always explicitly illustrated, one of ordinary skill in the art will recognize that one or more of the illustrated steps or functions may be repeatedly performed depending on the particular processing strategy being used.
While the disclosure has been shown and described with reference to the preferred embodiments, it will be understood by those skilled in the art that various changes and modifications may be made to these embodiments without departing from the spirit and scope of the disclosure as defined by the claims.
Claims (24)
1. A data processing method is applied to a data processing system, and the data processing system comprises a data compression component and a service coordination component;
the data processing method comprises the following steps:
compressing, by the data compression component, historical data that satisfies data compression conditions, and prohibiting, by the service coordination component, writing and/or reading of the historical data that is being compressed;
in response to completion of the compression of the historical data, allowing, by the service orchestration component, writing and/or reading of the compressed historical data.
2. The data processing method of claim 1, wherein the data compression condition comprises at least one of: a preconfigured time condition, a preconfigured read-write traffic condition, a preconfigured status condition, a preconfigured data table name condition.
3. The data processing method according to claim 2, wherein the preconfigured time condition comprises a time point of the historical data being before a predetermined time period of the current time instant, and/or
The pre-configured read-write flow conditions comprise that the read-write flow of the historical data is less than or equal to a preset flow threshold value, and/or
The pre-configured state condition comprises that historical data is in a non-read-write state, and/or
The preconfigured data table name condition includes a data table name of the historical data being included in the data table name specified to be compressed.
4. The data processing method according to any one of claims 1 to 3, characterized in that the data processing method further comprises: monitoring, by the service coordination component, write events and/or read events to historical data being compressed.
5. The data processing method of claim 4, wherein the data processing system further comprises a data writing component, the data processing method further comprising: responding to the monitoring of the write-in event by the service coordination component, and caching data to be written in the write-in event through the data write-in component;
in response to completion of the compression of the historical data, allowing, by the service orchestration component, the compressed historical data to be written and/or read comprises:
and responding to the completion of the compression of the historical data, and writing the compressed historical data by using the cached data to be written through the data writing component.
6. The data processing method of claim 4, wherein inhibiting, by the service orchestration component, writing and/or reading of historical data in compression comprises: in response to monitoring the read event, outputting, by the service orchestration component, a read prohibited notification for the historical data in compression.
7. The data processing method of any one of claims 1 to 3, wherein compressing, by the data compression component, the historical data satisfying the data compression condition comprises: and compressing the historical data meeting the data compression condition by the data compression component by using a data coding algorithm and/or a distributed storage algorithm.
8. The data processing method of claim 7, wherein the data encoding algorithm comprises one or more of the following algorithms: erasure coding algorithm, huffman coding algorithm, run length coding algorithm, and/or
The distributed storage algorithm comprises one or more of the following algorithms: snapshot storage algorithm, multi-copy storage algorithm.
9. The data processing method of claim 8, wherein the history data satisfying the data compression condition is compressed by the data compression component using an erasure code algorithm, and
compressing, by the data compression component, the historical data that satisfies the data compression condition using an erasure coding algorithm includes:
creating a directory based on an erasure code algorithm;
migrating historical data meeting data compression conditions to a directory based on an erasure code algorithm to compress the historical data;
and replacing the name of the directory based on the erasure code algorithm with the original directory name of the historical data.
10. The data processing method according to any one of claims 1 to 3, characterized in that the data processing method further comprises: through the data compression component, historical data meeting data compression conditions are backed up; and in response to the compression failure or the timeout of the historical data, rolling back by using the backup historical data to restore the historical data.
11. The data processing method according to any one of claims 1 to 3, wherein the data processing method is used for processing data of a new energy power generation device through the data processing system.
12. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 11.
13. A computing device, the computing device comprising:
a processor;
memory storing a computer program which, when executed by a processor, implements a data processing method as claimed in any one of claims 1 to 11.
14. A data processing system, characterized in that the data processing system comprises:
a data compression component configured to compress historical data satisfying a data compression condition;
a service coordination component configured to inhibit writing and/or reading of historical data being compressed; in response to completion of compression of the historical data, writing and/or reading of the compressed historical data is allowed.
15. The data processing system of claim 14, wherein the data compression conditions comprise at least one of: a preconfigured time condition, a preconfigured read-write traffic condition, a preconfigured status condition, a preconfigured data table name condition.
16. The data processing system of claim 15, wherein the preconfigured time condition comprises a time point of the historical data being before a predetermined time period of a current time, and/or
The pre-configured read-write flow conditions comprise that the read-write flow of the historical data is less than or equal to a preset flow threshold value, and/or
The pre-configured state condition comprises that historical data is in a non-read-write state, and/or
The preconfigured data table name condition includes a data table name of the historical data being included in the data table name specified to be compressed.
17. The data processing system of any of claims 14 to 16, wherein the service orchestration component is further configured to monitor write events and/or read events to historical data being compressed.
18. The data processing system of claim 17, wherein the data processing system further comprises a data writing component configured to:
responding to the service coordination component to monitor the write-in event, and caching data to be written in the write-in event;
and in response to the completion of the compression of the historical data, writing the compressed historical data by using the cached data to be written.
19. The data processing system of claim 17, wherein the service coordination component is configured to: in response to monitoring the read event, outputting a read inhibit notification for historical data being compressed.
20. The data processing system of any of claims 14 to 16, wherein the data compression component is configured to: and compressing the historical data meeting the data compression condition by using a data encoding algorithm and/or a distributed storage algorithm.
21. The data processing system of claim 20, wherein the data encoding algorithms include one or more of the following algorithms: erasure coding algorithm, huffman coding algorithm, run length coding algorithm, and/or
The distributed storage algorithm comprises one or more of the following algorithms: snapshot storage algorithm, multi-copy storage algorithm.
22. The data processing system of claim 21, wherein the data compression component is configured to:
creating a directory based on an erasure code algorithm;
migrating historical data meeting data compression conditions to a directory based on an erasure code algorithm to compress the historical data;
and replacing the name of the directory based on the erasure code algorithm with the original directory name of the historical data.
23. The data processing system of any of claims 14 to 16, wherein the data compression component is further configured to: backing up historical data meeting data compression conditions; and in response to the compression failure or the timeout of the historical data, rolling back by using the backup historical data to restore the historical data.
24. The data processing system of any of claims 14 to 16, wherein the data processing system is configured to process data of a new energy power plant.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110320495.6A CN112905127A (en) | 2021-03-25 | 2021-03-25 | Data processing method and data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110320495.6A CN112905127A (en) | 2021-03-25 | 2021-03-25 | Data processing method and data processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112905127A true CN112905127A (en) | 2021-06-04 |
Family
ID=76106457
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110320495.6A Pending CN112905127A (en) | 2021-03-25 | 2021-03-25 | Data processing method and data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112905127A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113753977A (en) * | 2021-09-06 | 2021-12-07 | 北京思源广泰科技有限公司 | Data processing method and system |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060212481A1 (en) * | 2005-03-21 | 2006-09-21 | Stacey Christopher H | Distributed open writable snapshot copy facility using file migration policies |
| CN101344893A (en) * | 2008-07-17 | 2009-01-14 | 中兴通讯股份有限公司 | History data access method and apparatus |
| CN102521256A (en) * | 2011-11-17 | 2012-06-27 | 广东电网公司电力科学研究院 | High-reliability data protection method of real-time/historical database |
| CN103198068A (en) * | 2012-01-07 | 2013-07-10 | 湖南省电力勘测设计院 | Application method of dynamic information database in distribution automation system |
| CN103593424A (en) * | 2013-11-07 | 2014-02-19 | 浪潮电子信息产业股份有限公司 | Configurable big-data compression processing system integrating software and hardware |
| CN106557272A (en) * | 2015-09-30 | 2017-04-05 | 中国科学院软件研究所 | A kind of efficient sensor historic data archiving method |
| CN107450856A (en) * | 2017-08-10 | 2017-12-08 | 北京元心科技有限公司 | Writing method and reading method of stored data, corresponding devices and terminals |
| CN107526743A (en) * | 2016-06-21 | 2017-12-29 | 伊姆西公司 | Method and apparatus for compressed file system metadata |
| CN109302449A (en) * | 2018-08-31 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Method for writing data, method for reading data, device and server |
| CN109815026A (en) * | 2018-12-18 | 2019-05-28 | 国电南京自动化股份有限公司 | Electric power time series database based on distributed component |
| CN109947799A (en) * | 2018-07-25 | 2019-06-28 | 光大环境科技（中国）有限公司 | The method and device that back-pressure contracting is read is carried out to historical data |
| CN110704199A (en) * | 2019-09-06 | 2020-01-17 | 深圳平安通信科技有限公司 | Data compression method and device, computer equipment and storage medium |
| CN111459404A (en) * | 2020-03-03 | 2020-07-28 | 平安科技（深圳）有限公司 | Data compression method and device, computer equipment and storage medium |
| CN111984610A (en) * | 2020-09-27 | 2020-11-24 | 苏州浪潮智能科技有限公司 | Data compression method and device and computer readable storage medium |
-
2021
- 2021-03-25 CN CN202110320495.6A patent/CN112905127A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060212481A1 (en) * | 2005-03-21 | 2006-09-21 | Stacey Christopher H | Distributed open writable snapshot copy facility using file migration policies |
| CN101344893A (en) * | 2008-07-17 | 2009-01-14 | 中兴通讯股份有限公司 | History data access method and apparatus |
| CN102521256A (en) * | 2011-11-17 | 2012-06-27 | 广东电网公司电力科学研究院 | High-reliability data protection method of real-time/historical database |
| CN103198068A (en) * | 2012-01-07 | 2013-07-10 | 湖南省电力勘测设计院 | Application method of dynamic information database in distribution automation system |
| CN103593424A (en) * | 2013-11-07 | 2014-02-19 | 浪潮电子信息产业股份有限公司 | Configurable big-data compression processing system integrating software and hardware |
| CN106557272A (en) * | 2015-09-30 | 2017-04-05 | 中国科学院软件研究所 | A kind of efficient sensor historic data archiving method |
| CN107526743A (en) * | 2016-06-21 | 2017-12-29 | 伊姆西公司 | Method and apparatus for compressed file system metadata |
| CN107450856A (en) * | 2017-08-10 | 2017-12-08 | 北京元心科技有限公司 | Writing method and reading method of stored data, corresponding devices and terminals |
| CN109947799A (en) * | 2018-07-25 | 2019-06-28 | 光大环境科技（中国）有限公司 | The method and device that back-pressure contracting is read is carried out to historical data |
| CN109302449A (en) * | 2018-08-31 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Method for writing data, method for reading data, device and server |
| CN109815026A (en) * | 2018-12-18 | 2019-05-28 | 国电南京自动化股份有限公司 | Electric power time series database based on distributed component |
| CN110704199A (en) * | 2019-09-06 | 2020-01-17 | 深圳平安通信科技有限公司 | Data compression method and device, computer equipment and storage medium |
| CN111459404A (en) * | 2020-03-03 | 2020-07-28 | 平安科技（深圳）有限公司 | Data compression method and device, computer equipment and storage medium |
| CN111984610A (en) * | 2020-09-27 | 2020-11-24 | 苏州浪潮智能科技有限公司 | Data compression method and device and computer readable storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113753977A (en) * | 2021-09-06 | 2021-12-07 | 北京思源广泰科技有限公司 | Data processing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11907561B2 (en) | Data backup method and apparatus | |
| CN105378676B (en) | The simple correcting and eleting codes locally generated | |
| US11307776B2 (en) | Method for accessing distributed storage system, related apparatus, and related system | |
| CN103944981A (en) | Cloud storage system and implement method based on erasure code technological improvement | |
| CN106708653B (en) | Mixed tax big data security protection method based on erasure code and multiple copies | |
| US11151030B1 (en) | Method for prediction of the duration of garbage collection for backup storage systems | |
| CN110442535B (en) | Method and system for improving reliability of distributed solid-state disk key value cache system | |
| CN109582213B (en) | Data reconstruction method and device and data storage system | |
| CN110609813B (en) | Data storage system and method | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN110765088A (en) | Log processing method and electronic equipment | |
| CN113064760B (en) | Database synthesis backup method and device, computer equipment and storage medium | |
| CN109165112B (en) | Fault recovery method, system and related components of metadata cluster | |
| CN112486918B (en) | File processing method, device, equipment and medium | |
| CN104679623A (en) | Server hard disk maintaining method, system and server monitoring equipment | |
| CN112905127A (en) | Data processing method and data processing system | |
| CN107330061B (en) | File deletion method and device based on distributed storage | |
| JP6943008B2 (en) | Control programs, control methods, and information processing equipment | |
| Giroire et al. | Peer-to-peer storage systems: a practical guideline to be lazy | |
| US20120151005A1 (en) | Image file download method | |
| CN113535482B (en) | Cloud backup chain data backup method and device, equipment and readable medium | |
| CN112540984B (en) | Data storage method, query method, device, electronic equipment and storage medium | |
| CN112181311B (en) | Cyclic recording method, equipment and storage medium | |
| CN114138559A (en) | Rapid backup method and system based on synthesis technology | |
| CN109787225B (en) | Distribution network model derivation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |