CN109921970B - Data processing method and system - Google Patents
Data processing method and system Download PDFInfo
- Publication number
- CN109921970B CN109921970B CN201910193252.3A CN201910193252A CN109921970B CN 109921970 B CN109921970 B CN 109921970B CN 201910193252 A CN201910193252 A CN 201910193252A CN 109921970 B CN109921970 B CN 109921970B
- Authority
- CN
- China
- Prior art keywords
- message
- field
- statistical period
- preset statistical
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention provides a data processing method and a system, which are characterized in that a CAN message in cache data in a preset statistical period is obtained, a message frame included by the CAN message is determined, the change times corresponding to each field in the message frame in the preset statistical period are determined, the structure of the message frame of the CAN message is recombined according to the change times corresponding to each field from high to low, and the message frame of the CAN message after the structure is recombined is compressed and stored. By the method, the message frame of the CAN message is structurally recombined according to the variation times corresponding to each field from high to low, and compressed storage data with the shortest length is obtained by compressing the recombined message frame, so that the compression efficiency of the compressed message frame is improved.
Description
Technical Field
The present invention relates to the field of packet compression, and more particularly, to a data processing method and system.
Background
CAN (Controller Area Network) is a serial communication protocol that is ISO international standardization. In the current automobile industry, the data interaction of the state of a plurality of automobile information and control commands is carried out by CAN messages on a CAN bus.
At present, a storage method of CAN message data is mainly based on an original data storage mode and a mode of compressing and then storing coarse granularity. The storage load on the vehicle-mounted ECU in the storage mode of the original data is large, and the corresponding communication transmission efficiency is low. The coarse-grained compression method is to compress and store the data of all message frames, thereby causing low compression efficiency.
Disclosure of Invention
In view of this, the present application provides a data processing method and system, which achieve the purpose of improving the compression efficiency of the message frame.
In order to achieve the above purpose, the proposed solution is as follows:
the first aspect of the invention discloses a data processing method, which comprises the following steps:
acquiring a CAN message in cache data in a preset statistical period, and determining a message frame included in the CAN message;
determining the number of times of change corresponding to each field in the message frame in a preset statistical period, wherein each field is obtained by dividing the message frame of the CAN message according to a preset unit length;
according to the change times corresponding to each field respectively, carrying out structural recombination on the message frame of the CAN message from high to low;
and compressing and storing the message frame of the CAN message after the structure is recombined.
Preferably, the acquiring the CAN packet in the cache data in the preset statistical period includes:
collecting and caching CAN messages in a CAN network through a collection node to obtain cache data;
and acquiring the CAN message in the cache data in a preset statistical period.
Preferably, the determining the number of times of change corresponding to each field in the packet frame in the preset statistical period includes:
dividing the message frame of the CAN message by taking bytes as unit length to obtain each field;
and determining the change times corresponding to each field in a preset statistical period.
Preferably, the determining the number of times of change corresponding to each field in the packet frame in the preset statistical period includes:
dividing the message frame of the CAN message by taking the bit as the unit length to obtain each field;
and determining the change times corresponding to each field in a preset statistical period.
Preferably, the determining the number of times of change corresponding to each field in the packet frame in the preset statistical period includes:
comparing whether the change times corresponding to the fields at a first time and the change times corresponding to the fields at a second time are changed or not in a preset statistical period, wherein the first time is the previous time of the second time;
if the field is changed, acquiring the change times corresponding to the fields as first change times;
and if the field is not changed, acquiring the number of times of change respectively corresponding to each field as a second number of times of change.
Preferably, the compressing and storing the message frame of the CAN message after the structure is reassembled includes:
carrying out XOR operation compression on the message frames of the CAN message after the structure recombination to obtain compressed data DNZ;
and storing the compressed data DNZ.
A second aspect of the present invention discloses a data processing system, including:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a CAN message in cache data in a preset statistical period and determining a message frame included by the CAN message;
the determining unit is used for determining the corresponding change times of each field in the message frame in a preset statistical period, wherein each field is obtained by dividing the message frame of the CAN message according to a preset unit length;
the recombination unit is used for performing structural recombination on the message frame of the CAN message according to the change times respectively corresponding to the fields from high to low;
and the compression storage unit is used for compressing and storing the message frames of the CAN messages after the structures are recombined.
Preferably, the acquiring unit for acquiring the CAN packet in the cache data in the preset statistical period includes:
the acquisition module is used for acquiring and caching the CAN messages in the CAN network through the acquisition node to obtain cache data;
and the acquisition module is used for acquiring the CAN messages in the cache data in a preset statistical period.
Preferably, the compression storage unit for compressing and storing the message frame of the CAN message after the structure is reassembled includes:
the compression module is used for carrying out XOR operation compression on the message frames of the CAN messages after the structure recombination to obtain compressed data DNZ;
and the storage module is used for storing the compressed data DNZ.
According to the technical scheme, the CAN message in the cache data in the preset statistical period is obtained, the message frame included by the CAN message is determined, the change times corresponding to each field in the message frame in the preset statistical period are determined, the message frame of the CAN message is structurally recombined according to the change times corresponding to each field from high to low, and the message frame of the CAN message after being structurally recombined is compressed and stored. By the method, the message frame of the CAN message is structurally recombined according to the variation times corresponding to each field from high to low, and compressed storage data with the shortest length is obtained by compressing the recombined message frame, so that the compression efficiency of the compressed message frame is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic flow chart illustrating a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart illustrating a process of acquiring a CAN packet in cache data in a preset statistical period, which is disclosed in the embodiment of the present invention;
fig. 3 is a schematic flow chart illustrating a process of determining the number of times of change respectively corresponding to each field in a message frame of a CAN message within a preset statistical period, where the length of each field is in bytes;
fig. 4 is a schematic flow chart illustrating a process of determining the number of times of change respectively corresponding to each field in a message frame of a CAN message in a preset statistical period, where the length of each field is in bits;
fig. 5 is a schematic flow chart illustrating the number of times of change respectively corresponding to each field in a message frame of a CAN message in a preset statistical period, disclosed in the embodiment of the present invention;
fig. 6 is a schematic diagram of message frame reassembly and compression of a CAN message in units of bytes disclosed in the embodiment of the present invention;
fig. 7 is a schematic diagram of message frame reassembly and compression of a CAN message in units of bits disclosed in the embodiment of the present invention;
FIG. 8 is a flow chart illustrating another data processing method according to the embodiment of the present invention;
FIG. 9 is a block diagram of a data processing system according to an embodiment of the present invention;
FIG. 10 is a block diagram of an acquisition unit of a data processing system according to an embodiment of the present invention;
FIG. 11 is a block diagram of a compressed storage unit of a data processing system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
As known from the background art, in the prior art, a storage method of CAN message data is mainly based on an original data storage manner and a coarse-grained compression and storage manner. The storage load on the vehicle-mounted ECU in the storage mode of the original data is large, and the corresponding communication transmission efficiency is low. The coarse-grained compression method is to compress and store the data of all message frames, thereby causing low compression efficiency.
Therefore, the invention discloses a data processing method and a system, which CAN improve the compression efficiency of compressed message frames by obtaining the CAN messages in the cache data in a preset statistical period, determining the message frames included by the CAN messages, determining the change times respectively corresponding to each field in the message frames in the preset statistical period, performing structural reorganization on the message frames of the CAN messages according to the change times respectively corresponding to each field from high to low, and performing compression processing based on the reorganized message frames to obtain the compressed storage data with the shortest length.
As shown in fig. 1, a schematic flow chart of a data processing method disclosed in the embodiment of the present invention specifically includes the following steps:
step S101: the method comprises the steps of obtaining a CAN message in cache data in a preset statistical period, and determining a message frame included by the CAN message.
In the process of implementing step S101 specifically, a CAN message in the cache data is acquired in a preset statistical period, and the CAN message is acquired by the acquisition node from the CAN message in the CAN network.
It should be noted that the preset statistical period is a period interval time, which may be 30 seconds or 50 seconds, and the specific statistical period is determined by a technician according to an actual situation.
It should be noted that the cache data refers to original data that has not been compressed and received in one statistical period. The number of the cache data can be multiple, and the specific cache data is determined by a technician according to the actual situation.
It should be noted that a message frame of a CAN message usually contains a plurality of fixed-position signal data. The signal data is divided into periodic signals and non-periodic signals from the generation time, and the message frames are generated when the timing period arrives no matter whether the physical values of the periodic signals change or not. The physical value of the adjacent two message frames generated by the non-periodic signal does not necessarily change.
The process of acquiring the CAN packet in the cache data in the preset statistical period related to step S101, as shown in fig. 2, shows a schematic flow chart of acquiring the CAN packet in the cache data in the preset statistical period disclosed in the embodiment of the present invention, and specifically includes the following steps:
step S201: and acquiring and caching the CAN message in the CAN network through the acquisition node to obtain cache data.
In the process of executing step S201, a collection node is connected to the CAN network, and the CAN packet in the CAN network is collected and cached to obtain cache data.
It should be noted that the collection node is responsible for connecting to a CAN network and extracting real-time running message data from the CAN network.
It should be noted that the number of the collection nodes may be multiple, and specifically set by a technician according to an actual situation.
And S202, acquiring the CAN message in the cache data in a preset statistical period.
Acquiring and caching CAN messages in a CAN network through an acquisition node by executing the steps S201-S202, and acquiring the CAN messages in the cached data in a preset statistical period after the cached data is obtained.
Step S102: and determining the change times corresponding to each field in the message frame in a preset statistical period.
In the process of implementing step S102 specifically, each field in the message frame is obtained by dividing the message frame of the CAN message based on a preset unit length in a preset statistical period.
It should be noted that each field is obtained by dividing the data frame of the CAN packet based on a preset unit length. The data frame includes: frame start, arbitration field, control field, data field, CRC field, reply field, and frame end.
It should be noted that the preset unit length includes: byte unit length and bit unit length.
It should be noted that bytes are the most basic unit of computer data processing, and information is mainly interpreted in units of bytes. The Byte (Byte) is abbreviated as B, and a Byte is specified to be 8 bits, i.e., 1B ═ 8 bits. Each byte consists of 8 binary bits. One bit (bit) of binary data is abbreviated as b, and transliterated as a bit, which is the minimum unit for a computer to store data.
It should be noted that, after receiving the message frame of the CAN message at the current time, by comparing whether each field of the message frame of the CAN message at the current time and the message frame of the CAN message at the previous time based on the current time changes, if the field changes, the number of times of change F of the corresponding field is increased by 1, and if the field does not change, the number of times of change F is not changed.
The process related to the step S102 of determining the respective corresponding change times of each field in the message frame based on the unit length of the byte in the preset statistical period is as shown in fig. 3, which shows a schematic flow chart of determining the respective corresponding change times of each field in the message frame of the CAN message based on the unit length of the byte in the preset statistical period, specifically including the following steps:
step S301: and dividing the message frame of the CAN message by taking bytes as unit length to obtain each field.
Step S302: and determining the change times corresponding to each field in a preset statistical period.
By executing the steps S301 to S302, the message frame of the CAN message is divided by taking bytes as unit length to obtain each field, and the change times corresponding to each field is determined in a preset statistical period.
Based on the above process of determining the number of changes corresponding to each field when the length is in bytes, the following example is given:
when the packet frames of the CAN packet are divided by taking a byte as a unit length in a statistical period T, the field B in the packet frame at this time is (B1, B2, B3, B4, B5, B6, B7, B8), and the number of times of change F corresponding to the field B is (F1, F2, F3, F4, F5, F6, F7, F8) (B1, B2, B3, B4, B5, B6, B7, B8).
The process related to the step S102 of determining the respective corresponding change times when each field in the message frame of the CAN message takes the byte as the unit length in the preset statistical period is illustrated in fig. 4, which is a schematic flow chart of determining the respective corresponding change times when each field in the message frame of the CAN message takes the bit as the unit length in the preset statistical period, specifically including the following steps:
step S401: and dividing the message frame of the CAN message by using the bit as the unit length to obtain each field.
Step S402: and determining the change times corresponding to each field in a preset statistical period.
And dividing the message frame of the CAN message by using the bit as the unit length to obtain each field by executing the steps S401 to S402, and determining the change times corresponding to each field in a preset statistical period.
Based on the above process of determining the number of changes corresponding to each field when the unit length is bit, the following example is given:
when the packet frames of the CAN packet are divided by using a bit as a unit length in a statistical period T, at this time, each field b in the packet frame is (b1, b2, b3, b4, b5, b6), and the number of times of change f corresponding to each field b is (b1, b2, b3, b4, b5, b6) (f1, f2, f3, f4, f5, f 6).
Based on the flow diagram disclosed in fig. 3 for determining the respective corresponding change times when each field in the message frame of the CAN message takes the byte as the unit length in the preset statistical period and the flow diagram disclosed in fig. 4 for determining the respective corresponding change times when each field in the message frame of the CAN message takes the bit as the unit length in the preset statistical period, as shown in fig. 5, the flow diagram for determining the respective corresponding change times of each field in the message frame of the CAN message in the preset statistical period disclosed in the embodiment of the present invention is specifically shown, and specifically includes the following steps:
step S501: and in a preset statistical period, comparing whether the change times corresponding to the fields at a first time and the change times corresponding to the fields at a second time are changed or not, wherein the first time is the previous time of the second time.
Based on the above implementation process of comparing whether the change times corresponding to the fields at the first time and the change times corresponding to the fields at the second time are changed, the following example is illustrated here:
for example, the number of times of change F1 corresponding to each field at 16 hours 05 minutes at the first time is different from the number of times of change F2 corresponding to each field at 06 hours at the second time 16 hours, where F2 is different from the number of times of change F1.
Step S502: and if the change occurs, acquiring the change times corresponding to the fields as first change times.
It should be noted that, when the number of times of change respectively corresponding to each field at the first time and the second time is changed, the number of times of the first change is increased by 1 correspondingly.
It should be noted that the number of each field may be multiple, and the determination of the specific number of fields is set by a technician according to an actual situation.
Step S503: and if the field is not changed, acquiring the number of times of change respectively corresponding to each field as a second number of times of change.
It should be noted that, when the number of times of change respectively corresponding to each field at the first time and the second time is not changed, the second number of times of change is not changed.
By executing the steps S501 to S503, in a preset statistical period, comparing whether the change times corresponding to the fields at the first time and the second time are changed, if so, obtaining the change times corresponding to the fields as the first change times, and if not, obtaining the change times corresponding to the fields as the second change times.
Step S103: and performing structural reorganization on the message frame of the CAN message according to the change times corresponding to the fields respectively from high to low.
In the process of implementing step S103 specifically, each field obtained by dividing the message frame of the CAN message according to the preset unit length is subjected to structural reorganization on the basis of the number of changes corresponding to each field from high to low.
It should be noted that the structural reassembly of the message frame of the CAN message is performed in a manner from high to low according to the change times corresponding to the fields, and the manner from high to low may be performed in a sorting manner or a grading manner, and specifically, the determination of the manner from high to low is set by a technician according to an actual situation.
Step S104: and compressing and storing the message frame of the CAN message after the structure is recombined.
In the process of implementing step S104, the message frames of the CAN report after the structure is reorganized are subjected to exclusive or operation compression processing.
It should be noted that, in a manner of recompressing an original message frame based on a change frequency of a unit length in a message frame of a CAN message, a message frame structure of the CAN message CAN be adaptively and dynamically recomposed according to a change rule of data in the message frame of the CAN message in each statistical period, so as to achieve an effect of an optimal compression ratio.
It should be noted that, after compressing and storing the message frame of the CAN message, whether to continue the message frame compression and storage process of the CAN message of the next statistical period may be set by a technician according to an actual situation.
The process of dividing the message frame of the CAN message according to the preset unit length to obtain each field involved in the above steps S103 to S104, and performing structure reorganization and compression on the message frame of the CAN message from high to low by the change times corresponding to each field, as shown in fig. 6, a message frame reorganization and compression diagram of the CAN message in units of bytes disclosed in the embodiment of the present invention is shown.
In fig. 6, the number of changes of the signals SIG _ i and SIG _ j is high, and the number of changes of the remaining signals is low.
It should be noted that the change times corresponding to the byte unit are sorted from high to low, and the byte sequence of the message frame is reassembled according to the sorting result.
It should be noted that, the recomposing and compressing method using bytes as units has lower computational complexity and faster operation.
The process of dividing the message frame of the CAN message according to the preset unit length to obtain each field involved in the above steps S103 to S104, and performing structure reorganization and compression on the message frame of the CAN message from high to low by the change times corresponding to each field, as shown in fig. 7, a message frame reorganization and compression diagram of the CAN message in units of bits disclosed in the embodiment of the present invention is shown.
In fig. 7, the number of changes of the signals SIG _ i and SIG _ j is high, and the number of changes of the remaining signals is low.
It should be noted that the change times corresponding to the unit of bit are sorted from high to low, and according to the sorting result, the bit sequence of the message frame is reassembled.
It should be noted that, in the recomposing compression method using bits as unit length, the operation complexity is slightly higher, but the compression effect is better.
In practical application, a compression storage scheme with byte as a unit length or bit as a unit length can be flexibly selected according to the computing capability of a computing platform and the requirement on data compression rate.
The embodiment of the invention discloses a data processing method, which comprises the steps of obtaining a CAN message in cache data in a preset statistical period, determining a message frame included by the CAN message, determining the change times respectively corresponding to each field in the message frame in the preset statistical period, carrying out structural reorganization on the message frame of the CAN message according to the change times respectively corresponding to each field from high to low, and compressing and storing the message frame of the CAN message after the structural reorganization. By the method, the message frame of the CAN message is structurally recombined according to the variation times corresponding to each field from high to low, and compressed storage data with the shortest length is obtained by compressing the recombined message frame, so that the compression efficiency of the compressed message frame is improved.
Based on the flow diagram of a data processing method in fig. 1, the flow diagram of another data processing method disclosed in the embodiment of the present invention, as shown in fig. 8, specifically includes the following steps:
step S801: the method comprises the steps of obtaining a CAN message in cache data in a preset statistical period, and determining a message frame included by the CAN message.
Step S802, determining the change times corresponding to each field in the message frame in a preset statistical period.
Optionally, the manner of specifically obtaining the number of changes respectively corresponding to each field in the message frame in step S802 may also be implemented by the execution manner of steps S301 to S302 disclosed in fig. 3 or steps S401 to S402 disclosed in fig. 4.
Step S803: and performing structural reorganization on the message frame of the CAN message according to the change times corresponding to the fields respectively from high to low.
The execution process of the above steps S801 to S803 is the same as the execution process of the steps S101 to S103 shown in fig. 1, and the execution principle is also the same, which can be referred to and is not described again here.
Step S804: and carrying out XOR operation compression on the message frames of the CAN message after the structure is recombined to obtain compressed data DNZ.
In the process of executing step S804, the message frames of the CAN message are structurally recombined according to the change times from high to low, so that the signals with high change times are gathered at one end of the frame in the message frames, and when performing the xor operation, the compressed storage data DNZ with the shortest length is obtained.
Step S805: and storing the compressed data DNZ.
The invention provides a data processing method, which comprises the steps of obtaining a CAN message in cache data in a preset statistical period, determining message frames included by the CAN message, determining the change times corresponding to each field in the message frames in the preset statistical period, carrying out structural reorganization on the message frames of the CAN message according to the change times corresponding to each field from high to low, carrying out XOR operation compression on the message frames of the CAN message after the structural reorganization to obtain compressed data DNZ, and storing the compressed data DNZ. By the method, the message frame of the CAN message is structurally recombined according to the variation times corresponding to each field from high to low, and compressed storage data with the shortest length is obtained by compressing the recombined message frame, so that the compression efficiency of the compressed message frame is improved.
Based on the above data processing method, the following specific implementation processes are exemplified:
after the counting period starts, acquiring a CAN message in cache data in the counting period, determining a message frame in the CAN message, and converting a first time 17: the number of changes Fa corresponding to the 00 time division byte a ═ (a1, a2) (F1, F2) and the second time 17: and comparing the change times Fb (F3, F4) corresponding to the 01 time-sharing byte B (B1, B2), sequencing the change times Fa and Fb from high to low according to the change times Fa and Fb respectively corresponding to the byte A and the byte B, carrying out structural reorganization on the message frames of the CAN message based on the sequenced result, and carrying out XOR operation compression processing on the reorganized message frames to obtain the compressed storage data DNZ with the shortest length.
Based on the data processing method disclosed in the embodiment of the present invention, the embodiment of the present invention correspondingly discloses a data processing system, and as shown in fig. 9, the data processing system 900 specifically includes:
the acquiring unit 901 is configured to acquire a CAN message in cache data in a preset statistical period, and determine a message frame included in the CAN message.
A determining unit 902, configured to determine the number of changes respectively corresponding to each field in the packet frame in a preset statistical period, where each field is obtained by dividing the packet frame of the CAN packet according to a preset unit length.
And the restructuring unit 903 is configured to perform structural restructuring on the message frame of the CAN message according to the change times corresponding to the fields respectively from high to low.
And a compression storage unit 904, configured to compress and store the message frame of the CAN message after the structure is reassembled.
Further, the acquiring unit 901, as shown in fig. 10, includes:
the acquisition module 1001 is configured to acquire and cache a CAN message in the CAN network through an acquisition node to obtain cache data.
The obtaining module 1002 is configured to obtain a CAN message in the cache data in a preset statistical period.
Further, the compressed storage unit 904, as shown in fig. 11, includes:
a compression module 1101, configured to perform xor operation compression on the message frames of the CAN message after the structure reorganization, so as to obtain compressed data DNZ.
A storage module 1102, configured to store the compressed data DNZ.
The specific principle and the execution process of each unit and each module in the data processing system disclosed in the above embodiment of the present invention are the same as those of the data processing method disclosed in the above embodiment of the present invention, and reference may be made to corresponding parts in the data processing method disclosed in the above embodiment of the present invention, and details are not described here.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (7)
1. A data processing method, comprising:
acquiring a CAN message in cache data in a preset statistical period, and determining a message frame included in the CAN message;
determining the number of times of change corresponding to each field in the message frame in a preset statistical period, wherein each field is obtained by dividing the message frame of the CAN message according to a preset unit length;
according to the change times corresponding to each field respectively, carrying out structural recombination on the message frame of the CAN message from high to low;
carrying out XOR operation compression on the message frames of the CAN message after the structure is recombined to obtain compressed data DNZ;
and storing the compressed data DNZ.
2. The method according to claim 1, wherein the obtaining the CAN packet in the cache data in the preset statistical period includes:
collecting and caching CAN messages in a CAN network through a collection node to obtain cache data;
and acquiring the CAN message in the cache data in a preset statistical period.
3. The method according to claim 1, wherein the determining the number of changes respectively corresponding to each field in the packet frame in a preset statistical period comprises:
dividing the message frame of the CAN message by taking bytes as unit length to obtain each field;
and determining the change times corresponding to each field in a preset statistical period.
4. The method according to claim 1, wherein the determining the number of changes respectively corresponding to each field in the packet frame in a preset statistical period comprises:
dividing the message frame of the CAN message by taking the bit as the unit length to obtain each field;
and determining the change times corresponding to each field in a preset statistical period.
5. The method according to claim 3 or 4, wherein the determining the number of changes respectively corresponding to each field in the packet frame in a preset statistical period comprises:
comparing whether the change times corresponding to the fields at a first time and the change times corresponding to the fields at a second time are changed or not in a preset statistical period, wherein the first time is the previous time of the second time;
if the field is changed, acquiring the change times corresponding to the fields as first change times;
and if the field is not changed, acquiring the number of times of change respectively corresponding to each field as a second number of times of change.
6. A data processing system, comprising:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a CAN message in cache data in a preset statistical period and determining a message frame included by the CAN message;
the determining unit is used for determining the corresponding change times of each field in the message frame in a preset statistical period, wherein each field is obtained by dividing the message frame of the CAN message according to a preset unit length;
the recombination unit is used for performing structural recombination on the message frame of the CAN message according to the change times respectively corresponding to the fields from high to low;
the compression storage unit is used for compressing and storing the message frames of the CAN messages after the structures are recombined;
wherein the compressed storage unit comprises:
the compression module is used for carrying out XOR operation compression on the message frames of the CAN messages after the structure recombination to obtain compressed data DNZ;
and the storage module is used for storing the compressed data DNZ.
7. The system according to claim 6, wherein the obtaining unit for obtaining the CAN packet in the buffered data in the preset statistical period comprises:
the acquisition module is used for acquiring and caching the CAN messages in the CAN network through the acquisition node to obtain cache data;
and the acquisition module is used for acquiring the CAN messages in the cache data in a preset statistical period.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910193252.3A CN109921970B (en) | 2019-03-14 | 2019-03-14 | Data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910193252.3A CN109921970B (en) | 2019-03-14 | 2019-03-14 | Data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109921970A CN109921970A (en) | 2019-06-21 |
| CN109921970B true CN109921970B (en) | 2021-06-29 |
Family
ID=66964792
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910193252.3A Active CN109921970B (en) | 2019-03-14 | 2019-03-14 | Data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109921970B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113271678B (en) * | 2021-05-17 | 2022-12-16 | 中国联合网络通信集团有限公司 | Frame structure adjusting method and device |
| CN114338847A (en) * | 2021-11-26 | 2022-04-12 | 国网北京市电力公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN114448574A (en) * | 2022-03-03 | 2022-05-06 | 湖南军安信达科技有限公司 | MAC frame data transmission processing equipment and data transmission processing method thereof |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198132A (en) * | 2013-04-12 | 2013-07-10 | 中国农业银行股份有限公司 | Method and device for data processing |
| CN107968782A (en) * | 2017-11-29 | 2018-04-27 | 北京经纬恒润科技有限公司 | A kind of message compression storage method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7996185B2 (en) * | 2008-04-29 | 2011-08-09 | Caterpillar Inc. | Machine data acquisition system with data compression |
-
2019
- 2019-03-14 CN CN201910193252.3A patent/CN109921970B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198132A (en) * | 2013-04-12 | 2013-07-10 | 中国农业银行股份有限公司 | Method and device for data processing |
| CN107968782A (en) * | 2017-11-29 | 2018-04-27 | 北京经纬恒润科技有限公司 | A kind of message compression storage method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109921970A (en) | 2019-06-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109921970B (en) | Data processing method and system | |
| KR101745456B1 (en) | Ecu for transmitting large data in hil test environment, system including the same and method thereof | |
| CN107204875B (en) | Data reporting link monitoring method and device, electronic equipment and storage medium | |
| CN112335202B (en) | Processing local area network diagnostic data | |
| CN109495547B (en) | Vehicle signal acquisition method and device, cloud server and vehicle | |
| CN106685429B (en) | Integer compression method and device | |
| CN109684290A (en) | Log storing method, device, equipment and computer readable storage medium | |
| CN111159002A (en) | Data edge acquisition method based on grouping, edge acquisition equipment and system | |
| CN112335203B (en) | Processing local area network diagnostic data | |
| CN111291770A (en) | Parameter configuration method and device | |
| CN111831540A (en) | Debugging information processing method, processing device and storage medium | |
| CN108460044B (en) | Data processing method and device | |
| CN111447229B (en) | Large-scale data acquisition method and device based on compressed sensing theory | |
| CN115865274A (en) | Data compression method, data decompression method and data compression device | |
| Fujimoto et al. | Parallel simulation of statistical multiplexers | |
| CN113746665B (en) | Log data processing method, device and storage medium | |
| CN110247666B (en) | System and method for hardware parallel compression | |
| CN113076107A (en) | Method for automatically acquiring and fusing logs through finite state machine | |
| KR102671446B1 (en) | CAN communication device and CAN frame compression communication method thereof | |
| CN115499338B (en) | Data processing method, device, medium and cloud network observation system | |
| CN113937795B (en) | Distribution transformer area three-phase load data transmission method based on dynamic cloud coding | |
| CN109542662A (en) | A kind of EMS memory management process, device, server and storage medium | |
| CN113759884B (en) | Method and system for generating input/output point product file of distributed control system | |
| CN114189565B (en) | Head area restoration system, method and related equipment | |
| CN112422613B (en) | Data processing method, data processing platform and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 4 / F, building 1, No.14 Jiuxianqiao Road, Chaoyang District, Beijing 100020 Applicant after: Beijing Jingwei Hengrun Technology Co., Ltd Address before: 8 / F, block B, No. 11, Anxiang Beili, Chaoyang District, Beijing 100101 Applicant before: Beijing Jingwei HiRain Technologies Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |