CN117533251A - Distributed file system for vehicle-mounted bus data recorder - Google Patents
Distributed file system for vehicle-mounted bus data recorder Download PDFInfo
- Publication number
- CN117533251A CN117533251A CN202410020462.3A CN202410020462A CN117533251A CN 117533251 A CN117533251 A CN 117533251A CN 202410020462 A CN202410020462 A CN 202410020462A CN 117533251 A CN117533251 A CN 117533251A
- Authority
- CN
- China
- Prior art keywords
- data
- vehicle
- scheduling
- hot standby
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007781 pre-processing Methods 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 31
- 238000004891 communication Methods 0.000 claims abstract description 18
- 238000000034 method Methods 0.000 claims description 23
- 238000003860 storage Methods 0.000 claims description 22
- 238000012937 correction Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 14
- 238000013500 data storage Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 5
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 3
- 238000013144 data compression Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R16/00—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for
- B60R16/02—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for electric constitutive elements
- B60R16/023—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for electric constitutive elements for transmission of signals between vehicle parts or subsystems
- B60R16/0231—Circuits relating to the driving or the functioning of the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention relates to a distributed file system for a vehicle-mounted bus data recorder, in particular to the technical field of vehicle-mounted bus data processing, which comprises a bus communication control module, a bus communication control module and a bus data processing module, wherein the bus communication control module is used for collecting vehicle-mounted bus data; the main node control module is used for starting the child nodes; the scheduling policy module is used for setting scheduling policies for the vehicle-mounted bus data, changing the writing scene according to the data characteristics, compensating the scheduling policies and judging the hot standby policies for the vehicle-mounted bus data; the data preprocessing module is used for preprocessing the vehicle-mounted bus data to obtain preprocessed program command data and Key-Value type quick index data; the data recording module is used for recording the preprocessed program command data and the Key-Value type quick index data in the child node and transmitting the preprocessed program command data and the Key-Value type quick index data to the hot standby equipment. The invention improves the recording efficiency of the vehicle-mounted bus data.
Description
Technical Field
The invention relates to the technical field of vehicle-mounted bus data processing, in particular to a distributed file system for a vehicle-mounted bus data recorder.
Background
In the field of vehicle-mounted bus data recorders, a local storage medium, such as a hard disk, a solid state disk or an internal memory, is generally used in the industry to store data generated by a vehicle, and then the data is exported to an external device in a USB interface or HTTP request mode so as to facilitate subsequent analysis or backup, but for the vehicle-mounted bus data recorders, it is particularly important to ensure the integrity and correctness of the data, and as the fields of intelligent driving and the like are expanded, the throughput of the vehicle bus data is increased, the performance requirement of the data recorders is also increased, the existing technology is difficult to meet the requirements, and the defects of data loss risk, storage limitation, high data management difficulty, poor data compatibility, performance limitation, unfavorable data backup and unfavorable cross-device collaborative work exist.
Chinese patent publication No.: CN106004718A discloses a CAN vehicle bus network system, which is characterized in that the CAN vehicle bus network system comprises the following structures: a signal receiver, an electronic map, a vehicle-mounted image pickup device, an image data processing system, a vehicle component motion signal generator and a vehicle running state data recorder. The CAN vehicle-mounted bus network comprises a plurality of high-speed CAN buses; the number of the high-speed CAN buses is two, and the high-speed CAN buses are respectively a power transmission CAN bus and a vehicle body comfort CAN bus; the integrated gateway integrates the functions of the vehicle body controller and the gateway, and is used as the vehicle body controller to intensively control various functions of vehicle body control. However, according to the scheme, information sharing and cross processing are only carried out on CAN vehicle-mounted bus data, and the vehicle-mounted bus data recording efficiency cannot be improved.
Disclosure of Invention
Therefore, the invention provides a distributed file system for a vehicle-mounted bus data recorder, which is used for solving the problem of low vehicle-mounted bus data processing efficiency in the prior art.
To achieve the above object, the present invention provides a distributed file system for an in-vehicle bus data recorder, comprising:
the bus communication control module is used for controlling the communication bus to collect vehicle-mounted bus data;
the main node control module is used for calculating the starting quantity of the sub-nodes according to the external access interface and starting the sub-nodes according to the starting quantity of the sub-nodes;
the scheduling policy module is used for setting scheduling policies for the vehicle-mounted bus data, changing the writing scene according to the data characteristics, compensating the scheduling policies according to the changing result, and judging the hot standby policies for the vehicle-mounted bus data;
the data preprocessing module is used for selecting a currently idle preprocessing sub-module from the bus data preprocessing pool to preprocess the vehicle-mounted bus data to obtain preprocessed program command data and Key-Value type quick index data;
the data recording module is used for recording the preprocessed program command data and the Key-Value type quick index data in the child node according to the scheduling strategy, and transmitting the preprocessed program command data and the Key-Value type quick index data to the hot standby equipment according to the hot standby strategy.
Further, the master node control module obtains the frequency A of the external access interface call, calculates the starting number B of the child nodes according to the frequency A of the external access interface call, sets B=A/A0×B0, wherein A0 is the frequency of the preset external access interface call, and B0 is the preset starting number of the child nodes, and the master node control module starts the child nodes according to the starting number of the child nodes.
Further, the scheduling policy module is provided with a policy setting unit for setting a scheduling policy for the vehicle-mounted bus data;
the strategy setting unit sets a recording scene as a time sequence scene, sets a scheduling strategy of the vehicle-mounted bus data as a time sequence scheduling strategy, records data according to the sequence of the sub-nodes, switches to the next sub-node for storage after the storage space of the last sub-node is full in the sequence of the sub-nodes, deletes the data with the earliest recording time sequence of the first sub-node in the sequence of the sub-nodes after all the sub-nodes are full, and records the latest data after the deletion.
Further, the scheduling policy module is provided with a policy compensation unit for changing the writing scene according to the data characteristics and compensating the scheduling policy according to the changing result;
the policy compensation unit calculates a data characteristic coefficient F according to the data quantity Q and the data generation speed V, sets f=0.5×q/q0+0.5×v/V0, Q0 is a preset data quantity, and V0 is a preset data generation speed, and compares the characteristic coefficient F with a preset characteristic coefficient F0, and determines a change condition of a writing scene according to a comparison result, wherein:
when F is less than or equal to F0, the strategy compensation unit judges that the writing scene is not changed;
when F is more than F0, the strategy compensation unit judges that the writing scene is changed, the writing scene is changed into a speed distribution scene, and the compensated scheduling strategy is the speed distribution strategy.
Further, the scheduling policy module is provided with a scene correction unit for acquiring scheduling options and correcting the compensation process according to the scheduling options;
the scene correction unit judges the correction condition of the compensation process according to the scheduling options, wherein:
if the scheduling options are speed distribution scenes, the scene correction unit does not correct the compensation process of the scheduling options;
and if the scheduling option is a time sequence scene, the scene correction unit corrects the compensation process of the scheduling option and corrects the data characteristic coefficient F to 0.
Further, the scheduling policy module is provided with a hot standby judging unit for judging the hot standby policy of the vehicle-mounted bus data according to the hot standby period;
the hot standby judging unit acquires a hot standby period, compares the hot standby period T with a preset hot standby period T0, and judges a hot standby strategy according to a comparison result, wherein:
when T is less than or equal to T0, the hot standby judging unit judges that the hot standby strategy is not executed on the child node;
when T > T0, the hot standby judging unit judges that the hot standby strategy is executed on the child node.
Further, the data preprocessing module is provided with a program command preprocessing unit for performing program command conversion preprocessing on the vehicle-mounted bus data;
the program command preprocessing unit preprocesses the vehicle-mounted bus data in the currently idle preprocessing submodule, codes the vehicle-mounted bus data into program commands, adds the program commands to the tail end of a program data storage set, and stores the program commands in the program-increasing data storage set through a self-defined high-speed storage file system.
Further, the data preprocessing module is provided with an index preprocessing unit for carrying out Key-Value type quick index processing on the vehicle-mounted bus data converted by the program command;
the index preprocessing unit acquires the vehicle-mounted bus data converted by the program command from the program-added data storage set, performs Key-Value type quick index processing on the vehicle-mounted bus data converted by the program command, and stores the processed Key-Value type quick index data in a NoSQL type Key-Value type quick index storage library.
Further, the data recording module is provided with a sub-node recording unit for recording the preprocessed program command data and Key-Value type quick index data in the sub-node according to a scheduling strategy;
and the child node recording unit records the preprocessed program command data and Key-Value type quick index data in the child node according to the scheduling strategy of the scheduling strategy module.
Further, the data recording module is provided with a hot standby recording unit for transmitting the preprocessed program command data and Key-Value type quick index data to hot standby equipment according to a hot standby strategy;
the hot standby recording unit transmits the preprocessed program command data and Key-Value quick index data to the hot standby sub-node of the sub-node according to the hot standby strategy of the scheduling strategy module
Compared with the prior art, the system has the advantages that the system is arranged in the vehicle-mounted bus data recorder and is used for carrying out distributed processing on the vehicle-mounted bus data in the vehicle-mounted bus data recorder so as to improve the recording efficiency of the vehicle-mounted bus data, the system controls the communication bus to collect the vehicle-mounted bus data through the bus communication control module so as to receive the vehicle-mounted bus data and record the vehicle-mounted bus data in the follow-up mode, the system calculates the starting number of sub nodes according to the external access interface through the main node control module and starts the sub nodes according to the starting number of the sub nodes so as to record the vehicle-mounted bus data through the sub nodes, the distributed storage of the vehicle-mounted bus data is realized, the recording efficiency of the vehicle-mounted bus data is improved, the system changes the writing scene according to the data characteristics through the scheduling policy module, compensates the scheduling policy according to the changing result so as to carry out real-time adjustment on the scheduling policy according to the changing of the writing scene, the system judges the data through the scheduling policy, the efficiency of the data exchange of the data of the bus is ensured, the data is convenient to realize the data processing efficiency of the data exchange with the vehicle-mounted bus data, the data is improved, the data is ensured to be shared, the data is convenient to realize, the data processing efficiency is improved, the vehicle-mounted bus data is easy to realize, the data is convenient to be shared, the data is stored, and has better data processing efficiency, and has better data processing efficiency, the system records the preprocessed program command data and Key-Value type quick index data in the child node according to a scheduling strategy through a data recording module, and transmits the preprocessed program command data and Key-Value type quick index data to hot standby equipment according to a hot standby strategy, so that the efficiency of storing, searching and exporting vehicle-mounted data is improved, and a user can quickly search massive vehicle-mounted bus data through an external access interface of a main node when inquiring the data, so that the recording efficiency of the vehicle-mounted bus data is improved.
Drawings
FIG. 1 is a schematic diagram of a distributed file system for a vehicle bus data recorder according to the present embodiment;
FIG. 2 is a schematic diagram of a scheduling policy module according to the present embodiment;
FIG. 3 is a schematic diagram of a data preprocessing module according to the present embodiment;
fig. 4 is a schematic structural diagram of a data recording module according to the present embodiment.
Detailed Description
In order that the objects and advantages of the invention will become more apparent, the invention will be further described with reference to the following examples; it should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are merely for explaining the technical principles of the present invention, and are not intended to limit the scope of the present invention.
Furthermore, it should be noted that, in the description of the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention can be understood by those skilled in the art according to the specific circumstances.
Referring to fig. 1, a schematic structural diagram of a distributed file system for a vehicle bus data recorder according to the present embodiment is shown, where the system includes:
the bus communication control module is used for controlling the communication bus to collect vehicle-mounted bus data;
the main node control module is used for calculating the starting quantity of the sub-nodes according to the external access interface, starting the sub-nodes according to the starting quantity of the sub-nodes, and connecting the main node control module with the bus communication control module;
the scheduling policy module is used for setting scheduling policies for the vehicle-mounted bus data, changing the writing scene according to the data characteristics, compensating the scheduling policies according to the changing result, and judging the hot standby policies for the vehicle-mounted bus data, and is connected with the main node control module;
the data preprocessing module is used for selecting a currently idle preprocessing sub-module from the bus data preprocessing pool to preprocess the vehicle-mounted bus data to obtain preprocessed program command data and Key-Value type quick index data, and the data preprocessing module is connected with the main node control module;
the data recording module is used for recording the preprocessed program command data and Key-Value type quick index data in the child node according to the scheduling strategy, and transmitting the preprocessed program command data and Key-Value type quick index data to the hot standby equipment according to the hot standby strategy, and is connected with the data preprocessing module and the scheduling strategy module.
Specifically, the system is arranged in a vehicle-mounted bus data recorder and is used for carrying out distributed processing on vehicle-mounted bus data in the vehicle-mounted bus data recorder so as to improve the recording efficiency of the vehicle-mounted bus data, the system controls the communication bus to collect the vehicle-mounted bus data through the bus communication control module so as to receive the vehicle-mounted bus data and record the vehicle-mounted bus data in the follow-up process, the system calculates the starting number of sub nodes according to an external access interface through the master node control module and starts the sub nodes according to the starting number of the sub nodes so as to record the vehicle-mounted bus data through the sub nodes, the distributed storage of the vehicle-mounted bus data is realized, the recording efficiency of the vehicle-mounted bus data is improved, the system sets a scheduling strategy for the vehicle-mounted bus data through the scheduling strategy module, so as to use different writing scenes, the system changes the writing scenes according to the data characteristics through a dispatching policy module, compensates the dispatching policy according to the changing result, so as to adjust the dispatching policy in real time according to the changing of the writing scenes, thereby improving the recording efficiency of the vehicle-mounted bus data, judges the vehicle-mounted bus data through a hot standby policy through the dispatching policy module, judges the execution of the vehicle-mounted bus data, ensures the safety of the data, thereby improving the recording efficiency of the vehicle-mounted bus data, preprocesses the vehicle-mounted bus data through a data preprocessing module, improving the data processing efficiency, simplifying the data exchange and sharing, realizing the data compression and storage optimization, standardizing the data format, facilitating the subsequent data processing and analysis, thereby improving the recording efficiency of the vehicle-mounted bus data, the system records the preprocessed program command data and Key-Value type quick index data in the child node according to a scheduling strategy through a data recording module, and transmits the preprocessed program command data and Key-Value type quick index data to hot standby equipment according to a hot standby strategy, so that the efficiency of storing, searching and exporting vehicle-mounted data is improved, and a user can quickly search massive vehicle-mounted bus data through an external access interface of a main node when inquiring the data, so that the recording efficiency of the vehicle-mounted bus data is improved.
Specifically, the vehicle-mounted bus data comprises vehicle-mounted bus data source CAN data, vehicle-mounted bus data source LIN data, vehicle-mounted bus data source UART data, vehicle-mounted bus data source ETH data, vehicle-mounted bus data source FLEXRAY data and vehicle-mounted bus data source etc.
Specifically, the master node control module obtains the frequency a of the external access interface call, calculates the starting number B of the child nodes according to the frequency a of the external access interface call, sets b=a/a0×b0, wherein A0 is the frequency of the preset external access interface call, B0 is the preset starting number of the child nodes, and the master node control module starts the child nodes according to the starting number of the child nodes.
Specifically, the frequency of the external access interface call by the user may use the log analysis tool to obtain call data of the external access interface, including information such as a timestamp of the interface call, identity of a caller, call frequency, etc., where in this embodiment, the frequency of the external access interface call is preset to be 10 times/min, and the number of preset sub-nodes is preset to be 2.
Referring to fig. 2, a schematic structural diagram of a scheduling policy module according to the present embodiment is shown, where the scheduling policy module includes:
the strategy setting unit is used for setting a scheduling strategy for the vehicle-mounted bus data;
the strategy compensation unit is used for changing the writing scene according to the data characteristics and compensating the scheduling strategy according to the changing result, and is connected with the strategy setting unit;
the scene correction unit is used for acquiring the scheduling options, correcting the compensation process according to the scheduling options and is connected with the strategy compensation unit;
and the hot standby judging unit is used for carrying out hot standby strategy judgment on the vehicle-mounted bus data according to the hot standby period and is connected with the strategy setting unit.
Specifically, the policy setting unit sets a recording scene as a time sequence scene, sets a scheduling policy of the vehicle-mounted bus data as a time sequence scheduling policy, records data according to a sub-node sequence, switches to a next sub-node for storage after the storage space of a previous sub-node is full in the sub-node sequence, deletes data with the earliest recording time sequence from the first sub-node in the sub-node sequence after all the sub-nodes are full, and records the latest data after the deletion.
Specifically, the time sequence scene refers to a data writing scene with high time sequence requirement, the time sequence scheduling policy refers to a data recording policy applicable to the time sequence with high time sequence requirement, the sequence of the sub-nodes refers to a sequence obtained after the sub-nodes are sequenced according to a preset mode, the preset mode of the sub-node sequence is not limited in this embodiment, a person skilled in the art can set the preset mode according to actual conditions, if the preset mode can be set to sequence according to the starting time of the sub-nodes from the minimum to the maximum, the last sub-node refers to an adjacent previous sub-node taking a designated sub-node as a coordinate in the sequence of the sub-nodes, the storage space write-full refers to a condition that when the last sub-node is the designated sub-node, the next sub-node refers to a sub-node adjacent to the last sub-node, the storage space of all the started sub-nodes is full, the first sub-node refers to a condition that the sub-node in the sub-node sequence is sequenced according to the starting time, the last sub-node is full, the vehicle bus is used for recording the latest data according to the time sequence of the latest bus, and the latest data is recorded according to the time sequence bus.
Specifically, the policy setting unit sets the scheduling policy of the vehicle-mounted bus data as a time sequence scheduling policy, so that the recording of the vehicle-mounted bus data meets the requirement of a writing scene with high time sequence requirement, and the recording efficiency of the vehicle-mounted bus data is improved.
Specifically, the policy compensation unit calculates a data characteristic coefficient F according to the data amount Q and the data generation speed V, sets f=0.5×q/q0+0.5×v/V0, Q0 is a preset data amount, and V0 is a preset data generation speed, and compares the characteristic coefficient F with a preset characteristic coefficient F0, and determines a change condition of the writing scene according to a comparison result, where:
when F is less than or equal to F0, the strategy compensation unit judges that the writing scene is not changed;
when F is more than F0, the strategy compensation unit judges that the writing scene is changed, the writing scene is changed into a speed distribution scene, and the compensated scheduling strategy is the speed distribution strategy.
Specifically, the data volume refers to the size of the vehicle-mounted bus data volume collected by the bus communication control module in a preset compensation period, the data generation speed refers to the average production speed of the vehicle-mounted bus data volume collected by the bus communication control module in the preset compensation period, if the vehicle-mounted bus data volume is large in one minute, a large amount of effective data is generated, the bus communication control module collects the effective data, the vehicle-mounted bus data volume is large in one minute, the production speed is high, the preset data volume refers to a preset value representing the large data volume, if the preset compensation period is 1 minute, the vehicle-mounted bus data of 100M is set to be the preset data volume, the preset data generation speed refers to a preset value representing the high data generation speed, if the preset compensation period is 1 minute, the preset data generation speed is set to be 10M/s, the write-in scene refers to a data recording applicable scene of a scheduling strategy, the write-in scene comprises a time sequence scene and a speed distribution scene, the speed distribution scene refers to a data recording scene requiring the write-in speed and the data type, if the write-in speed and the data type are not met, the write-in the data can be set by a special strategy, and the data can be recorded by a person in the field defining the data distribution policy according to the data type, and the data type can be set in the field.
Specifically, the strategy compensation unit changes the writing scene according to the data characteristic coefficient, changes the writing scene into a speed distribution scene, and compensates the scheduling strategy, wherein the compensated scheduling strategy is the speed distribution strategy so as to adapt to the data recording scene with requirements on the writing speed and the data type, thereby improving the recording efficiency of the vehicle bus data.
Specifically, the scene correction unit determines a correction condition of the compensation process according to the scheduling option, wherein:
if the scheduling options are speed distribution scenes, the scene correction unit does not correct the compensation process of the scheduling options;
and if the scheduling option is a time sequence scene, the scene correction unit corrects the compensation process of the scheduling option and corrects the data characteristic coefficient F to 0.
Specifically, the scheduling options refer to selection of a writing scene by a user, the embodiment does not limit the acquisition mode of the scheduling options, and a person skilled in the art can freely set the scheduling options, for example, the scheduling options can be set outside the vehicle-mounted bus data recorder and externally connected with a vehicle-mounted cabin central control screen, and pushed to the vehicle-mounted cabin central control screen for the user to select, and the user selection result is obtained as the scheduling options.
Specifically, when the scheduling option is a time sequence scene, the scene correction unit corrects the compensation process of the scheduling option, corrects the data characteristic coefficient F to 0, adjusts the scheduling strategy according to the user option, and does not change the writing scene when the scheduling option is the time sequence scene, thereby further improving the recording efficiency of the vehicle-mounted bus data.
Specifically, the hot standby judging unit acquires a hot standby period, compares the hot standby period T with a preset hot standby period T0, and judges a hot standby policy according to a comparison result, wherein:
when T is less than or equal to T0, the hot standby judging unit judges that the hot standby strategy is not executed on the child node;
when T > T0, the hot standby judging unit judges that the hot standby strategy is executed on the child node.
Specifically, the hot standby period refers to a period of time from the current time to the time when the last child node performs hot standby, the preset hot standby period refers to a preset period of hot standby judgment on the child node, if the preset hot standby period is set to be one week, the hot standby policy refers to setting one hot standby child node on the child node, and periodically backing up vehicle bus data from the child node to the hot standby child node.
Specifically, the hot standby judging unit performs hot standby strategy judgment according to the hot standby period so as to backup the vehicle-mounted bus data, and ensure the storage safety of the vehicle-mounted bus data, thereby improving the recording efficiency of the vehicle-mounted bus data.
Fig. 3 is a schematic structural diagram of a data preprocessing module according to the present embodiment, where the data preprocessing module includes:
a program command preprocessing unit for performing program command conversion preprocessing on the vehicle-mounted bus data;
the index preprocessing unit is used for carrying out Key-Value type quick index processing on the vehicle-mounted bus data after program command conversion, and is connected with the program command preprocessing unit.
Specifically, the program command preprocessing unit preprocesses the vehicle-mounted bus data in the currently idle preprocessing sub-module, encodes the vehicle-mounted bus data into program commands, adds the program commands to the tail end of the program data storage set, and stores the program commands in the program-increasing data storage set through a self-defined high-speed storage file system.
Specifically, the present embodiment does not limit the determination mode of the currently idle preprocessing sub-module, and a person skilled in the art can freely set the method, which only needs to meet the determination requirement of the idle preprocessing sub-module, if the method can be used for determining the currently idle preprocessing sub-module by a polling method, the method of encoding the vehicle-mounted bus data into the program command is not specifically limited, and the person skilled in the art can freely set the method, which only needs to meet the conversion requirement of the program command, if the method of encoding the vehicle-mounted bus data into the program command can be set as follows: the method comprises the steps of obtaining the format and protocol of bus data, writing a conversion program according to the format and protocol of the bus data, reading the bus data according to the conversion program, analyzing the bus data, converting the bus data into program commands, debugging and testing by using a testing tool or analog data, integrating the conversion program into a system to carry out joint debugging and testing after the conversion program meets the expectation, and encoding the conversion program into the program commands as vehicle-mounted bus data when the whole system correctly receives and processes the bus data and converts the bus data into the corresponding program commands.
Specifically, the program command preprocessing unit preprocesses the vehicle-mounted bus data in the current idle preprocessing submodule, codes the vehicle-mounted bus data into program commands, thereby improving the data processing efficiency, simplifying data exchange and sharing, realizing data compression and storage optimization, standardizing the data format, facilitating subsequent data processing and analysis, storing in a program-increasing data storage set through a self-defined high-speed storage file system, improving the efficiency of storing, searching and exporting the vehicle-mounted bus data, and improving the recording efficiency of the vehicle-mounted bus data.
Specifically, the index preprocessing unit acquires the vehicle-mounted bus data converted by the program command from the program-added data storage set, performs Key-Value type quick index processing on the vehicle-mounted bus data converted by the program command, and stores the processed Key-Value type quick index data in a NoSQL type Key-Value type quick index storage library.
Specifically, the Key-Value type fast index processing mode is not specifically limited, a person skilled in the art can freely set the Key-Value type fast index processing mode, if the index processing requirement on the vehicle-mounted bus data is met, the operation of preprocessing the converted vehicle-mounted bus data including data cleaning, format conversion, standardization and the like can be set, keys with uniqueness and identifiability are designed according to the characteristics and query requirement of the vehicle-mounted bus data, keys capable of accurately representing the characteristics and attributes of the data are designed, for example, information such as device IDs, time stamps, geographic positions and the like is used as part of keys, values containing actual data values or other related information are designed according to the types and query requirement of the data, the values are processed by using methods such as serialization or compression, so that storage space is reduced, query efficiency is improved, a proper storage structure is selected to store Key-Value pairs, such as a hash table, a B+ tree, an LSM tree and the like, an index structure is built by using an existing index library or self-realizing index, the index structure is used according to the query requirement, the Key, the index is matched, the index processing mode is optimized after the index is used, the index processing mode is finished, the index processing performance is optimized, and the index processing is finished, and the index performance is optimized, and the index processing is consistent, and the index processing is optimized.
Fig. 4 is a schematic structural diagram of a data recording module according to the present embodiment, where the data recording module includes:
the child node recording unit is used for recording the preprocessed program command data and Key-Value type quick index data in the child node according to the scheduling strategy;
and the hot standby recording unit is used for transmitting the preprocessed program command data and Key-Value type quick index data to hot standby equipment according to a hot standby strategy, and the hot standby recording unit is connected with the child node recording unit.
Specifically, the child node recording unit records the preprocessed program command data and Key-Value quick index data in the child node according to the scheduling policy of the scheduling policy module.
Specifically, the hot standby recording unit transmits the preprocessed program command data and the Key-Value quick index data to the hot standby sub-node of the sub-node according to the hot standby strategy of the scheduling strategy module.
Specifically, the hot standby device refers to a device for backing up the preprocessed program command data and the Key-Value quick index data stored in the child nodes, and comprises hot standby child nodes of all the child nodes.
Specifically, the child node recording unit and the hot standby recording unit are used for recording and hot standby the vehicle-mounted bus data, so that a user can quickly search massive vehicle-mounted bus data through an external access interface of the main node when inquiring the data, and the recording efficiency of the vehicle-mounted bus data is improved.
Thus far, the technical solution of the present invention has been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of protection of the present invention is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such modifications and substitutions will be within the scope of the present invention.
Claims (10)
1. A distributed file system for an on-board bus data logger, comprising:
the bus communication control module is used for controlling the communication bus to collect vehicle-mounted bus data;
the main node control module is used for calculating the starting quantity of the sub-nodes according to the external access interface and starting the sub-nodes according to the starting quantity of the sub-nodes;
the scheduling policy module is used for setting scheduling policies for the vehicle-mounted bus data, changing the writing scene according to the data characteristics, compensating the scheduling policies according to the changing result, and judging the hot standby policies for the vehicle-mounted bus data;
the data preprocessing module is used for selecting a currently idle preprocessing sub-module from the bus data preprocessing pool to preprocess the vehicle-mounted bus data to obtain preprocessed program command data and Key-Value type quick index data;
the data recording module is used for recording the preprocessed program command data and the Key-Value type quick index data in the child node according to the scheduling strategy, and transmitting the preprocessed program command data and the Key-Value type quick index data to the hot standby equipment according to the hot standby strategy.
2. The distributed file system for a vehicle-mounted bus data recorder according to claim 1, wherein the master node control module obtains a frequency a of external access interface calls, calculates a sub-node start number B according to the frequency a of external access interface calls, sets b=a/a0×b0, A0 is a preset frequency of external access interface calls, B0 is a preset sub-node start number, and the master node control module starts the sub-nodes according to the sub-node start number.
3. The distributed file system for a vehicle-mounted bus data recorder according to claim 1, wherein the scheduling policy module is provided with a policy setting unit for setting a scheduling policy for vehicle-mounted bus data;
the strategy setting unit sets a recording scene as a time sequence scene, sets a scheduling strategy of the vehicle-mounted bus data as a time sequence scheduling strategy, records data according to the sequence of the sub-nodes, switches to the next sub-node for storage after the storage space of the last sub-node is full in the sequence of the sub-nodes, deletes the data with the earliest recording time sequence of the first sub-node in the sequence of the sub-nodes after all the sub-nodes are full, and records the latest data after the deletion.
4. A distributed file system for a vehicle bus data recorder according to claim 3, wherein the scheduling policy module is provided with a policy compensation unit for changing a write scene according to data characteristics and compensating a scheduling policy according to a change result;
the policy compensation unit calculates a data characteristic coefficient F according to the data quantity Q and the data generation speed V, sets f=0.5×q/q0+0.5×v/V0, Q0 is a preset data quantity, and V0 is a preset data generation speed, and compares the characteristic coefficient F with a preset characteristic coefficient F0, and determines a change condition of a writing scene according to a comparison result, wherein:
when F is less than or equal to F0, the strategy compensation unit judges that the writing scene is not changed;
when F is more than F0, the strategy compensation unit judges that the writing scene is changed, the writing scene is changed into a speed distribution scene, and the compensated scheduling strategy is the speed distribution strategy.
5. The distributed file system for an on-board bus data recorder according to claim 4, wherein the scheduling policy module is provided with a scene correction unit for acquiring scheduling options and correcting the compensation process according to the scheduling options;
the scene correction unit judges the correction condition of the compensation process according to the scheduling options, wherein:
if the scheduling options are speed distribution scenes, the scene correction unit does not correct the compensation process of the scheduling options;
and if the scheduling option is a time sequence scene, the scene correction unit corrects the compensation process of the scheduling option and corrects the data characteristic coefficient F to 0.
6. The distributed file system for the vehicle-mounted bus data recorder according to claim 5, wherein the scheduling policy module is provided with a hot standby determination unit for performing hot standby policy determination on vehicle-mounted bus data according to a hot standby period;
the hot standby judging unit acquires a hot standby period, compares the hot standby period T with a preset hot standby period T0, and judges a hot standby strategy according to a comparison result, wherein:
when T is less than or equal to T0, the hot standby judging unit judges that the hot standby strategy is not executed on the child node;
when T > T0, the hot standby judging unit judges that the hot standby strategy is executed on the child node.
7. The distributed file system for a vehicle-mounted bus data recorder according to claim 1, wherein the data preprocessing module is provided with a program command preprocessing unit for performing program command conversion preprocessing on vehicle-mounted bus data;
the program command preprocessing unit preprocesses the vehicle-mounted bus data in the currently idle preprocessing submodule, codes the vehicle-mounted bus data into program commands, adds the program commands to the tail end of a program data storage set, and stores the program commands in the program-increasing data storage set through a self-defined high-speed storage file system.
8. The distributed file system for the vehicle-mounted bus data recorder according to claim 7, wherein the data preprocessing module is provided with an index preprocessing unit for performing Key-Value type fast index processing on the vehicle-mounted bus data after program command conversion;
the index preprocessing unit acquires the vehicle-mounted bus data converted by the program command from the program-added data storage set, performs Key-Value type quick index processing on the vehicle-mounted bus data converted by the program command, and stores the processed Key-Value type quick index data in a NoSQL type Key-Value type quick index storage library.
9. The distributed file system for a vehicle-mounted bus data recorder according to claim 1, wherein the data recording module is provided with a sub-node recording unit for recording the preprocessed program command data and Key-Value type fast index data in the sub-node according to a scheduling policy;
and the child node recording unit records the preprocessed program command data and Key-Value type quick index data in the child node according to the scheduling strategy of the scheduling strategy module.
10. The distributed file system for the vehicle-mounted bus data recorder according to claim 9, wherein the data recording module is provided with a hot standby recording unit for transmitting the preprocessed program command data and Key-Value type quick index data to the hot standby equipment according to a hot standby strategy;
and the hot standby recording unit transmits the preprocessed program command data and the Key-Value type quick index data to the hot standby sub-node of the sub-node according to the hot standby strategy of the scheduling strategy module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410020462.3A CN117533251B (en) | 2024-01-08 | Distributed file system for vehicle-mounted bus data recorder |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410020462.3A CN117533251B (en) | 2024-01-08 | Distributed file system for vehicle-mounted bus data recorder |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117533251A true CN117533251A (en) | 2024-02-09 |
| CN117533251B CN117533251B (en) | 2024-05-10 |
Family
ID=
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528649A (en) * | 2016-10-14 | 2017-03-22 | 宁波华太车载技术有限公司 | Massive data storage and retrieval system and massive data storage and retrieval methods for new energy vehicles |
| CN110750480A (en) * | 2019-10-18 | 2020-02-04 | 苏州浪潮智能科技有限公司 | Dual-computer hot standby system |
| US20220019555A1 (en) * | 2020-07-17 | 2022-01-20 | Rubrik, Inc. | Snapshot and restoration of distributed file system |
| CN114385755A (en) * | 2021-12-30 | 2022-04-22 | 苏州中科先进技术研究院有限公司 | Distributed storage system |
| CN116264579A (en) * | 2022-07-14 | 2023-06-16 | 中移(苏州)软件技术有限公司 | Distributed log analysis processing method, device, system and storage medium |
| CN116795507A (en) * | 2023-06-08 | 2023-09-22 | 中科云谷科技有限公司 | Distributed task scheduling method, system and scheduling device |
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528649A (en) * | 2016-10-14 | 2017-03-22 | 宁波华太车载技术有限公司 | Massive data storage and retrieval system and massive data storage and retrieval methods for new energy vehicles |
| CN110750480A (en) * | 2019-10-18 | 2020-02-04 | 苏州浪潮智能科技有限公司 | Dual-computer hot standby system |
| US20220019555A1 (en) * | 2020-07-17 | 2022-01-20 | Rubrik, Inc. | Snapshot and restoration of distributed file system |
| CN114385755A (en) * | 2021-12-30 | 2022-04-22 | 苏州中科先进技术研究院有限公司 | Distributed storage system |
| CN116264579A (en) * | 2022-07-14 | 2023-06-16 | 中移(苏州)软件技术有限公司 | Distributed log analysis processing method, device, system and storage medium |
| CN116795507A (en) * | 2023-06-08 | 2023-09-22 | 中科云谷科技有限公司 | Distributed task scheduling method, system and scheduling device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107273280A (en) | A kind of log processing method, device, electronic equipment and storage medium | |
| CN102244758A (en) | Video-recording-file-based data acquisition method and equipment | |
| CN108848517B (en) | Rail transit signal coverage monitoring method | |
| EP4170514A1 (en) | Data association query method and apparatus, and device and storage medium | |
| CN107393308A (en) | A kind of method, apparatus and managing system of car parking for identifying car plate | |
| CN117533251B (en) | Distributed file system for vehicle-mounted bus data recorder | |
| CN113111140A (en) | Method for rapidly analyzing multi-source marine business observation data | |
| CN110505307B (en) | Method and system for exchanging traffic flow data between networks | |
| CN117533251A (en) | Distributed file system for vehicle-mounted bus data recorder | |
| CN113328927B (en) | Industrial Internet of things gateway with calculation, storage and network exchange functions | |
| CN113253707B (en) | System integration testing method and device for novel train control vehicle-mounted equipment | |
| WO2021147319A1 (en) | Data processing method, apparatus, device, and medium | |
| CN102222373A (en) | On-vehicle recorder | |
| CN117061165A (en) | Safety protection system based on space-time data lake technology of monitoring and control system | |
| CN115412620B (en) | Substation data analysis method based on communication protocol database system | |
| CN113784237A (en) | Method, device, network equipment and computer medium for reporting collected data | |
| CN116010388A (en) | Data verification method, data acquisition server and data verification system | |
| CN115113821B (en) | 5G big data power calculation service system based on quantum encryption | |
| KR20210056583A (en) | System and method for collecting vehicle data of autonomous electric vehicle | |
| CN109828489B (en) | General type comprehensive monitoring preposed communication controller software | |
| US7848735B2 (en) | Communication control apparatus, method of controlling sender terminal number, and computer product | |
| CN111861778A (en) | Electric cabinet manufacturing management system | |
| CN111654671A (en) | Video data storage method, device, equipment and storage medium | |
| CN116760115B (en) | Space-time feature fusion power system unit combination optimization method and system thereof | |
| CN213024059U (en) | Data recording device and vehicle-mounted data recording device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: Room 01, 8th Floor, Building 1, No. 86 Jianguo Road, Chaoyang District, Beijing, 100020 Applicant after: Zhidi Technology (Anhui) Co.,Ltd. Address before: Room 01, 8th Floor, Building 1, No. 86 Jianguo Road, Chaoyang District, Beijing, 100020 Applicant before: Zhidi Automotive Technology (Beijing) Co.,Ltd. Country or region before: China |
|
| CB02 | Change of applicant information |
Country or region after: China Address after: 230000, A2-05, 2nd Floor, Building A4, Phase I, Zhong'an Chuanggu Science and Technology Park, No. 900 Wangjiang West Road, High tech Zone, Hefei City, Anhui Province Applicant after: Zhidi Technology (Anhui) Co.,Ltd. Address before: Room 01, 8th Floor, Building 1, No. 86 Jianguo Road, Chaoyang District, Beijing, 100020 Applicant before: Zhidi Technology (Anhui) Co.,Ltd. Country or region before: China |
|
| GR01 | Patent grant |