CN117891400A - Model simulation data storage method, device, equipment and storage medium - Google Patents
Model simulation data storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117891400A CN117891400A CN202311842462.3A CN202311842462A CN117891400A CN 117891400 A CN117891400 A CN 117891400A CN 202311842462 A CN202311842462 A CN 202311842462A CN 117891400 A CN117891400 A CN 117891400A
- Authority
- CN
- China
- Prior art keywords
- data
- simulation
- target
- storage
- variable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004088 simulation Methods 0.000 title claims abstract description 237
- 238000013500 data storage Methods 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000013506 data mapping Methods 0.000 claims description 46
- 238000005192 partition Methods 0.000 claims description 44
- 238000013507 mapping Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 7
- 238000006243 chemical reaction Methods 0.000 abstract description 5
- 238000013499 data model Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a model simulation data storage method, a device, equipment and a storage medium, wherein the method comprises the following steps: obtaining a target simulation data table of a Modelica model; converting the table attribute format of the target simulation data table to obtain a target storage data table; and storing the target storage data table in a preset distributed database. The technical scheme of the embodiment of the invention solves the problems that in the existing Modelica model simulation data storage technology, single storage occupies more disk IO of a single server, and the efficiency and the instantaneity of data storage are insufficient, and can be used for storing a model simulation data table in a distributed database after format conversion, thereby improving the efficiency and the instantaneity of data storage.
Description
Technical Field
The embodiment of the invention relates to the technical field of software engineering, in particular to a model simulation data storage method, device, equipment and storage medium.
Background
The form of the industrial software at the present stage is superior to that of a single client, and each large industrial software provides an online cloud platform, so that a user can conveniently acquire the industrial software at any time and any place. In the present stage of online model modeling and simulation cloud platform, a traditional single-machine file system storage mode is used, in a cloud simulation scene, single-user triggering concurrent simulation occupies all disk IO (input output) of a single server, and the efficiency and instantaneity of data storage are to be improved.
Disclosure of Invention
The embodiment of the invention provides a method, a device, equipment and a storage medium for storing model simulation data, which can convert the format of a model simulation data table and store the converted format in a distributed database, thereby improving the efficiency and the instantaneity of data storage.
In a first aspect, an embodiment of the present invention provides a method for storing model simulation data, where the method includes:
obtaining a target simulation data table of a Modelica model;
converting the table attribute format of the target simulation data table to obtain a target storage data table;
and storing the target storage data table in a preset distributed database.
In a second aspect, an embodiment of the present invention provides a model simulation data storage apparatus, including:
the simulation data table acquisition module is used for acquiring a target simulation data table of the Modelica model;
the storage data table generation module is used for converting the table attribute format of the target simulation data table to obtain a target storage data table;
and the data table storage module is used for storing the target storage data table in a preset distributed database.
In a third aspect, an embodiment of the present invention provides a computer apparatus, including:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the model simulation data storage method of any of the embodiments.
In a fourth aspect, embodiments of the present invention provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the model simulation data storage method of any of the embodiments.
According to the technical scheme provided by the embodiment of the invention, the target simulation data table of the Modelica model is obtained; converting the table attribute format of the target simulation data table to obtain a target storage data table; and storing the target storage data table in a preset distributed database. The technical scheme of the embodiment of the invention solves the problems that in the existing Modelica model simulation data storage technology, single storage occupies more disk IO of a single server, and the efficiency and the instantaneity of data storage are insufficient, and can be used for storing a model simulation data table in a distributed database after format conversion, thereby improving the efficiency and the instantaneity of data storage.
Drawings
FIG. 1 is a flow chart of a method for storing model simulation data provided by an embodiment of the invention;
FIG. 2 is a flow chart of yet another method for model simulation data storage provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of a model simulation data storage device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flowchart of a method for storing model simulation data according to an embodiment of the present invention, where the method may be implemented by a model simulation data storage device, and the device may be implemented by software and/or hardware.
As shown in fig. 1, the model simulation data storage method includes the steps of:
s110, acquiring a target simulation data table of the Modelica model.
The target simulation data table may be a table of simulation data generated by a model. The technical scheme of the embodiment of the invention needs to store the target simulation data table in real time. The data simulation variables generate corresponding simulation data in real time. That is, the real-time simulation data of all the data simulation variables are recorded in the target simulation data table.
S120, converting the table attribute format of the target simulation data table to obtain a target storage data table.
Wherein the table attribute format may be table rank attribute data of the target simulation data table. Specifically, the column data of the distributed database has weak storage capacity, so that row data and column data of the target simulation data table can be converted, and finally, a table capable of being stored in the distributed database in real time is obtained. The target storage data table may be a storage data table corresponding to the finally determined target simulation data table. Specifically, the simulation variables in the target simulation data table can be respectively determined, the mapping positions corresponding to each simulation variable are respectively determined, and finally the simulation data of each simulation variable are mapped to the corresponding mapping positions, so that the target storage data table is obtained.
S130, storing the target storage data table in a preset distributed database.
The preset distributed database can be a preset database which can be stored in a distributed mode. Alternatively, the preset distributed database may be an Apache Cassandra database. Apache Cassandra is an open source distributed and decentralized/distributed storage system (database) that manages large amounts of structured data distributed throughout the world. It provides high availability services without a single point of failure. Has the following characteristics: the single table can store 1000 hundred million data; the single library is within 100 tables; the upper limit of the number of the single table fields is controlled to be 20-60.
The cloud real-time simulation technology based on Apache transfers the fields in the column dimension to the row dimension through a reasonably designed data model, reasonably applies the advantage capability of Cansandra row-direction storage, also supports the scene when the number of Modelica model variables is too large, and enables the disk IO not to be the bottleneck of large-scale simulation result access, so that the real-time storage and reading of massive simulation result data generated by the cloud Modelica model in the simulation process are possible.
According to the technical scheme provided by the embodiment of the invention, the target simulation data table of the Modelica model is obtained; converting the table attribute format of the target simulation data table to obtain a target storage data table; and storing the target storage data table in a preset distributed database. The technical scheme of the embodiment of the invention solves the problems that in the existing Modelica model simulation data storage technology, single storage occupies more disk IO of a single server, and the efficiency and the instantaneity of data storage are insufficient, and can be used for storing a model simulation data table in a distributed database after format conversion, thereby improving the efficiency and the instantaneity of data storage.
Fig. 2 is a flowchart of another method for storing model simulation data provided by the embodiment of the present invention, where the embodiment of the present invention is applicable to a scenario of storing simulation data generated by a model, and on the basis of the foregoing embodiment, how to convert a table attribute format of a target simulation data table to obtain a target storage data table and how to store the target storage data table in a preset distributed database.
As shown in fig. 2, the model simulation data storage method includes the steps of:
s210, acquiring a target simulation data table of a Modelica model, determining at least one data simulation variable according to the target simulation data table, and numbering each data simulation variable to obtain a simulation variable number.
The target simulation data table may be a table of simulation data generated by a model. The technical scheme of the embodiment of the invention needs to store the target simulation data table in real time. Exemplary, specific formats of the target simulation data table are shown in Table 1 below:
table 1 simulation result data format
| Time | varName 2 | varName 3 | … | varName n |
| 0 | value 12 | value 13 | … | value 1n |
| 0.002 | value 22 | value 23 | … | value 2n |
| … | … | … | … | … |
| m | value m2 | value m3 | … | value mn |
The data simulation variables generate corresponding simulation data in real time. That is, the real-time simulation data of all the data simulation variables are recorded in the target simulation data table. Specifically, the names of all the data simulation variables are in the header of the target assurance data table, so that all the data simulation variables can be determined according to the header of the target simulation data table.
Further, the simulation variable number may be a number of each data simulation variable. After the data simulation variables are determined, the data simulation variables can be numbered respectively, and then the simulation variable numbers corresponding to the data simulation variables are obtained. The simulation variable number is determined, so that the simulation variable number can be used as identification information of a data control variable, and the storage position of simulation data can be determined according to the simulation variable number.
Optionally, determining at least one data simulation variable according to the target simulation data table, and numbering each data simulation variable to obtain a simulation variable number, including: identifying header data of a target simulation data table, and determining at least one data simulation variable; sequentially mapping the data simulation variables into a preset linear set to obtain a target variable linear set; and respectively determining the simulation variable numbers of each data simulation variable according to the arrangement sequence of the variables in the target variable linear set.
The header data may be variable name data in the header of the target simulation data table. Specifically, the name of each simulation variable in the header data can be identified, and at least one data simulation variable is obtained. Further, the preset linear set may be a linear set of preset storage variable names. The linear set of target variables may be a set that includes all of the data simulation variables in the target simulation data table. Specifically, after each data simulation variable in the target simulation data table is determined, the data simulation variables can be mapped into a preset linear set in sequence, so that a target variable linear set is obtained, and further, according to the arrangement sequence of each simulation variable in the target variable linear set, the simulation variable number corresponding to each data simulation variable can be determined.
S220, based on the simulation variable numbers and a preset row variable threshold, the data mapping positions of each data simulation variable are respectively determined.
The preset row variable threshold may be a number threshold of simulation variables in the same row. In particular, since the column capacity of the distributed data store is weak, the number of variables in the same row in the data table can be limited by setting a preset row variable threshold. The data mapping locations may be storage locations of the simulation variable data in a storage table. Specifically, the data mapping position can be determined together according to the simulation variable number and the preset row variable threshold.
Optionally, determining the data mapping position of each data simulation variable based on the simulation variable number and the preset row variable threshold value includes: the simulation variable number is rounded up to a preset row variable threshold value to obtain a data mapping row position; taking the remainder of the simulation variable number for a preset row variable threshold value to obtain a data mapping column position; and determining the data mapping position according to the data mapping row position and the data mapping column position.
Wherein the data mapping row location may be a row location of the simulation variable data in a memory table. Specifically, the simulation variable number can be rounded up to a preset row variable threshold value to obtain the data mapping row position. Further, the data mapping column position may be a column position that prevents variable data from being in the data table. Specifically, the simulation variable number may be used to balance a preset row variable threshold value, so as to obtain a data mapping column position. Further, the data mapping row locations and the data mapping column locations may be combined to determine the data mapping locations.
S230, mapping the simulation data of each data simulation variable to a data mapping position to obtain a target storage data table.
The target storage data table may be a storage data table corresponding to the finally determined target simulation data table. Specifically, according to the data mapping position determined in the above steps, the simulation data of each simulation variable is mapped to the corresponding data mapping position, so as to obtain the target storage data table.
Optionally, mapping the simulation data of each data simulation variable to a data mapping location to obtain a target storage data table, including: acquiring an initial storage data table; determining at least one data mappable position in the initial stored data table according to the data mapping position; and sequentially mapping the simulation data of the data simulation variables to the data mappable positions to obtain a target storage data table.
The initial stored data table may be an original data table in which data mapping is not performed. The data mappable locations may be locations in the initial storage data table where simulation data may be mapped. Specifically, since there may be a plurality of simulation data of each data simulation variable, a plurality of mapping positions identical to the data mapping position format may be sequentially determined downward according to the data mapping position format corresponding to the data simulation variable. And finally mapping the simulation data of the data simulation variable to a data mappable position, thereby obtaining a target storage data table.
S240, obtaining the memory value of the memory table of the target memory data table.
The memory value of the memory table may be the memory value occupied by the target memory data table. Specifically, after the target data table is determined, the memory data of the target storage data target may be analyzed, so as to determine the memory value of the storage table.
S250, disassembling the target storage data table into at least one target storage data sub-table under the condition that the memory value of the storage table is larger than a preset storage threshold value.
The preset storage threshold may be a storage memory threshold of a single distributed storage partition. By setting the preset storage threshold, the memory size of the table stored in a single distributed partition may be limited. Further, the target storage data index may be a partial data table of the target storage data table. Under the condition that the memory value of the memory table is larger than the preset memory threshold, the memory occupation of the target memory data object is excessively large and the target memory data object cannot be stored in one distributed partition, so that the target memory data table can be disassembled into a plurality of target memory data sub-tables, and the memory value of each target memory data sub-table is smaller than the preset memory threshold.
S260, respectively storing the target storage data sub-tables in a data partition library of a preset distributed database.
The preset distributed database can be a preset database which can be stored in a distributed mode. Alternatively, the preset distributed database may be an Apache Cassandra database. Apache Cassandra is an open source distributed and decentralized/distributed storage system (database) that manages large amounts of structured data distributed throughout the world. It provides high availability services without a single point of failure. Has the following characteristics: the single table can store 1000 hundred million data; the single library is within 100 tables; the upper limit of the number of the single table fields is controlled to be 20-60.
The data partition library may be a sub-database of a preset distributed database. After determining the plurality of target storage data sub-tables, each target storage data index may be stored in a corresponding data partition library, respectively. The method and the device can avoid the problem of overlarge memory used in the data partition library while the data table is stored in a distributed mode.
Optionally, storing the target storage data sub-tables in a data partition library of a preset distributed database respectively, including: determining the number of partition libraries according to the number of the target storage data sub-tables; determining at least one partition storage address according to the number of the fixed partition libraries and the table attribute information of the target simulation data table; and storing each target storage data sub-table in the partition storage address respectively.
The number of partition libraries may be the number of data partition libraries for storing the target storage data sub-table. Specifically, the number of partition libraries is the same as the number of target storage data sub-tables. The table attribute information may be related attribute information of the target simulation data table. Specifically, the table data information includes, but is not limited to, generation time information of the table, simulation variable type information, identification information of the single simulation, and the like. The partition storage address may be a storage address of the target storage data sub-table in the data partition library. Specifically, at least one partition storage address may be determined according to the number of the fixed partition libraries and table attribute information of the target simulation data table. Wherein the number of partition memory addresses is the same as the number of partition banks. That is, each target storage data sub-table has a corresponding partition storage address. For example, partition storage addresses associated with the target storage data sub-tables may be determined according to the table attribute information, and then each target storage data sub-table may be stored in a corresponding partition storage address.
The cloud real-time simulation technology based on Apache transfers the fields in the column dimension to the row dimension through a reasonably designed data model, reasonably applies the advantage capability of Cansandra row-direction storage, also supports the scene when the number of Modelica model variables is too large, and enables the disk IO not to be the bottleneck of large-scale simulation result access, so that the real-time storage and reading of massive simulation result data generated by the cloud Modelica model in the simulation process are possible.
Illustratively, the steps for data storage based on the Cansandra technique are as follows:
(1) All variables are stored in an array:
varArr＝[varName1,varName2,…,varName n]
(2) Constructing a data model:
in the model we define v1, v2 … v50, meaning that a row can store the values of rowvarnum=50 variables, a Cansandra list can store a comparably poor number of fields, and 1000 hundred million rows can be stored on a row, and the fields on the column can be transferred to a row by means of varGroup and step fields, where vargroup=variable is at varrrposition/rowVarNum. The actual variables in varArr are mapped to n in the data model (n=the position of the variable in varArr%rowvarnum, if n= 0then n=rowvarnum). The simulation task ID (taskId) and the bucket (bucket) are used as main partition keys to divide the large data partition into small partitions. Ideally, the partition size should not exceed 100MB. By adding a bucket field in the partition key, we can easily control the partition to within 100MB. When acquiring data, we need to access these buckets and aggregate the data together. The daily simulation tasks were kept using a separate table, 30 tables per month.
In the following, let us assume that v1, v2 and v3 total 3 variables are defined in the data model, and one simulation needs to store data of 5 variables and 3 steps of data for illustration:
variable array: vararr= [ var1, var2, var3, var4, var5]
The variable number 5 exceeds 3 defined in the data model, the variables are grouped into 1 group var1, var2, var3 and 2 group var4, var5, and after the data insertion, the following table 2 is shown:
table 2 example result data store
| taskId | bucket | step | varGroup | v1 | v2 | v3 |
| 1 | 1 | 1 | 1 | 0.1 | 0.2 | 0.3 |
| 1 | 1 | 1 | 2 | 0.6 | 0.7 | / |
| 1 | 1 | 2 | 1 | 0.11 | 0.21 | 0.31 |
| 1 | 1 | 2 | 2 | 0.61 | 0.71 | / |
| 1 | 1 | 3 | 1 | 0.1 | 0.22 | 0.3 |
| 1 | 1 | 3 | 2 | 0.6 | 0.73 | / |
Wherein, the taskId represents a simulation identifier; the socket represents a storage partition corresponding to the data; step represents the number of groups of data; varGroup represents the number of rows of the same set of data. As can be seen from table 2, there are 3 sets of data in the table, each set of data includes two rows of data, and the two rows of data in the same set of data are stored in different memory partitions.
According to the technical scheme provided by the embodiment of the invention, the target simulation data table of the Modelica model is obtained, at least one data simulation variable is determined according to the target simulation data table, and each data simulation variable is numbered to obtain a simulation variable number; based on the simulation variable number and a preset row variable threshold, respectively determining the data mapping position of each data simulation variable; mapping the simulation data of each data simulation variable to a data mapping position to obtain a target storage data table; acquiring a memory value of a memory table of a target memory data table; under the condition that the memory value of the memory table is larger than a preset memory threshold value, disassembling the target memory data table into at least one target memory data sub-table; and respectively storing the target storage data sub-tables in a data partition library of a preset distributed database. The technical scheme of the embodiment of the invention solves the problems that in the existing Modelica model simulation data storage technology, single storage occupies more disk IO of a single server, and the efficiency and the instantaneity of data storage are insufficient, and can be used for storing a model simulation data table in a distributed database after format conversion, thereby improving the efficiency and the instantaneity of data storage.
Fig. 3 is a schematic structural diagram of a model simulation data storage device provided by the embodiment of the present invention, where the embodiment of the present invention is applicable to a scenario of storing simulation data generated by a Modelica model, and the device may be implemented by software and/or hardware, and integrated into a computer device having an application development function.
As shown in fig. 3, the model simulation data storage apparatus includes: a simulation data table acquisition module 310, a stored data table generation module 320, and a data table storage module 330.
The simulation data table obtaining module 310 is configured to obtain a target simulation data table of the Modelica model; the storage data table generating module 320 is configured to convert a table attribute format of the target simulation data table to obtain a target storage data table; the data table storage module 330 is configured to store the target storage data table in a preset distributed database.
According to the technical scheme provided by the embodiment of the invention, the target simulation data table of the Modelica model is obtained; converting the table attribute format of the target simulation data table to obtain a target storage data table; and storing the target storage data table in a preset distributed database. The technical scheme of the embodiment of the invention solves the problems that in the existing Modelica model simulation data storage technology, single storage occupies more disk IO of a single server, and the efficiency and the instantaneity of data storage are insufficient, and can be used for storing a model simulation data table in a distributed database after format conversion, thereby improving the efficiency and the instantaneity of data storage.
In an alternative embodiment, the stored data table generating module 320 is specifically configured to: determining at least one data simulation variable according to the target simulation data table, and numbering each data simulation variable to obtain a simulation variable number; based on the simulation variable numbers and a preset row variable threshold value, respectively determining the data mapping position of each data simulation variable; and mapping the simulation data of each data simulation variable to a data mapping position to obtain the target storage data table.
In an alternative embodiment, the stored data table generating module 320 includes: a data mapping position determining unit for: rounding up the preset row variable threshold value by the simulation variable number to obtain a data mapping row position; taking the remainder of the simulation variable number for the preset row variable threshold value to obtain a data mapping column position; and determining the data mapping position according to the data mapping row position and the data mapping column position.
In an alternative embodiment, the stored data table generating module 320 includes: a simulation data mapping unit for: acquiring an initial storage data table; determining at least one data mappable position in the initial stored data table according to the data mapping position; and mapping the simulation data of the data simulation variables to the data mappable positions in sequence to obtain the target storage data table.
In an alternative embodiment, the data table storage module 330 is specifically configured to: acquiring a memory value of a memory table of the target memory data table; under the condition that the memory value of the memory table is larger than a preset memory threshold value, disassembling the target memory data table into at least one target memory data sub-table; and respectively storing the target storage data sub-tables in a data partition library of a preset distributed database.
In an alternative embodiment, the data table storage module 330 includes: the data sub-table storage unit is used for: determining the number of partition libraries according to the number of the target storage data sub-tables; determining at least one partition storage address according to the number of the fixed partition libraries and the table attribute information of the target simulation data table; and storing each target storage data sub-table in the partition storage address respectively.
In an alternative embodiment, the stored data table generating module 320 includes: a simulation variable number determination unit configured to: identifying header data of the target simulation data table, and determining at least one data simulation variable; sequentially mapping the data simulation variables into a preset linear set to obtain a target variable linear set; and respectively determining the simulation variable number of each data simulation variable according to the arrangement sequence of the variables in the target variable linear set.
The model simulation data storage device provided by the embodiment of the invention can execute the model simulation data storage method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention. Fig. 4 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in fig. 4 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention. Computer device 12 may be any terminal device with computing capabilities and may be configured in a model simulation data storage device.
As shown in FIG. 4, the computer device 12 is in the form of a general purpose computing device. Components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, a bus 18 that connects the various system components, including the system memory 28 and the processing units 16.
Bus 18 may be one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Computer device 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. The computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, commonly referred to as a "hard disk drive"). Although not shown in fig. 4, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each drive may be coupled to bus 18 through one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored in, for example, system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.
The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and/or any devices (e.g., network card, modem, etc.) that enable the computer device 12 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 22. Moreover, computer device 12 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 20. As shown in fig. 4, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be appreciated that although not shown in fig. 4, other hardware and/or software modules may be used in connection with computer device 12, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing a model simulation data storage method provided by the present embodiment, the method including:
obtaining a target simulation data table of a Modelica model;
converting the table attribute format of the target simulation data table to obtain a target storage data table;
and storing the target storage data table in a preset distributed database.
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a model simulation data storage method as provided by any embodiment of the present invention, comprising:
obtaining a target simulation data table of a Modelica model;
converting the table attribute format of the target simulation data table to obtain a target storage data table;
and storing the target storage data table in a preset distributed database.
The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
It will be appreciated by those of ordinary skill in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be centralized on a single computing device, or distributed over a network of computing devices, or they may alternatively be implemented in program code executable by a computer device, such that they are stored in a memory device and executed by the computing device, or they may be separately fabricated as individual integrated circuit modules, or multiple modules or steps within them may be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (10)
1. A method for storing model simulation data, comprising:
obtaining a target simulation data table of a Modelica model;
converting the table attribute format of the target simulation data table to obtain a target storage data table;
and storing the target storage data table in a preset distributed database.
2. The method of claim 1, wherein converting the table attribute format of the target simulation data table to obtain a target storage data table comprises:
determining at least one data simulation variable according to the target simulation data table, and numbering each data simulation variable to obtain a simulation variable number;
based on the simulation variable numbers and a preset row variable threshold value, respectively determining the data mapping position of each data simulation variable;
and mapping the simulation data of each data simulation variable to a data mapping position to obtain the target storage data table.
3. The method of claim 2, wherein the determining the data mapping location of each data simulation variable based on the simulation variable number and a preset row variable threshold, respectively, comprises:
rounding up the preset row variable threshold value by the simulation variable number to obtain a data mapping row position;
taking the remainder of the simulation variable number for the preset row variable threshold value to obtain a data mapping column position;
and determining the data mapping position according to the data mapping row position and the data mapping column position.
4. The method of claim 2, wherein mapping the simulation data of each data simulation variable to a data mapping location results in the target storage data table, comprising:
acquiring an initial storage data table;
determining at least one data mappable position in the initial stored data table according to the data mapping position;
and mapping the simulation data of the data simulation variables to the data mappable positions in sequence to obtain the target storage data table.
5. The method of claim 1, wherein storing the target storage data table in a preset distributed database comprises:
acquiring a memory value of a memory table of the target memory data table;
under the condition that the memory value of the memory table is larger than a preset memory threshold value, disassembling the target memory data table into at least one target memory data sub-table;
and respectively storing the target storage data sub-tables in a data partition library of a preset distributed database.
6. The method according to claim 5, wherein storing the target storage data sub-tables in the data partition library of the preset distributed database, respectively, comprises:
determining the number of partition libraries according to the number of the target storage data sub-tables;
determining at least one partition storage address according to the number of the fixed partition libraries and the table attribute information of the target simulation data table;
and storing each target storage data sub-table in the partition storage address respectively.
7. The method of claim 2, wherein determining at least one data simulation variable from the target simulation data table and numbering each data simulation variable to obtain a simulation variable number, comprises:
identifying header data of the target simulation data table, and determining at least one data simulation variable;
sequentially mapping the data simulation variables into a preset linear set to obtain a target variable linear set;
and respectively determining the simulation variable number of each data simulation variable according to the arrangement sequence of the variables in the target variable linear set.
8. A model simulation data storage apparatus, the apparatus comprising:
the simulation data table acquisition module is used for acquiring a target simulation data table of the Modelica model;
the storage data table generation module is used for converting the table attribute format of the target simulation data table to obtain a target storage data table;
and the data table storage module is used for storing the target storage data table in a preset distributed database.
9. A computer device, the computer device comprising:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the model simulation data storage method of any of claims 1-7.
10. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a model simulation data storage method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311842462.3A CN117891400A (en) | 2023-12-28 | 2023-12-28 | Model simulation data storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311842462.3A CN117891400A (en) | 2023-12-28 | 2023-12-28 | Model simulation data storage method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117891400A true CN117891400A (en) | 2024-04-16 |
Family
ID=90650017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311842462.3A Pending CN117891400A (en) | 2023-12-28 | 2023-12-28 | Model simulation data storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117891400A (en) |
-
2023
- 2023-12-28 CN CN202311842462.3A patent/CN117891400A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110515739B (en) | Deep learning neural network model load calculation method, device, equipment and medium | |
| CN111090628B (en) | Data processing method and device, storage medium and electronic equipment | |
| CN110297689B (en) | Intelligent contract execution method, device, equipment and medium | |
| CN111324610A (en) | Data synchronization method and device | |
| CN109522228A (en) | Interface automatic test data configuration method, apparatus, platform and storage medium | |
| CN111143446A (en) | Data structure conversion processing method and device of data object and electronic equipment | |
| CN105706092A (en) | Methods and systems of four-valued simulation | |
| CN111324441A (en) | Operating environment switching method and device, computer equipment and storage medium | |
| CN107066205B (en) | Data storage system | |
| CN105843899B (en) | A kind of big data automation analytic method for simplifying programming and system | |
| CN111611622A (en) | Block chain-based file storage method and electronic equipment | |
| CN109284108A (en) | Date storage method, device, electronic equipment and storage medium | |
| CN116244387A (en) | Entity relationship construction method, device, electronic equipment and storage medium | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN117891400A (en) | Model simulation data storage method, device, equipment and storage medium | |
| CN116089527A (en) | Data verification method, storage medium and device | |
| CN113806556A (en) | Method, device, equipment and medium for constructing knowledge graph based on power grid data | |
| CN114020789A (en) | Bill data query method, system, device and storage medium | |
| CN113887927A (en) | Hotel order data processing method, equipment and medium | |
| CN111522832A (en) | Big data real-time updating method, system, equipment and storage medium | |
| JP4671041B2 (en) | Modular physical resource group identification method, apparatus and program thereof | |
| CN117592311B (en) | Multi-level simulation method, device and equipment for workflow and readable medium | |
| CN116470968B (en) | Ground test method and device for communication function of aerospace science system | |
| CN116451005B (en) | Spark-based distributed grid algebra operation method, system and equipment | |
| CN117608526A (en) | Data simulation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |