WO2022198688A1 - 一种非实时数据传递方法 - Google Patents
一种非实时数据传递方法 Download PDFInfo
- Publication number
- WO2022198688A1 WO2022198688A1 PCT/CN2021/083747 CN2021083747W WO2022198688A1 WO 2022198688 A1 WO2022198688 A1 WO 2022198688A1 CN 2021083747 W CN2021083747 W CN 2021083747W WO 2022198688 A1 WO2022198688 A1 WO 2022198688A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- storage medium
- target
- same
- data
- destination
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000005540 biological transmission Effects 0.000 title abstract description 11
- 238000009434 installation Methods 0.000 claims abstract description 21
- 238000012546 transfer Methods 0.000 claims description 5
- 230000001360 synchronised effect Effects 0.000 claims description 2
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
Definitions
- the present invention relates to data transmission technology, in particular to a non-real-time data transmission method.
- the purpose of the present invention is to solve the problem of safe and efficient transfer of non-real-time massive data into and out of various data center data warehouses.
- an embodiment of the present invention provides a non-real-time data transmission method, including:
- order information includes a source data warehouse address, a target data warehouse address and data file information, and the source data warehouse address points to the source data warehouse;
- the data indicated by the data file information is synchronized to the target storage medium, and the target storage medium is transported to the target data warehouse address.
- determine the target storage medium including:
- the address information determine whether there is a same-destination storage medium with the same address information as the target data warehouse address;
- the target storage medium can be determined from the same-destination storage medium, determining the target storage medium from the same-destination storage medium;
- a new storage medium is initialized and mounted from the source data bin as the target storage medium, and the storage space of the new storage medium is not occupied.
- the method further includes:
- order information corresponding to the target storage medium where the order information includes an order number
- the data corresponding to each order information is correspondingly stored in the waybill directory.
- the judging whether the target storage medium can be determined from the same-destination storage medium according to the data file information and availability including:
- determining the target storage medium from the preselected storage medium may determine the target storage medium from the same-destination storage medium.
- the judging whether there is a preselected storage medium that meets the shipping safety condition and the shipping time limit condition in the same-destination storage medium includes:
- the transport security condition of the same-destination storage medium is higher than the transport security condition determined in the data file information, it is determined that a preselected storage medium exists in the same-destination storage medium.
- the judging whether there is a preselected storage medium that meets the shipping safety condition and the shipping time limit condition in the same-destination storage medium includes:
- the delivery time limit condition of the same-destination storage medium is stricter than the delivery time limit condition determined in the data file information, it is determined that a preselected storage medium exists in the same-destination storage medium.
- a new storage medium is initialized from the source data bin to be mounted as the target storage medium.
- a new storage medium is initialized from the source data bin to be mounted as a target storage medium.
- the method further includes:
- FIG. 1 is a flowchart of a non-real-time data transmission method provided by a specific embodiment of the present invention
- FIG. 2 is a flowchart of a method for determining a target storage medium provided by a specific embodiment of the present invention
- FIG. 3 is a flowchart of a method for judging whether a target storage medium can be determined from the same-destination storage medium according to data file information and availability according to a specific embodiment of the present invention.
- FIG. 1 is a flowchart of a non-real-time data transmission method provided by a specific embodiment of the present invention. 1, the present invention includes the following steps:
- the order information is the information about the generation of the digital media delivery waybill by placing an order on the mobile phone/website of the digital media delivery service system (client/large client), generally including the source warehouse address, target warehouse address and data file information, wherein , the source data warehouse address points to the source data warehouse, the target data warehouse address points to the destination data warehouse, and the data file information includes the size of the data to be transmitted, the transport security conditions, and the transport time limit conditions.
- the order information also includes: source warehouse contact person, unit, and mobile phone number, and destination warehouse contact person, unit, and mobile phone number.
- the data content description file needs to describe the directory structure, file recording format, and data item details of the data carried by the medium so that the receiver can analyze and store it; encrypted data files need to transmit decryption information through a secure channel.
- the encrypted data file is to prevent data leakage
- the data digest is to prevent tampering.
- the encryption method includes ordinary point-to-point encryption and decryption, and can also include end-to-end data encryption and tamper-proofing through blockchain. It is recommended to use a cipher machine that meets the requirements of national secrets to generate the key. If there are no implementation conditions or the confidentiality requirements are not high, low-level methods such as pseudo-random passwords can also be generated by computer algorithms.
- S20 Determine the target storage medium according to the availability of the storage medium in each storage medium installation position of the source data bin and the address of the target data bin.
- the source data warehouse server (which needs to support a distributed file system, such as HDFS) configures the storage backup strategy according to the waybills that need to be processed on the day, and automatically synchronizes to the storage devices (such as mounted hard disks, backup each other).
- a distributed file system such as HDFS
- HDFS has the characteristics of a high-availability distributed file system. It is designed to be deployed on low-cost hardware, supports a variety of storage backup strategies, and is suitable for large OLAP online analysis data sets (large data set).
- the source data bin includes a storage medium of a plurality of storage medium mounting positions.
- the source data warehouse manages the storage medium of each storage medium installation position through a form. Since the transmission of the storage device between the source data warehouse and the destination data warehouse is not real-time, it is possible that when the source data warehouse receives the order information, it may find that the target data warehouse address of this order information is the same as the target data warehouse address of a previous order information. The warehouse address is the same. At this time, it can be considered to store the data indicated by the order information with the same address of the two target warehouses in the same storage device. That is, a target storage medium that satisfies the requirements of the data file information is selected from the storage media of multiple storage medium installation positions.
- step S20 may further include the following steps:
- the address information includes two types, the address information is empty and the address information is not empty. This traversal is mainly to filter out the storage medium whose address information is not empty.
- step S22 Determine, according to the address information, whether there is a same-destination storage medium whose address information is the same as the address of the target data warehouse. If there is, go to step S23; if not, go to step S25.
- step S23 Determine whether the target storage medium can be determined from the storage medium with the same destination according to the data file information and availability. If yes, go to step S24; if not, go to step S25.
- the same destination storage medium is filtered again to determine the only target storage medium.
- step S23 may further include the following steps:
- Shipping safety conditions include safety protection levels.
- Shipping time limit conditions include the latest time limit for shipping to the destination warehouse address.
- step S232 Determine whether there is a preselected storage medium that meets the transport safety condition and the transport time limit condition in the storage medium at the same destination. If yes, go to step S223; if not, go to step S235.
- step S233 Determine whether the availability of the preselected storage medium satisfies the data size. If satisfied, go to step S224; if not, go to step S235.
- S234 Determine a target storage medium from the preselected storage medium.
- S235 Initialize a new storage medium mount from the source data bin as a target storage medium.
- Storing data destined for the same target data warehouse address in the same target storage medium can not only save the number of storage devices occupied, but also save the cost of transportation.
- order information corresponding to the target storage medium can be determined, and the order information includes an order number; a waybill is created with the waybill number as the name. Directory; correspondingly store the data corresponding to each order information in the waybill directory.
- the method further includes: updating the availability of the storage medium stored in each storage medium installation position of the source data warehouse.
- the source data warehouse server (which needs to support a distributed file system, such as HDFS) configures the storage backup strategy according to the waybills that need to be processed on the day, and automatically synchronizes to the storage devices ( For example, the mounted storage media are mutually backup).
- a distributed file system such as HDFS
- the method further includes: hot-pulling the target storage medium; Storage medium; update the availability of storage medium in each storage medium installation position of the source data warehouse.
- Hot-pull the corresponding storage device according to the installation position of the storage medium that needs to be delivered on the day, put it into the data media delivery box, and lock it; the courier will generate the waybill label with the terminal, paste it on the delivery box, and send the dynamic password to the recipient (existing).
- real-time tracking can be carried out according to the location information of the data media delivery box during the delivery.
- the data media delivery box is delivered to the destination warehouse, the client/large client user reports the password to the courier, the courier unlocks the lock and takes out the storage device, and hot-plugs it to the server in the destination warehouse exchange area; automatically loads the data files to the destination warehouse file system , and perform data backup operations through ETL tools or file system data backup strategies to realize data storage.
- the data media delivery box has the characteristics of shockproof, anti-pressure, waterproof, fire-proof, anti-theft, lightning-proof, anti-magnetic, and traceable.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Development Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
一种非实时数据传递方法。该方法包括：接收订单信息（S10），订单信息包括源数仓地址、目标数仓地址和数据文件信息，源数仓地址指向源数仓；根据源数仓各个存储介质安装位的存储介质的可用情况和目标数仓地址，确定目标存储介质（S20），目标存储介质满足数据文件信息的需求；将数据文件信息指示的数据同步到目标存储介质，并将目标存储介质运输到目标数仓地址（S30）。该方法能够实现安全高效的非实时海量数据交换传递。
Description
本发明涉及数据传输技术，尤其涉及一种非实时数据传递方法。
从国家治理的角度，需要汇集各地方各部门各类经营过程数据，之后在国家平台、部委平台进行大数据事后分析，发现问题线索，向各级政府部门和企业提供数据产品服务。行业管理很多情况下并不需要实时数据远距离传递的支持，代价太高，也没有实质性作用。目前国家各行业各地方政务大数据中心基本上都存在一系列问题，包括能耗过高、利用率低等问题，这些问题和数据传递效率低也有直接关系。非实时海量数据跨省跨国跨星全域、接近无限容量、安全高效传递出入各类数据中心数据仓库的问题急需解决。
发明内容
本发明的目的在于解决非实时海量数据安全高效传递出入各类数据中心数据仓库的问题。
为解决上述问题，本发明实施例提供了一种非实时数据传递方法，包括：
接收订单信息，所述订单信息包括源数仓地址、目标数仓地址和数据文件信息，所述源数仓地址指向源数仓；
根据所述源数仓各个存储介质安装位的存储介质的可用情况和目标数仓地址，确定目标存储介质，所述目标存储介质满足所述数据文件信息的需求；
将所述数据文件信息指示的数据同步到所述目标存储介质，并将所述目标存储介质运输到目标数仓地址。
可选的，根据所述源数仓各个存储介质安装位的存储介质的可用 情况，确定目标存储介质，包括：
遍历所述源数仓各个存储介质安装位的存储介质关联的地址信息；
根据所述地址信息判断是否存在地址信息与所述目标数仓地址相同的同目的地存储介质；
若存在所述同目的地存储介质，则判断能否根据数据文件信息和可用情况从所述同目的地存储介质中确定目标存储介质；
若可以从所述同目的地存储介质中确定目标存储介质，则从所述同目的地存储介质中确定目标存储介质；
若无法从所述同目的地存储介质中确定目标存储介质，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质，所述新的存储介质的存储空间未被占用。
可选的，所述若可以从所述同目的地存储介质中确定目标存储介质，则从所述同目的地存储介质中确定目标存储介质之后，还包括：
确定所述目标存储介质对应的订单信息，所述订单信息包括订单号；
以所述运单号为名称建立运单目录；
将每个订单信息对应的数据对应的存储在所述运单目录下。
可选的，还包括：
若不存在所述同目的地存储介质，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
可选的，所述判断能否根据数据文件信息和可用情况从所述同目的地存储介质中确定目标存储介质，包括：
从数据文件信息中确定运送安全条件、运送时限条件和数据大小；
判断所述同目的地存储介质中是否有符合所述运送安全条件和运送时限条件的预选存储介质；
若存在预选存储介质，则判断所述预选存储介质的可用情况是否满足所述数据大小；
若满足所述数据大小，则从所述预选存储介质中确定目标存储介质可以从同目的地存储介质中确定目标存储介质。
可选的，所述判断所述同目的地存储介质中是否有符合所述运送安全条件和运送时限条件的预选存储介质，包括：
确定所述同目的地存储介质的运送安全条件；
若所述同目的地存储介质的运送安全条件高于所述数据文件信息中确定的运送安全条件，则确定所述同目的地存储介质中存在预选存储介质。
可选的，所述判断所述同目的地存储介质中是否有符合所述运送安全条件和运送时限条件的预选存储介质，包括：
确定所述同目的地存储介质的运送时限条件；
若所述同目的地存储介质的运送时限条件严格于所述数据文件信息中确定的运送时限条件，则确定所述同目的地存储介质中存在预选存储介质。
可选的，还包括：
若不存在预选存储介质，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
可选的，还包括：
若所述预选存储介质的可用情况不满足所述数据大小，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
可选的，所述将所述数据文件信息指示的数据同步到所述目标存储介质之后，还包括：
热拔所述目标存储介质；
在所述目标存储介质对应的存储介质安装位上安插备用的存储介质安装位的存储介质；
更新源数仓各个存储介质安装位的存储介质的可用情况。
通过本发明提供的方案，从技术上实现几乎无限带宽、数仓机架存储介质安装位到数仓机架存储介质安装位、跨省跨国、城乡全覆盖、分级分类、安全高效的非实时海量数据交换传递。
图1为本发明具体实施方式提供的一种非实时数据传递方法的流程图；
图2为本发明具体实施方式提供的确定目标存储介质的方法的流程图；
图3为本发明具体实施方式提供的判断能否根据数据文件信息和可用情况从同目的地存储介质中确定目标存储介质的方法的流程图。
下面结合附图和实施例对本发明作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本发明，而非对本发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本发明相关的部分而非全部结构。
图1为本发明具体实施方式提供的一种非实时数据传递方法的流程图。参考图1，本发明包括如下步骤：
S10、接收订单信息。
所述订单信息是通过数字媒质专递服务系统(用户端/大客户端)手机/网站下单，生成数字媒质专递运单的信息，一般包括源数仓地址、目标数仓地址和数据文件信息，其中，源数仓地址指向源数仓，目标数仓地址指向目的数仓，数据文件信息包括需要传输的数据的大小、运送安全条件、运送时限条件等。
当然，在实际使用中，订单信息还包括：源数仓联系人、单位、手机号，目的数仓联系人、单位、手机号。
在此基础上，数据内容说明文件需要对媒质所承载的数据的目录结构、文件记录格式、数据项详细情况说明以便接收方解析入库；加密数据文件需从保密途径传送解密信息。
其中，加密数据文件是为了防止数据泄漏，数据摘要是为了防篡改。加密方式包括普通点到点加密解密，也可包括通过区块链实现端到端的数据加密防篡改。密钥推荐使用符合国密要求的密码机生成，如无实施条件或保密要求不高也可以通过计算机算法产生伪随机密码等低密级方法。
S20、根据所述源数仓各个存储介质安装位的存储介质的可用情况和目标数仓地址，确定目标存储介质。
源数仓服务器(需支持分布式文件系统，如HDFS)根据当天需处理运单情况配置存储备份策略，通过配置同步工具自动同步到源数仓交换区数个指定存储介质安装位中的存储设备(如挂载的硬盘，互为备份)。
其中，HDFS有着高可用性分布式文件系统的特点，设计用来部署在低廉的(low-cost)硬件上，支持多种存储备份策略，适合超大OLAP在线分析数据集(large data set)。
交换至远端的数据修改删除支持说明：传统的数据库日志打开后，在定期(如一周一次)全量备份数据库并交换的基础上可短期(如每天)增量备份新日志记录以支持增量数据交换；传统数据库日志一般包括增删改等DDL/DML操作记录，定期交换增量数据库日志即可修改删除远端库中数据。数仓hive/hdfs主要用于存储大量过程监管数据，而这些数据主要用于系统留痕事后分析处理，一般没有删改需求。可通过加密、摘要等安全机制实现交换数据的一致性和保密要求。
源数仓包括多个存储介质安装位的存储介质。一般的，源数仓通过表单对每个存储介质安装位的存储介质进行管理。由于源数仓与目的数仓之间的存储设备的传递不是实时的，因此可能存在源数仓接收 到订单信息时，发现这一订单信息的目标数仓地址与之前某一订单信息的目标数仓地址一致。此时可以考虑将两个目标数仓地址一致的订单信息指示的数据存储在相同的存储设备中。也就是从多个存储介质安装位的存储介质中选择满足数据文件信息的需求的目标存储介质。
参考图2，步骤S20还可以包括如下的步骤：
S21、遍历所述源数仓各个存储介质安装位的存储介质关联的地址信息。
遍历源数仓各个存储介质安装位的存储介质关联的地址信息，一般的，地址信息包括两类，地址信息为空和地址信息不为空。此次遍历主要是筛选出地址信息不为空的存储介质安装位的存储介质。
S22、根据地址信息判断是否存在地址信息与目标数仓地址相同的同目的地存储介质。若存在，则执行步骤S23；若不存在，则执行步骤S25。
从地址信息不为空的存储介质安装位的存储介质中判断是否有地址信息与目标数仓地址相同的存储介质安装位的存储介质。如果有，则将其确定为同目的地存储介质。并判断能否根据数据文件信息和可用情况从同目的地存储介质中确定目标存储介质。如果没有，从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
S23、判断能否根据数据文件信息和可用情况从同目的地存储介质中确定目标存储介质。若可以则执行步骤S24；若不能，则执行步骤S25。
根据文件信息和可用情况对同目的地存储介质再次进行筛选，确定唯一的目标存储介质。
参考图3，步骤S23还可以包括如下的步骤：
S231、从数据文件信息中确定运送安全条件、运送时限条件和数据大小。
运送安全条件包括安全保护等级。运送时限条件包括最迟运送到 目标数仓地址的时间限制。
S232、判断同目的地存储介质中是否有符合运送安全条件和运送时限条件的预选存储介质。若有，则执行步骤S223；若没有，则执行步骤S235。
从运送安全条件来看：
确定所述同目的地存储介质的运送安全条件；若所述同目的地存储介质的运送安全条件高于所述数据文件信息中确定的运送安全条件，定所述同目的地存储介质中存在预选存储介质。
从运送时限条件来看：
确定所述同目的地存储介质的运送时限条件；若所述同目的地存储介质的运送时限条件严格于所述数据文件信息中确定的运送时限条件，定所述同目的地存储介质中存在预选存储介质。
S233、判断所述预选存储介质的可用情况是否满足所述数据大小。若满足，则执行步骤S224；若不满足，则执行步骤S235。
S234、则从所述预选存储介质中确定目标存储介质。
当然，同目的地存储介质中可能存在多个符合条件的存储介质。可以从这些符合条件的存储介质中选择剩余空间最小的存储介质架作为目标存储介质。
S235、从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
S24、从所述同目的地存储介质中确定目标存储介质。
将去往同一目标数仓地址的数据存储到同一目标存储介质中，既可以节约占用存储设备的数量，也可以节约运输的成本。
此时存在一个目标存储介质中有多个订单信息的情况，为了便于区分，可以确定所述目标存储介质对应的订单信息，所述订单信息包括订单号；以所述运单号为名称建立运单目录；将每个订单信息对应的数据对应的存储在所述运单目录下。
S25、从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
S30、将所述数据文件信息指示的数据同步到所述目标存储介质，并将所述目标存储介质运输到目标数仓地址。
将所述数据文件信息指示的数据同步到所述目标存储介质之后，还包括：更新源数仓各个存储介质安装位存的储介质的可用情况。
源数仓服务器(需支持分布式文件系统，如HDFS)根据当天需处理运单情况配置存储备份策略，通过配置同步工具自动同步到源数仓交换区数个指定存储介质安装位中的存储设备(如挂载的存储介质，互为备份)。
将所述数据文件信息指示的数据同步到所述目标存储介质之后，还包括：热拔所述目标存储介质；在所述目标存储介质对应的存储介质安装位上安插备用的存储介质安装位的存储介质；更新源数仓各个存储介质安装位的存储介质的可用情况。
根据当天到期需运送的存储介质安装位热拔对应存储设备，放入数据媒质专递箱，上锁；快递员手持终端生成运单标签，贴在专递箱上，动态密码发给收件人(存在于订单信息中)，运送途中可根据数据媒质专递箱位置信息进行实时跟踪。数据媒质专递箱运送到目的数仓，用户端/大客户端用户向快递员报密码，快递员开锁取出存储设备，热插上目的数仓交换区服务器；自动加载数据文件至目的数仓文件系统，通过ETL工具或文件系统数据备份策略进行数据备份操作，实现数据入库。
在此基础上，数据媒质专递箱具备防震、防压、防水、防火、防盗、防雷电、防磁、可跟踪等产品特性。
通过本发明提供的方案，从技术上实现几乎无限带宽、数仓机架存储介质安装位到数仓机架存储介质安装位、跨省跨国、城乡全覆盖、分级分类、安全高效的非实时海量数据交换传递。
虽然，上文中已经用一般性说明、具体实施方式及试验，对本发明作了详尽的描述，但在本发明基础上，可以对之作一些修改或改进，这对本领域技术人员而言是显而易见的。因此，在不偏离本发明精神的基础上所做的这些修改或改进，均属于本发明要求保护的范围。
Claims (10)
- 一种非实时数据传递方法，其特征在于，包括：接收订单信息，所述订单信息包括源数仓地址、目标数仓地址和数据文件信息，所述源数仓地址指向源数仓；根据所述源数仓各个存储介质安装位的存储介质的可用情况和目标数仓地址，确定目标存储介质，所述目标存储介质满足所述数据文件信息的需求；将所述数据文件信息指示的数据同步到所述目标存储介质，并将所述目标存储介质运输到目标数仓地址。
- 根据权利要求1所述的方法，其特征在于，根据所述源数仓各个存储介质安装位的存储介质的可用情况，确定目标存储介质，包括：遍历所述源数仓各个存储介质安装位的存储介质关联的地址信息；根据所述地址信息判断是否存在地址信息与所述目标数仓地址相同的同目的地存储介质；若存在所述同目的地存储介质，则判断能否根据数据文件信息和可用情况从所述同目的地存储介质中确定目标存储介质；若可以从所述同目的地存储介质中确定目标存储介质，则从所述同目的地存储介质中确定目标存储介质；若无法从所述同目的地存储介质中确定目标存储介质，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质，所述新的存储介质的存储空间未被占用。
- 根据权利要求2所述的方法，其特征在于，所述若可以从所述同目的地存储介质中确定目标存储介质，则从所述同目的地存储介质中确定目标存储介质之后，还包括：确定所述目标存储介质对应的订单信息，所述订单信息包括订单号；以所述运单号为名称建立运单目录；将每个订单信息对应的数据对应的存储在所述运单目录下。
- 根据权利要求2所述的方法，其特征在于，还包括：若不存在所述同目的地存储介质，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
- 根据权利要求2所述的方法，其特征在于，所述判断能否根据数据文件信息和可用情况从所述同目的地存储介质中确定目标存储介质，包括：从数据文件信息中确定运送安全条件、运送时限条件和数据大小；判断所述同目的地存储介质中是否有符合所述运送安全条件和运送时限条件的预选存储介质；若存在预选存储介质，则判断所述预选存储介质的可用情况是否满足所述数据大小；若满足所述数据大小，则从所述预选存储介质中确定目标存储介质可以从同目的地存储介质中确定目标存储介质。
- 根据权利要求5所述的方法，其特征在于，所述判断所述同目的地存储介质中是否有符合所述运送安全条件和运送时限条件的预选存储介质，包括：确定所述同目的地存储介质的运送安全条件；若所述同目的地存储介质的运送安全条件高于所述数据文件信息中确定的运送安全条件，则确定所述同目的地存储介质中存在预选存储介质。
- 根据权利要求6所述的方法，其特征在于，所述判断所述同目的地存储介质中是否有符合所述运送安全条件和运送时限条件的预选存储介质，包括：确定所述同目的地存储介质的运送时限条件；若所述同目的地存储介质的运送时限条件严格于所述数据文件信息中确定的运送时限条件，则确定所述同目的地存储介质中存在预选存储介质。
- 根据权利要求5所述的方法，其特征在于，还包括：若不存在预选存储介质，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
- 根据权利要求5所述的方法，其特征在于，还包括：若所述预选存储介质的可用情况不满足所述数据大小，则从所述源数仓中初始化一个新的存储介质挂载作为目标存储介质。
- 根据权利要求1所述的方法，其特征在于，所述将所述数据文件信息指示的数据同步到所述目标存储介质之后，还包括：热拔所述目标存储介质；在所述目标存储介质对应的存储介质安装位上安插备用的存储介质安装位的存储介质；更新源数仓各个存储介质安装位的存储介质的可用情况。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110309606.3A CN113032472A (zh) | 2021-03-23 | 2021-03-23 | 一种非实时数据传递方法 |
| CN202110309606.3 | 2021-03-23 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022198688A1 true WO2022198688A1 (zh) | 2022-09-29 |
Family
ID=76472897
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2021/083747 WO2022198688A1 (zh) | 2021-03-23 | 2021-03-30 | 一种非实时数据传递方法 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113032472A (zh) |
| WO (1) | WO2022198688A1 (zh) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978444A (zh) * | 2019-02-22 | 2019-07-05 | 东莞市大易产业链服务有限公司 | 仓储调度方法及系统、存储介质、电子设备 |
| US10395060B2 (en) * | 2016-10-17 | 2019-08-27 | Microsoft Technology Licensing, Llc | Multiple message retrieval for secure electronic communication |
| CN111241203A (zh) * | 2020-02-10 | 2020-06-05 | 江苏满运软件科技有限公司 | Hive数据仓库同步方法、系统、设备及存储介质 |
| CN111756818A (zh) * | 2020-06-05 | 2020-10-09 | 腾讯科技（深圳）有限公司 | 一种文件传送方法、装置、设备及存储介质 |
-
2021
- 2021-03-23 CN CN202110309606.3A patent/CN113032472A/zh active Pending
- 2021-03-30 WO PCT/CN2021/083747 patent/WO2022198688A1/zh active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10395060B2 (en) * | 2016-10-17 | 2019-08-27 | Microsoft Technology Licensing, Llc | Multiple message retrieval for secure electronic communication |
| CN109978444A (zh) * | 2019-02-22 | 2019-07-05 | 东莞市大易产业链服务有限公司 | 仓储调度方法及系统、存储介质、电子设备 |
| CN111241203A (zh) * | 2020-02-10 | 2020-06-05 | 江苏满运软件科技有限公司 | Hive数据仓库同步方法、系统、设备及存储介质 |
| CN111756818A (zh) * | 2020-06-05 | 2020-10-09 | 腾讯科技（深圳）有限公司 | 一种文件传送方法、装置、设备及存储介质 |
Non-Patent Citations (1)
| Title |
|---|
| "A Study of Haizi's Poetry", 31 December 2009, CN, ISBN: 9787530652282, article YOU, ZAI: "Facing the Sea With Spring Blossoms", pages: 562 - 566, XP009540980 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113032472A (zh) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8171065B2 (en) | Relational objects for the optimized management of fixed-content storage systems | |
| JP6479020B2 (ja) | 分散ストレージシステムにおけるオブジェクトの階層チャンキング | |
| CN102411637B (zh) | 分布式文件系统的元数据管理方法 | |
| CN105554070A (zh) | 一种基于警务大数据中心服务建设的方法 | |
| CN105678189A (zh) | 加密数据文件存储和检索系统及方法 | |
| CN109145176A (zh) | 电子文档管理系统 | |
| CN101398823B (zh) | 利用虚拟文件系统技术实现远程存储的方法及系统 | |
| CN107391306A (zh) | 一种异构数据库备份文件恢复方法 | |
| CN104156278A (zh) | 一种文件版本控制系统及其方法 | |
| CN102737127A (zh) | 一种海量数据存储方法 | |
| CN104965845A (zh) | 一种小文件的定位方法及系统 | |
| CN102012933A (zh) | 分布式文件系统及利用其存储数据和提供服务的方法 | |
| CN108197260A (zh) | 一种文档管理系统 | |
| CN102111285A (zh) | 一种日志信息管理方法及系统 | |
| CN109451004A (zh) | 基于云网桥的云存储系统及方法 | |
| CN102779138A (zh) | 实时数据的硬盘存取方法 | |
| US20180165295A1 (en) | Systems and methods for document search and aggregation with reduced bandwidth and storage demand | |
| CN110750372A (zh) | 基于共享内存的日志系统及日志管理方法 | |
| CN103823807A (zh) | 一种去除重复数据的方法、装置及系统 | |
| CN101827239A (zh) | 海量移动流媒体图像、视频数据存储与跨域资源访问 | |
| WO2022198688A1 (zh) | 一种非实时数据传递方法 | |
| CN105678190A (zh) | 数据封存审计系统 | |
| CN103530203A (zh) | 遥感卫星数据的异地备份系统及方法 | |
| CN103778231A (zh) | 一种数据银行管理操作记录信息的方法及系统 | |
| CN116401324A (zh) | 面向锂电池制造业的实时数仓方法及系统 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21932306 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21932306 Country of ref document: EP Kind code of ref document: A1 |