WO2020077563A1

WO2020077563A1 - Data storage method and apparatus, and electronic device and computer readable storage medium

Info

Publication number: WO2020077563A1
Application number: PCT/CN2018/110652
Authority: WO
Inventors: 郭昌飞; 白涛; 尚俊宏
Original assignee: 北京比特大陆科技有限公司
Priority date: 2018-10-17
Filing date: 2018-10-17
Publication date: 2020-04-23
Also published as: CN109564567B; CN109564567A

Abstract

A data storage method and apparatus, and an electronic device and a computer readable storage medium. The method comprises: obtaining a data packet to be stored (S101), wherein said data packet comprises data to be stored, and data source information and target database information of said data; determining a storage rule for said data according to said data source information and target database information (S102), wherein the storage rule for said data comprises a storage data source sub-rule for said data and a storage database sub-rule for said data; and storing said data according to said storage rule (S103). The method can achieve effective storage of data and improve data quality, thereby increasing the utilization rate of storage resources and saving storage resources.

Description

Data storage method, device, electronic equipment and computer readable storage medium

Technical field

Embodiments of the present invention relate to the field of data storage technology, and in particular, to a data storage method, device, electronic device, and computer-readable storage medium.

Background technique

With the development of data technology and the progress of society, the public has higher and higher requirements for security systems, and the amount of monitoring data that needs to be stored is also increasing. How to distinguish effective data from it and how to carry out effective data Effective storage to improve the utilization rate of storage resources and save storage resources are urgent problems to be solved in the industry.

Summary of the invention

Embodiments of the present invention provide a data storage method, device, electronic device, and computer-readable storage medium.

In a first aspect, an embodiment of the present invention provides a data storage method.

Specifically, the data storage method includes:

Obtaining a data package to be stored, wherein the data package to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

Determine the data storage rule to be stored according to the data source information of the data to be stored and the target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of the data source to be stored and a sub-rule of the data storage database ;

Storing the data to be stored according to the data storage rule to be stored.

With reference to the first aspect, in a first implementation manner of the first aspect of the embodiments of the present invention, the determining the data storage rule to be stored according to the data source information and the data target database information of the data to be stored includes:

Determine the data source category of the data to be stored according to the data source information of the data to be stored;

Determining the sub-rule of the data source of the data to be stored according to the type of the data source of the data to be stored;

Determine the category of the target database of data to be stored according to the information of the target database of data to be stored;

The sub-rule of the data storage database to be stored is determined according to the category of the target database of the data to be stored.

With reference to the first aspect and the first implementation manner of the first aspect, in a second implementation manner of the first aspect of the embodiments of the present invention, the storing the data to be stored according to the data storage rule to be stored, include:

Determine whether the data to be stored conforms to the data storage rule to be stored;

Storing the data to be stored in accordance with the data storage rule to be stored in the target database.

With reference to the first aspect, the first implementation manner of the first aspect, and the second implementation manner of the first aspect, in a third implementation manner of the first aspect of the present disclosure, the data source sub-rule for storing data to be stored includes: One or more of the following rules: data quality meets preset quality conditions, storage time interval meets preset interval conditions; and / or,

The sub-rules of the data storage database to be stored include one or more of the following rules: data similarity meets a preset similarity condition, data size meets a preset size condition, and video storage condition.

Combining the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, and the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect, the present disclosure The storing the data to be stored according to the data storage rule to be stored includes:

Judging whether the quality of the data to be stored meets the preset quality condition, and discarding the data to be stored when the quality of the data to be stored does not meet the preset quality condition;

When the quality of the data to be stored satisfies the preset quality condition, determine whether the similarity between the data to be stored and the target database satisfies the preset similarity condition, and when the data to be stored When the similarity with the target database does not satisfy the preset similarity condition, create a new database, and store the data to be stored in the newly created database;

When the similarity between the data to be stored and the target database satisfies the preset similarity condition, it is determined whether the storage time interval of the data to be stored meets the preset interval condition, and when the pending When the storage time interval of the stored data does not satisfy the preset interval condition, discard the data to be stored;

When the storage time interval of the data to be stored satisfies the preset interval condition, determine whether the data size of the data to be stored meets the preset size condition, and when the data size of the data to be stored does not satisfy the When the preset size condition is stated, part of the data to be stored is stored in the target database;

When the data size of the data to be stored meets the preset size condition, the data to be stored is stored in the target database.

Combining the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, and the fourth implementation manner of the first aspect, the present disclosure is in the first In a fifth implementation manner of the aspect, after the data size of the data to be stored meets the preset size condition, after storing the data to be stored in the target database, the method further includes:

Obtain the attribute information of the data to be stored, and when the attribute information of the data to be stored meets the video storage condition, request to obtain and store the video data related to the data to be stored.

Combining the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, the third implementation of the first aspect, the fourth implementation of the first aspect and the first aspect of the first aspect Five implementations. In a sixth implementation of the first aspect of the present disclosure, before storing the data to be stored according to the data storage rule to be stored, the method further includes:

Pre-process the data to be stored.

In a second aspect, a data storage device is provided in an embodiment of the present invention.

Specifically, the data storage device includes:

The obtaining module is configured to obtain a data packet to be stored, wherein the data packet to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

The determining module is configured to determine the data storage rule to be stored based on the data source information of the data to be stored and the data target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of data source to be stored and a data Storage data storage database sub-rules;

The storage module is configured to store the data to be stored according to the data storage rule to be stored.

With reference to the second aspect, in a first implementation manner of the second aspect of the embodiments of the present invention, the determination module includes:

The first determining submodule is configured to determine the data source category of the data to be stored according to the data source information of the data to be stored;

A second determining sub-module configured to determine the sub-rule of the data source of the data to be stored according to the type of the data source of the data to be stored;

The third determining submodule is configured to determine the category of the data target database to be stored based on the data target database information to be stored;

The fourth determination submodule is configured to determine the sub-rule of the data storage database to be stored according to the category of the target database of the data to be stored.

With reference to the second aspect and the first implementation manner of the second aspect, in a second implementation manner of the second aspect of the embodiments of the present invention, the storage module includes:

A judgment sub-module configured to judge whether the data to be stored conforms to the data storage rule to be stored;

The storage submodule is configured to store the data to be stored that meets the data storage rule to be stored in the target database.

With reference to the second aspect, the first implementation manner of the second aspect, and the second implementation manner of the second aspect, in a third implementation manner of the second aspect of the present disclosure, the data source sub-rule for storing data to be stored includes: One or more of the following rules: data quality meets preset quality conditions, storage time interval meets preset interval conditions; and / or,

Combining the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, and the third implementation manner of the second aspect, in the fourth implementation manner of the second aspect, the disclosure The storage module is configured to:

Combining the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, and the fourth implementation manner of the second aspect, the present disclosure is in the second In a fifth implementation manner of the aspect, the storage module is further configured to:

When the data size of the data to be stored meets the preset size condition, after the data to be stored is stored in the target database, the attribute information of the data to be stored is acquired, and when the data to be stored When the attribute information of meets the video storage condition, request to obtain video data related to the data to be stored and store it.

Combining the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, the fourth implementation manner of the second aspect and the second aspect Five implementations. The sixth implementation of the second aspect of the present disclosure further includes:

The preprocessing module is configured to preprocess the data to be stored.

In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory is used to store one or more computer instructions that support a data storage device to perform the data storage method in the first aspect described above. The processor is configured to execute computer instructions stored in the memory. The data storage device may further include a communication interface for the data storage device to communicate with other devices or communication networks.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium for storing computer instructions used by a data storage device, which includes a computer involved in performing the data storage method in the first aspect described above as a data storage device instruction.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

The above technical solution realizes the hierarchical storage of the data to be stored by separately setting the hierarchical storage rules of the data source and the hierarchical storage rules of the target database. The technical solution can realize effective storage of data, improve data quality, improve utilization of storage resources, and save storage resources.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the embodiments of the present invention.

BRIEF DESCRIPTION

With reference to the drawings, through the following detailed description of the non-limiting embodiments, other features, objects, and advantages of the embodiments of the present invention will become more apparent. In the drawings:

1 shows a flowchart of a data storage method according to an embodiment of the present invention;

FIG. 2 shows a flowchart of step S102 of the data storage method according to the embodiment shown in FIG. 1;

FIG. 3 shows a flowchart of step S103 of the data storage method according to the embodiment shown in FIG. 1;

4 shows a flowchart of step S103 of a data storage method according to another embodiment of the present invention;

5 shows a flowchart of a data storage method according to another embodiment of the present invention;

6 shows a structural block diagram of a data storage device according to an embodiment of the present invention;

7 shows a structural block diagram of the determination module 602 of the data storage device according to the embodiment shown in FIG. 6;

8 shows a structural block diagram of a storage module 603 of the data storage device according to the embodiment shown in FIG. 6;

9 shows a structural block diagram of a data storage device according to another embodiment of the present invention;

10 shows a structural block diagram of an electronic device according to an embodiment of the present invention;

11 is a schematic structural diagram of a computer system suitable for implementing a data storage method according to an embodiment of the present invention.

detailed description

Hereinafter, exemplary embodiments of the embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for clarity, parts not related to the description of the exemplary embodiment are omitted in the drawings.

In the embodiments of the present invention, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, acts, components, parts, or combinations thereof disclosed in this specification, and are not intended to The possibility that one or more other features, numbers, steps, acts, components, parts, or combinations thereof exist or are added is excluded.

In addition, it should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict. The embodiments of the present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

The technical solution provided by the embodiment of the present invention realizes the hierarchical storage of the data to be stored by separately setting the hierarchical storage rules of the data source and the hierarchical storage rules of the target database. The technical solution can realize effective storage of data, improve data quality, improve utilization of storage resources, and save storage resources.

FIG. 1 shows a flowchart of a data storage method according to an embodiment of the present invention. As shown in FIG. 1, the data storage method includes the following steps S101-S103:

In step S101, obtain a data packet to be stored, wherein the data packet to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

In step S102, a data storage rule to be stored is determined according to the data source information of the data to be stored and the target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of data source to be stored and a data to be stored Data storage database sub-rules;

In step S103, the data to be stored is stored according to the data storage rule to be stored.

As mentioned above, with the development of data technology and social progress, the public has higher and higher requirements for security systems, and the amount of monitoring data that needs to be stored is also increasing. How to distinguish effective data from it, How to effectively store effective data to improve the utilization rate of storage resources and save storage resources is an urgent problem to be solved in the industry.

In view of the above problems, in this embodiment, a data storage method is proposed, which implements hierarchical storage of data to be stored by separately setting data source hierarchical storage rules and target database hierarchical storage rules. The technical solution can realize effective storage of data, improve data quality, improve utilization of storage resources, and save storage resources.

The data packet to be stored may be a data packet of any form, any type, and any size. For ease of understanding, the following uses the monitoring data as an example to explain and explain the present invention.

In an optional implementation manner of this embodiment, the data packet to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored.

The data source information of the data to be stored refers to the information of the data source to obtain the data to be stored. For example, if the data to be stored is video surveillance data, the corresponding data source information refers to the collection Information of the camera of the video monitoring data. In order to more effectively implement the hierarchical storage of the data to be stored, the data source information of the data to be stored may include the level or category information of the data source, that is, the data source may be graded or classified according to preset rules in advance, and then the data Source level or category information is carried in the data source information and sent. For example, for cameras that collect video surveillance data, the cameras can be ranked or classified according to factors such as the purpose of their settings, geographic location, human traffic in the geographic location, frequency of historical events in the geographic location, and so on. The cameras are divided into levels 1-4, of which level 1 indicates normal locations, level 2 indicates crowded locations, level 3 indicates locations where accidents occur frequently, and level 4 indicates key monitoring locations.

Wherein, the target database information of the data to be stored refers to the information of the database storing the data to be stored. The target database information of the data to be stored may be preset according to the attribute information of the data to be stored, and the target database The information is carried in the data source information and sent, wherein the attribute information of the data to be stored includes one or more of the following information: education level, religious belief, birthplace, historical crime information, whether it is involved in drugs, whether it is at large , Whether it involves independence or terrorism, etc. In order to more effectively realize the hierarchical storage of the data to be stored, the target database information of the data to be stored may include the level or category information of the target database, that is, the database used to store the data may be hierarchically or in accordance with preset rules Classification, for example, for a database that stores monitoring personnel, the database can be classified or classified according to the nature of the monitoring personnel. For example, the database can be divided into 1-5 levels, where level 1 represents the ordinary personnel database and level 2 represents the focus on personnel Database, level 3 means a database with criminal records, level 4 means a database of ordinary fugitives, and level 5 means a database of high-risk terrorists.

Based on the level or category information of the data source and the target database, when storing the data to be stored later, different storage rules can be determined according to the level or category information of the data source and the target database, and then the Hierarchical storage of data to be stored.

In an optional implementation manner of this embodiment, as shown in FIG. 2, the step S102 is the step of determining the data storage rule to be stored based on the data source information of the data to be stored and the target database information of the data to be stored, It includes the following steps S201-S204:

In step S201, the data source category of the data to be stored is determined according to the data source information of the data to be stored;

In step S202, the sub-rule of the data source of the data to be stored is determined according to the type of the data source of the data to be stored;

In step S203, the category of the target database of the data to be stored is determined according to the target database information of the data to be stored;

In step S204, the sub-rule of the data storage database to be stored is determined according to the category of the target database of the data to be stored.

In order to achieve hierarchical storage of data, truly achieve effective storage of effective data, improve utilization of storage resources, and save storage resources, in this embodiment, different storage rules are determined according to the level or category information of the data source and target database. Specifically, the data source category of the data to be stored is determined according to the data source information of the data to be stored; then the sub-rule of the data source of the data to be stored is determined according to the data source category of the data to be stored; The data target database information to be stored determines the category of the data target database to be stored; and then determines the sub-rule of the data storage database to be stored according to the category of the data target database to be stored.

In an optional implementation manner of this embodiment, as shown in FIG. 3, the step S103, that is, the step of storing the data to be stored according to the data storage rule to be stored, includes the following steps S301-S302:

In step S301, determine whether the data to be stored conforms to the data storage rule to be stored;

In step S302, data to be stored that meets the data storage rule to be stored is stored in the target database.

In order to effectively achieve hierarchical storage of data and improve data quality, in this embodiment, the data to be stored and the determined storage rules are compared and judged one by one, and the data that meets the storage rules is stored in the target database in.

In an optional implementation manner of this embodiment, for monitoring data, the sub-rules for storing data sources of data to be stored may include one or more of the following rules: data quality meets preset quality conditions, and storage time interval meets Preset interval conditions. Among them, the preset quality condition is a requirement for data quality, such as whether a certain monitoring image is clear, whether the resolution of a certain monitoring image meets the resolution requirements, whether there are too many noise points in a certain monitoring image, a certain 1. Monitor the image for missing pixels and so on. Wherein, the preset interval condition is a requirement for the data storage interval, which is to improve the effectiveness of the data to be stored and avoid excessive repetition of information, for example, to determine whether the time interval between adjacent data to be stored is greater than A time interval threshold and so on.

In an optional implementation manner of this embodiment, for monitoring data, the sub-rules of the data storage database to be stored may include one or more of the following rules: data similarity meets a preset similarity condition, and data size meets Preset size conditions, video storage conditions. The preset similarity condition is a requirement for the similarity between the stored data and the target database data, and the data to be stored that meets the similarity condition is considered to be similar to the data in the target database. Data stored in the target database, otherwise it cannot be stored in the database. The preset size condition refers to a requirement for the size of the data to be stored. If the data to be stored is too large, the key data in the data to be stored can be extracted for storage, and other unimportant ones are discarded. data. Wherein, the video storage condition refers to a requirement on whether to acquire and store video data related to the stored data. For example, if the data to be stored is a person who needs to be focused on, you can request to acquire and store and Video data of the preset time point and preset time length related to the person, so as to be further viewed later.

It should be noted that the specific sub-rules for the data source to be stored and the sub-rules for the database to be stored are related to the levels or categories of the data source and target database, that is, the levels or categories of the data source and target database are different. The content or parameters of the data source sub-rules of the data storage to be stored and the sub-rules of the data storage database to be stored are also different, so as to truly achieve hierarchical storage of data. Wherein, the correspondence between the sub-rules of the data source of the data to be stored and the sub-rules of the database of the data to be stored and the levels or categories of the data source and the target database can be pre-stored in the memory or database to facilitate the content of specific rules Query and confirm. In actual application, those skilled in the art can set and adjust the specific content of the rules according to the needs of the actual application and the characteristics of the data to be stored, the data source and the database. The above are all exemplary descriptions and cannot be understood as Limitation of the invention.

In another optional implementation manner of this embodiment, as shown in FIG. 4, the step S103, that is, the step of storing the data to be stored according to the data storage rule to be stored, includes the following steps S401-S405 :

In step S401, it is determined whether the quality of the data to be stored meets the preset quality condition, and when the quality of the data to be stored does not satisfy the preset quality condition, the data to be stored is discarded;

In step S402, when the quality of the data to be stored satisfies the preset quality condition, it is determined whether the similarity between the data to be stored and the target database satisfies the preset similarity condition, and when When the similarity between the data to be stored and the target database does not satisfy the preset similarity condition, a new database is created, and the data to be stored is stored in the newly created database;

In step S403, when the similarity between the data to be stored and the target database satisfies the preset similarity condition, it is determined whether the storage time interval of the data to be stored meets the preset interval condition, And when the storage time interval of the data to be stored does not satisfy the preset interval condition, discard the data to be stored;

In step S404, when the storage time interval of the data to be stored meets the preset interval condition, it is determined whether the data size of the data to be stored meets the preset size condition, and when the data to be stored When the data size does not satisfy the preset size condition, store part of the data to be stored in the target database;

In step S405, when the data size of the data to be stored satisfies the preset size condition, the data to be stored is stored in the target database.

This embodiment takes monitoring data as an example, and describes in detail the steps for storing the data to be stored according to the data storage rule of the data to be stored. Specifically, first determine whether the quality of the data to be stored meets the preset quality condition, if the quality of the data to be stored does not meet the preset quality condition, it may be considered that the data to be stored has no stored value, To save storage resources, you can directly discard it; if the quality of the data to be stored meets the preset quality condition, continue to determine whether the similarity between the data to be stored and the target database meets the A preset similarity condition, if the similarity between the data to be stored and the target database does not satisfy the preset similarity condition, it may be considered that there is no similarity to the data in the target database, or The target database does not store historical data of the monitoring object involved in the data to be stored, a new database may be created, and the data to be stored may be stored in the newly created database; if the data to be stored and the target The similarity between the databases meets the preset similarity condition, and the data to be stored is considered to be similar to the data in the target database , It can be stored in the target database, and then continue to determine whether the storage time interval of the data to be stored meets the preset interval condition, if the storage time interval of the data to be stored does not meet the preset interval condition, It can be considered that the data to be stored has a short interval, too much repeated information and less effective information, so it can be discarded, or for the accuracy of data statistics, the number of data processing statistics of the target database can also be increased by Value; if the storage time interval of the data to be stored meets the preset interval condition, continue to determine whether the data size of the data to be stored meets the preset size condition, if the data size of the data to be stored If the preset size condition is not met, for example, the data to be stored is too large, in order to save storage resources, part of the data to be stored may be stored in the target database, and other parts may be discarded; If the data size of the data meets the preset size condition, the data to be stored may be stored in the target data In.

It should be noted that the above judgment order for the specific content of the rule is not fixed, and those skilled in the art can adjust the rule judgment order according to the needs of actual application, the characteristics of the data to be stored, and the specific content of the rule. No specific restrictions.

In an optional implementation manner of this embodiment, in step S405, that is, when the data size of the data to be stored meets the preset size condition, the data to be stored is stored in the target database. After the steps, you can also include the steps:

In this embodiment, it is also judged according to the attribute information of the data to be stored whether to request to obtain the video data related to the data to be stored and store it, for example, if the data to be stored is a person who needs to be focused on, Then, you can request to obtain and store video data related to the person at a preset time point and a preset time length, so that you can view it later. Wherein, the preset time point and preset time length can be set according to actual application needs, for example, the preset time point can be set as the generation time of the data to be stored, and the preset time length can be set as 2 minutes.

In an optional implementation manner of this embodiment, the step S103, that is, before the step of storing the data to be stored according to the data storage rule to be stored, further includes preprocessing the data to be stored Steps, as shown in FIG. 5, the method includes the following steps S501-S504:

In step S501, a data package to be stored is obtained, wherein the data package to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

In step S502, a data storage rule to be stored is determined according to the data source information of the data to be stored and the target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of data source to be stored and a data to be stored Data storage database sub-rules;

In step S503, pre-process the data to be stored;

In step S504, the data to be stored is stored according to the data storage rule to be stored.

In an optional implementation manner of this embodiment, the preprocessing may include one or more of the following processes: deduplication processing, denoising processing, compression processing, and so on.

In this embodiment, in order to further improve the validity of the data, improve the quality of the data, and save computing resources, before storing the data to be stored, the data to be stored is also subjected to deduplication processing, denoising processing, and / or Pre-processing such as compression processing.

The following is an embodiment of the device of the present invention, which can be used to execute the method embodiment of the present invention.

6 shows a structural block diagram of a data storage device according to an embodiment of the present invention. The device may be implemented as part or all of an electronic device through software, hardware, or a combination of both. As shown in FIG. 6, the data storage device includes:

The obtaining module 601 is configured to obtain a data packet to be stored, wherein the data packet to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

The determining module 602 is configured to determine a data storage rule to be stored based on the data source information of the data to be stored and the data target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule for data source to be stored and Sub-rules of the data storage database to be stored;

The storage module 603 is configured to store the data to be stored according to the data storage rule to be stored.

In consideration of the above problems, in this embodiment, a data storage device is proposed, which implements hierarchical storage of data to be stored by setting data source hierarchical storage rules and target database hierarchical storage rules, respectively. The technical solution can realize effective storage of data, improve data quality, improve utilization of storage resources, and save storage resources.

In an optional implementation manner of this embodiment, as shown in FIG. 7, the determination module 602 includes:

The first determining submodule 701 is configured to determine the data source category of the data to be stored according to the data source information of the data to be stored;

The second determining sub-module 702 is configured to determine the sub-rule of the data source of the data to be stored according to the type of the data source of the data to be stored;

The third determining submodule 703 is configured to determine the category of the data target database to be stored according to the data target database information to be stored;

The fourth determination sub-module 704 is configured to determine the sub-rule of the data storage database to be stored according to the category of the target database of the data to be stored.

In order to achieve hierarchical storage of data, truly achieve effective storage of effective data, improve utilization of storage resources, and save storage resources, in this embodiment, different storage rules are determined according to the level or category information of the data source and target database. Specifically, the first determining submodule 701 determines the data source category of the data to be stored according to the data source information of the data to be stored; the second determining submodule 702 determines the data source category according to the data source category of the data to be stored Stored data storage data source sub-rules; third determination sub-module 703 determines the category of the data-to-be-stored data target database according to the data-to-be-stored data target database information; fourth determination sub-module 704 according to the type of the data-to-be-stored target database Determine the sub-rule of the data storage database to be stored.

In an optional implementation manner of this embodiment, as shown in FIG. 8, the storage module 603 includes:

The judgment sub-module 801 is configured to judge whether the data to be stored conforms to the data storage rule to be stored;

The storage submodule 802 is configured to store the data to be stored that meets the data storage rule to be stored in the target database.

In order to effectively achieve hierarchical storage of data and improve data quality, in this embodiment, the judgment sub-module 801 compares and judges the data to be stored with the determined storage rule one by one, and the storage sub-module 802 will conform to the storage The rule data is stored in the target database.

In another optional implementation manner of this embodiment, the storage module 603 is configured to:

This embodiment takes monitoring data as an example to describe the operation flow of the storage module 603 in detail. Specifically, first determine whether the quality of the data to be stored meets the preset quality condition, if the quality of the data to be stored does not meet the preset quality condition, it may be considered that the data to be stored has no stored value, To save storage resources, you can directly discard it; if the quality of the data to be stored meets the preset quality condition, continue to determine whether the similarity between the data to be stored and the target database meets the A preset similarity condition, if the similarity between the data to be stored and the target database does not satisfy the preset similarity condition, it may be considered that there is no similarity to the data in the target database, or The target database does not store historical data of the monitoring object involved in the data to be stored, a new database may be created, and the data to be stored may be stored in the newly created database; if the data to be stored and the target The similarity between the databases meets the preset similarity condition, and the data to be stored is considered to be similar to the data in the target database , It can be stored in the target database, and then continue to determine whether the storage time interval of the data to be stored meets the preset interval condition, if the storage time interval of the data to be stored does not meet the preset interval condition, It can be considered that the data to be stored has a short interval, too much duplicate information, and less effective information, so it can be discarded, or for the accuracy of data statistics, the number of data processing statistics of the target database can also be increased accordingly Value; if the storage time interval of the data to be stored meets the preset interval condition, continue to determine whether the data size of the data to be stored meets the preset size condition, if the data size of the data to be stored If the preset size condition is not met, for example, the data to be stored is too large, in order to save storage resources, part of the data to be stored may be stored in the target database, and other parts may be discarded; If the data size of the data meets the preset size condition, the data to be stored may be stored in the target data In.

In an optional implementation manner of this embodiment, the storage module is further configured to: when the data size of the data to be stored meets the preset size condition, store the data to be stored in the target After the database, the attribute information of the data to be stored is obtained, and when the attribute information of the data to be stored meets the video storage condition, the video data related to the data to be stored is requested to be obtained and stored.

In an optional implementation manner of this embodiment, the device further includes a part for preprocessing the data to be stored, that is, as shown in FIG. 9, the device includes:

The obtaining module 901 is configured to obtain a data packet to be stored, wherein the data packet to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

The determining module 902 is configured to determine a data storage rule to be stored based on the data source information of the data to be stored and the target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of data source to be stored and Sub-rules of the data storage database to be stored;

The preprocessing module 903 is configured to preprocess the data to be stored

The storage module 904 is configured to store the data to be stored according to the data storage rule to be stored.

In a specific application scenario, an amusement park with 100 channels of surveillance video deployed requires real-time face analysis of the 100 channels of video. Assuming that only one person is captured by each camera at the same time, then the storage solution according to the prior art In a year, tens of billions of data will be generated and stored. Even after tracking and deduplication, it will generate billions of data, with a total data size of 1000T. If the hierarchical storage technology solution proposed by the present invention is adopted, it is assumed that the following settings are made: the same person passes the same camera within 10 minutes and only calculates the frequency and does not store the data; when the data to be stored involves fugitives and criminals, video acquisition and Storage; implement key monitoring on 10 of them. Then 100-channel video will only produce about 50T of data a year, that is, the amount of processed data is compressed by 20 times. It can be seen that the technical solution of the present invention can achieve effective storage of data, improve data quality, greatly improve the utilization rate of storage resources, and save storage resources.

An embodiment of the present invention also discloses an electronic device. FIG. 10 shows a structural block diagram of an electronic device according to an embodiment of the present invention. As shown in FIG. 10, the electronic device 1000 includes a memory 1001 and a processor 1002;

The memory 1001 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 1002 to implement any of the above method steps.

As shown in FIG. 11, the computer system 1100 includes a central processing unit (CPU) 1101 that can be loaded into a random access memory (RAM) 1103 according to a program stored in a read-only memory (ROM) 1102 or from a storage section 1108 Instead, various processes in the above-described embodiment are executed. In the RAM 1103, various programs and data necessary for the operation of the system 1100 are also stored. The CPU 1101, ROM 1102, and RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc .; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 1108 including a hard disk, etc. ; And a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. The driver 1110 is also connected to the I / O interface 1105 as necessary. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as necessary, so that the computer program read out therefrom is installed into the storage portion 1108 as needed.

In particular, according to embodiments of the present invention, the method described above may be implemented as a computer software program. For example, embodiments of the present invention include a computer program product that includes a computer program tangibly contained on a readable medium thereof, the computer program containing program code for performing the data storage method. In such an embodiment, the computer program may be downloaded and installed from the network through the communication section 1109, and / or installed from the removable medium 1111.

The flowchart and block diagrams in the drawings illustrate the possible implementation architecture, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the road map or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of the code contains one or more Executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession can actually be executed in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented with dedicated hardware-based systems that perform specified functions or operations Or, it can be realized by a combination of dedicated hardware and computer instructions.

The units or modules described in the embodiments of the present invention may be implemented in software or hardware. The described units or modules may also be provided in the processor, and the names of these units or modules in some cases do not constitute a limitation on the units or modules themselves.

As another aspect, an embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium may be a computer-readable storage medium included in the device described in the foregoing embodiments; or may exist alone , A computer-readable storage medium that is not installed in the device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to perform the methods described in the embodiments of the present invention.

The above description is only the preferred embodiment of the present invention and the explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover the above-mentioned Other technical solutions formed by any combination of technical features or their equivalents. For example, the above features and the technical features disclosed in the embodiments of the present invention (but not limited to) having similar functions are replaced with each other to form a technical solution.

Claims

A data storage method is characterized by including:

Obtaining a data package to be stored, wherein the data package to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

Determine the data storage rule to be stored according to the data source information of the data to be stored and the target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of the data source to be stored and a sub-rule of the data storage database ;

Storing the data to be stored according to the data storage rule to be stored.
The method according to claim 1, wherein the determining the data storage rule to be stored according to the data source information and the data target database information of the data to be stored comprises:

Determine the data source category of the data to be stored according to the data source information of the data to be stored;

Determining the sub-rule of the data source of the data to be stored according to the type of the data source of the data to be stored;

Determine the category of the target database of data to be stored according to the information of the target database of data to be stored;

The sub-rule of the data storage database to be stored is determined according to the category of the target database of the data to be stored.
The method according to claim 1 or 2, wherein the storing the data to be stored according to the data storage rule to be stored comprises:

Determine whether the data to be stored conforms to the data storage rule to be stored;

Storing the data to be stored in accordance with the data storage rule to be stored in the target database.
The method according to any one of claims 1-3, wherein the data source sub-rules for storing data to be stored include one or more of the following rules: data quality meets preset quality conditions, and storage time interval meets Preset interval conditions; and / or,

The sub-rules of the data storage database to be stored include one or more of the following rules: data similarity meets a preset similarity condition, data size meets a preset size condition, and video storage condition.
The method according to claim 4, wherein the storing the data to be stored according to the data storage rule includes:

Judging whether the quality of the data to be stored meets the preset quality condition, and discarding the data to be stored when the quality of the data to be stored does not meet the preset quality condition;

When the quality of the data to be stored satisfies the preset quality condition, determine whether the similarity between the data to be stored and the target database satisfies the preset similarity condition, and when the data to be stored When the similarity with the target database does not satisfy the preset similarity condition, create a new database, and store the data to be stored in the newly created database;

When the similarity between the data to be stored and the target database satisfies the preset similarity condition, it is determined whether the storage time interval of the data to be stored meets the preset interval condition, and when the pending When the storage time interval of the stored data does not satisfy the preset interval condition, discard the data to be stored;

When the storage time interval of the data to be stored satisfies the preset interval condition, determine whether the data size of the data to be stored meets the preset size condition, and when the data size of the data to be stored does not satisfy the When the preset size condition is stated, part of the data to be stored is stored in the target database;

When the data size of the data to be stored meets the preset size condition, the data to be stored is stored in the target database.
The method according to claim 5, wherein after the data size of the data to be stored meets the preset size condition, after storing the data to be stored in the target database, the method further includes :

Obtain the attribute information of the data to be stored, and when the attribute information of the data to be stored meets the video storage condition, request to obtain and store the video data related to the data to be stored.
The method according to any one of claims 1-6, wherein before storing the data to be stored according to the data storage rule to be stored, the method further comprises:

Pre-process the data to be stored.
A data storage device, including:

The obtaining module is configured to obtain a data packet to be stored, wherein the data packet to be stored includes data to be stored, data source information of the data to be stored, and target database information of the data to be stored;

The determining module is configured to determine the data storage rule to be stored based on the data source information of the data to be stored and the data target database information of the data to be stored, wherein the data storage rule to be stored includes a sub-rule of data source to be stored and a data Storage data storage database sub-rules;

The storage module is configured to store the data to be stored according to the data storage rule to be stored.
The apparatus according to claim 8, wherein the determination module comprises:

The first determining submodule is configured to determine the data source category of the data to be stored according to the data source information of the data to be stored;

A second determining sub-module configured to determine the sub-rule of the data source of the data to be stored according to the type of the data source of the data to be stored;

The third determining submodule is configured to determine the category of the data target database to be stored based on the data target database information to be stored;

The fourth determination submodule is configured to determine the sub-rule of the data storage database to be stored according to the category of the target database of the data to be stored.
The device according to claim 8 or 9, wherein the storage module comprises:

A judgment sub-module configured to judge whether the data to be stored conforms to the data storage rule to be stored;

The storage submodule is configured to store the data to be stored that meets the data storage rule to be stored in the target database.
The device according to any one of claims 8 to 10, wherein the sub-rules for storing the data source of the data to be stored include one or more of the following rules: data quality meets a preset quality condition, and storage time interval meets Preset interval conditions; and / or,

The sub-rules of the data storage database to be stored include one or more of the following rules: data similarity meets a preset similarity condition, data size meets a preset size condition, and video storage condition.
The apparatus according to claim 11, wherein the storage module is configured to:

Judging whether the quality of the data to be stored meets the preset quality condition, and discarding the data to be stored when the quality of the data to be stored does not meet the preset quality condition;

When the quality of the data to be stored satisfies the preset quality condition, determine whether the similarity between the data to be stored and the target database satisfies the preset similarity condition, and when the data to be stored When the similarity with the target database does not satisfy the preset similarity condition, create a new database, and store the data to be stored in the newly created database;

When the similarity between the data to be stored and the target database satisfies the preset similarity condition, it is determined whether the storage time interval of the data to be stored meets the preset interval condition, and when the pending When the storage time interval of the stored data does not satisfy the preset interval condition, discard the data to be stored;

When the storage time interval of the data to be stored satisfies the preset interval condition, determine whether the data size of the data to be stored meets the preset size condition, and when the data size of the data to be stored does not satisfy the When the preset size condition is stated, part of the data to be stored is stored in the target database;

When the data size of the data to be stored meets the preset size condition, the data to be stored is stored in the target database.
The apparatus according to claim 12, wherein the storage module is further configured to:

When the data size of the data to be stored meets the preset size condition, after the data to be stored is stored in the target database, the attribute information of the data to be stored is acquired, and when the data to be stored When the attribute information of meets the video storage condition, request to obtain video data related to the data to be stored and store it.
The device according to any one of claims 8-13, further comprising:

The preprocessing module is configured to preprocess the data to be stored.
An electronic device characterized by comprising a memory and a processor; wherein,

The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any one of claims 1-7.
A computer-readable storage medium on which computer instructions are stored, characterized in that, when the computer instructions are executed by a processor, the method steps of any one of claims 1-7 are implemented.