CN111190908B

CN111190908B - Data management method, device and system

Info

Publication number: CN111190908B
Application number: CN201811362178.5A
Authority: CN
Inventors: 吴超豪; 郑方舟; 高剑
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2018-11-15
Filing date: 2018-11-15
Publication date: 2023-09-22
Anticipated expiration: 2038-11-15
Also published as: CN111190908A

Abstract

The embodiment of the application provides a data management method and a data management device, wherein in the method, data are classified into different sub-databases according to different standards, a storage space is divided into a large number of independent data units, and data in the sub-databases are accurately stored in each data unit by using index information, memory information and three-level management information of the sub-databases. When various sub-databases with uncertain data volumes are required to be stored and managed, the technical scheme provided by the embodiment of the application can realize fine management of the storage space by dividing the storage space into a large number of data units with smaller space, improve the utilization rate of the storage space and avoid the waste of storage resources.

Description

Data management method, device and system

Technical Field

The present application relates to the field of information technologies, and in particular, to a method, an apparatus, and a system for multiplexing a shared memory.

Background

With the development of the video monitoring technology at present, the monitoring is developed from pure video recording to intelligent, so that not only the collected video data needs to be stored, but also useful information in the video data needs to be extracted, and the data extracted from the video data and used for describing the video content is called as feature data. By extracting feature data from the video, it is achieved that the video content is described with the feature data.

A working camera will collect a large amount of video data every day, a large amount of feature data will be generated correspondingly, and a video monitoring system usually includes a plurality of cameras, so that more feature data will be generated when the video monitoring system works, and how to store and manage the feature data becomes a problem to be solved.

In the prior art, feature data are stored in a form of a fixed storage space, namely, feature data acquired by each camera every day are respectively stored by utilizing the storage space with a fixed size, and all feature data generated by each camera every day can be classified as a feature database. Because each camera monitors different scenes and times, the cameras of different scenes or the data amount of the feature database generated by the same camera at different times has different amounts. In order to ensure that each feature database has enough storage space, the fixed storage space needs to be set according to the maximum data volume of the feature database, so that the storage space is wasted for some feature databases with smaller data volumes.

Disclosure of Invention

In view of this, the embodiment of the application provides a data management method and system, which divide a storage space into a large number of data units to store data, and manage each data unit by adopting three levels of management information, so as to realize fine management of the storage space and improve the utilization rate of the storage space.

In order to achieve the above object, in a first aspect, an embodiment of the present application provides a data management method, where when data of a sub-database needs to be added, management information in a storage device is read, where the storage device includes a plurality of data units, and the management information includes one or more of sub-database index information, data unit status information, and data management information; and then, determining available data units of the data of the sub-database to be added according to the management information, writing the data of the sub-database to be added into the available data units, and updating the management information in the storage device after adding the data. The method provided by the embodiment of the application divides the storage space in the storage device into a plurality of smaller data units, and then utilizes the three-level management information to finely manage the data units, so that the waste of storage resources can be avoided, and the utilization rate of the storage device is obviously improved.

Wherein the available data units are data units to which sub-database data to be added can be written.

Optionally, before writing the sub-database data, the storage device needs to be initialized, i.e. the storage device is divided into a portion storing management information and a portion storing data, the portion storing data being further divided into a plurality of data units. The size of the data unit can be set according to the actual storage requirement. It should be noted that, the storage device needs to be initialized only when the technical scheme provided by the embodiment of the application is used for the first time, and the storage device does not need to be initialized every time sub-database data is added.

There are a number of ways to determine the available data units according to different circumstances.

Optionally, firstly, inquiring index information of a sub-database, and when the index information of the sub-database comprises a sub-database to be added, taking a data unit of the last added data in the index information of the sub-database to be added as an available data unit; and when the storage space of the data unit added with the sub-database data to be added last time is insufficient, inquiring the state information of the data unit, and acquiring an unused data unit as an available data unit.

Optionally, when the sub-database to be added is not included in the sub-database index information, the data of the sub-database is added for the first time at this time, the state information of the data unit needs to be queried, and an unused data unit is obtained as an available data unit. Furthermore, index information corresponding to the sub-database needs to be added, so that the available data units can be rapidly positioned when the sub-database is added next time.

Optionally, after adding the data to the available data units, the management information needs to be updated, specifically, the data management information corresponding to the available data units and the index information of the sub-database to be added need to be updated. The corresponding management information is updated every time data is added, so that the available data units can be quickly positioned when the sub-database data is added next time, and the data of the sub-database can be conveniently searched.

Optionally, after writing the data of the sub-database to be added, the storage space of the available data unit is full, and the state information of the data unit corresponding to the available data unit needs to be updated, so that the state of the available data unit is marked as full storage space.

Optionally, in some cases, the stored data of the sub-database needs to be deleted, and when the sub-database is deleted, the data management information is queried according to the identification of the sub-database to be deleted, so as to obtain the data management information to be cleared, wherein the data management information to be cleared comprises the identification of the sub-database to be deleted; and then, clearing the data management information to be cleared, updating the state information of the data units, marking the state of the data units corresponding to the data management information to be cleared as an available state, and deleting the index information of the sub-database to be deleted. According to the data management method provided by the embodiment of the application, when the sub-database is deleted, the data actually stored in the data unit is not required to be deleted, only the management information is required to be modified, the corresponding data management information is cleared, the state of the data unit is updated, and the data unit is put into an available resource pool again. At this time, although the data stored in the data unit is not deleted, since the management information has been deleted, it is impossible to determine or identify which type of data or which structure the data stored in the data unit is, the data stored in the data unit becomes meaningless pure data and can be overwritten with new data. The method for deleting the data provided by the embodiment of the application only needs to modify the corresponding management information, does not need to operate the data unit, and improves the deleting efficiency.

In a second aspect, an embodiment of the present application provides a data management device, where the device has a function of implementing the method described in the first aspect. The functions can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the functions described above.

In a third aspect, an embodiment of the present application provides a data management apparatus, including: a processor, a memory, a bus, and a communication interface; the memory is configured to store computer-executable instructions, the processor being coupled to the memory via the bus, the processor executing the computer-executable instructions of the memory when the data management device is operating to cause the data management device to perform the data management method as described in any of the first aspects above.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having instructions stored therein that, when executed on a computer, cause the computer to perform the data management method of any one of the first aspects described above.

In a fifth aspect, embodiments of the present application provide a computer program product comprising instructions which, when run on a computer, enable the computer to perform the data management method of any one of the first aspects above.

The technical effects of any one of the design manners of the second aspect to the fifth aspect may be referred to the technical effects of the different design manners of the first aspect, and will not be repeated here.

In a sixth aspect, an embodiment of the present application further provides a data management system, including a data management device and a storage device as described in any one of the above aspects.

The data management method provided by the embodiment of the application classifies the data into different sub-databases according to different standards, divides the storage space into a large number of independent data units, and accurately stores the data in the sub-databases in each data unit by utilizing the index information, the memory information and the three-level management information of the data management units of the sub-databases. When various data with obvious tag information and uncertain data volume need to be stored and managed, the technical scheme provided by the embodiment of the application can realize fine management of the storage space by dividing the storage space into a large number of data units with smaller space, improve the utilization rate of the storage space and avoid the waste of storage resources.

These and other aspects of the application will be more readily apparent from the following description of the embodiments.

Drawings

FIG. 1 is a schematic diagram of a data management method according to the prior art;

FIG. 2 is a schematic diagram of a data management system according to an embodiment of the present application;

FIG. 3 is a schematic diagram of mapping relationship between three levels of management information and data units according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a process of adding data in the data management method according to the embodiment of the present application;

fig. 6 is a schematic flow chart of obtaining available data units of added data according to an embodiment of the present application;

FIG. 7 is a schematic diagram of another flow chart of adding data in the data management method according to the embodiment of the application;

FIG. 8 is a schematic diagram of data stored by the data management method according to the embodiment of the present application;

fig. 9 is a schematic flow chart of deleting data in the data management method according to the embodiment of the present application;

FIG. 10 is a schematic diagram of another process for deleting data in the data management method according to the embodiment of the present application;

fig. 11 is a schematic structural diagram of a data management device according to an embodiment of the present application.

Detailed Description

First, some terms in the present application will be explained in order to be understood by those skilled in the art.

1. The feature data refers to data for converting pictures into attributes or characteristics of the descriptive images through a feature extraction algorithm in video monitoring, and belongs to structured or semi-structured data. The feature data may include character characteristics, weather, shooting address, time in the image, or information of an object in the image, such as information of a vehicle color, a model number, a license plate, and the like, and the feature data may also include information of a face feature, and the like. The embodiment of the application takes the storage of the characteristic data as an example, but is not limited to the storage of the characteristic data, and can also store other data with similar characteristics, such as cases of hospitals or resident information of public security departments, and the like, which are divided into different types according to labels.

2. Sub-databases refer to the division of data into different sub-databases according to a specific division criterion. Taking the feature data as an example, the feature data obtained by converting an image collected by a certain camera deployed in a certain place may be referred to as a sub-database, or the feature data obtained by converting an image set of a person or a car may be referred to as a sub-database. Different division standards may be provided for other data, for example, all cases in a certain department of a hospital on a certain day may be regarded as a sub-database, and all cases of a certain patient may be regarded as a sub-database.

3. Plural, two or more.

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings.

The feature data corresponding to the video data collected by the current storage camera is usually set to the same storage space for each sub-database, and then different sub-databases are stored in different storage spaces. However, there may be a large difference value in the amount of data collected by each sub-database according to the monitoring time and the monitoring scene. As shown in fig. 1, taking the data amount collected by a market, a school and a roadway on a weekday and a holiday as an example, since the market generally has more people flow on the holiday and the school has more people flow on the weekday, the data amount collected by a camera installed in the market on the weekday is smaller than that of the school on the holiday, the school is opposite to the former, more characteristic data is collected on the weekday, and little characteristic data is collected on the holiday. For cameras installed in a roadway, since the traffic is generally small, the feature data collected by such cameras is relatively small, whether on weekdays or holidays. If a fixed storage space is provided, it is necessary to set the fixed storage space to be large enough to ensure that the data of each sub-database can be stored. When the fixed storage space is set in the above manner, for sub-databases collected in most scenes or time periods, there is a waste of the storage space, resulting in a low utilization rate of the storage space.

In order to solve the problem of low storage space utilization rate when feature data are stored in the prior art, the embodiment of the application provides a data management method and device.

As shown in fig. 2, a possible data management system according to an embodiment of the present application includes a data management device and a storage device.

The data management device in fig. 2 is configured to read management information in the storage device when data needs to be added, determine a data unit for storing the data that needs to be added according to the management information in the storage device, write the data that needs to be added to a corresponding address in the storage device, and correspondingly modify the management information in the storage device. The data management device is also used for reading the management information in the storage device when the data need to be deleted, and modifying the corresponding management information according to the label information of the data needing to be deleted, without deleting the data needing to be deleted, which is stored in the data unit.

The data management apparatus in fig. 2 may be implemented by a general-purpose central processing unit (Central Processing Unit, CPU) and a memory, in which program instructions for implementing the data management method provided by the embodiment of the present application are stored, and the CPU executes the data management method provided by the embodiment of the present application by calling the program instructions. The data management apparatus may also be implemented using an Application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA), or other programmable logic device, or a transistor logic device, where the ASIC or FPGA is designed with circuitry for implementing the data management method provided by the embodiments of the present application.

The memory device in fig. 2 is divided into two blocks of memory space: an information storage space and a data storage space are managed. The management information storage space stores management information of the storage device, wherein the management information comprises management information of data stored in the storage device and state management information of the data storage space, and specifically, the management information comprises sub-database index information, data unit state information and data management information. The data storage space in the storage device is divided into a plurality of data units, the size of the data units can be set according to the characteristics of the stored data, and if the characteristic data of video monitoring needs to be stored, the size of the data units can be set to 8KB, 16KB, 32KB and the like, and the embodiment of the application is not limited to the above. It should be noted that, in order to implement fine management of the storage space, the size of the data unit in the embodiment of the present application is generally set smaller than that of the fixed storage space in the prior art.

Index information of each sub-database currently stored in the storage device is recorded in index information of the sub-database in the management information, and the index information comprises addresses of data units written in the sub-database last time and is used for quickly determining the data units capable of being written with new data when the data of the sub-database is written next time. The sub-database index information may further include a remaining storage space of the data unit written last time by the sub-database, for determining whether the data unit can also be written with data.

The data unit status information unit is used to record the available status of each data unit in the data storage space, and the address of the next available data unit.

The data management information is used for further identifying the data in each data unit, and comprises an ID of a sub-database to which the data stored in the data unit belongs, the number of data entries stored in the data unit, the number of the stored data in the sub-database to which the data belongs, and the like, and each data unit storing the data has corresponding data management information.

The data units are used for storing data, each data unit can store a plurality of pieces of data belonging to the same sub-database, and the data of different sub-databases cannot be stored in the same data unit.

The storage device in fig. 2 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), a compact disc read-only memory (compact disc read-only memory) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

Fig. 3 shows a manner of determining the data unit corresponding to the sub-database by three kinds of management information, that is, the mapping relationship between the three kinds of management information and the data unit.

In the mapping relationship shown in fig. 3, the data units are set to correspond to the data unit status information and the data management information one by one, that is, one data unit status information and one data management information correspond to one data unit, and are sequentially corresponding, and the address of the data unit corresponding to the data unit status information or the data management information can be determined by determining the order in which a certain data unit status information or data management information is located in all the data unit status information or the data management information.

Table 1 is a data structure of database index information provided in an embodiment of the present application.

Sub-database ID

Data unit address of last added data

In the data unit to which data was last addedData volume

Table 1 one possible data structure of sub-database index information

The sub-database ID in table 1 may also be implicitly deduced by the storage order of the index information itself, for example, the 1 st index information corresponds to the sub-database ID1, and the 2 nd index information corresponds to the sub-database ID2, i.e. the field of the sub-database ID may not be needed in the data structure.

It should be noted that in one embodiment of the present application, the sub-database index information may not be needed to find the available data units, and the available data units may be determined by traversing other management information, for specific implementation, see later embodiments.

Table 2 is a data structure of status information of a data unit according to an embodiment of the present application.

Address of data unit

Status of data units

Table 2 one possible data structure of data unit status information

The state of the data units in table 2 may be an available state, a used state, or a full storage space state, etc. As another implementation, the data unit status information may be implemented in a stack or a queue, for example, when initializing, all available data unit addresses are put in the stack, a certain data unit is used when adding data, which is equivalent to taking one data unit address out of the stack, and after deleting data, releasing one data unit, which is equivalent to putting the address of the data unit back in the stack. Thus, the field of "state of data unit" is not needed in the state information of data unit, and the state can be marked only by means of the storage mode of the state information of data unit.

Table 3 shows a data structure of data management information provided in an embodiment of the present application.

Sub-database ID

Number of data stored in data unit

Table 3 one possible data structure of data management information

The data management information is used for describing information of data stored in the corresponding data unit, such as a sub-database to which the data belongs, the number of the stored data, and the like.

As another implementation, the data management information may also be combined with the data unit status information, i.e. one piece of management information is used to record the status information of the data unit and the information of the currently stored data at the same time, and table 4 shows one possible data structure.

Address of data unit

Sub-database ID

Number of data stored in data unit

Table 4 another possible data structure of data management information

The data structure shown in table 4 may simultaneously represent the current status information of the data unit and the information of the stored data, and when the data unit does not store data yet, the sub-database ID field and the data number field stored in the data unit are both empty, and after the data unit stores data, the information of the stored data needs to be written into the sub-database ID and the data number field stored in the data unit.

Table 5 shows one possible data structure of a data unit.

Data offset

Data

Table 5 one possible data structure of data units

Wherein, the data offset is used to represent the position of the data in the sub-database, if the data in the sub-database is stored sequentially, only one initial offset can be used to represent the offsets of all the data in each data unit. As another implementation, the data offset may also be stored in the data management information.

The data structures shown in tables 1 to 5 are merely schematic data structures of management information and data units, and the fields therein may be increased or decreased or arranged according to need, which is not particularly limited in the embodiment of the present application. For example, the above possible data structure is based on the premise that the data units are set to correspond to the data unit state information and the data management information one by one in sequence, or the data units are not set to correspond to the data unit state information and the data management information one by one, at this time, the identification field of the data unit can be added to the data unit state information and the data management information, and the address of the data unit can be changed to a number or other identifications, so that the one-to-one correspondence between the data unit and the data unit state information and the data management information can be realized, that is, each piece of data unit state information describes the state of a unique data unit, and the data management information characterizes the information of the data stored in the corresponding data unit.

For example, as shown in fig. 4, the data management apparatus in fig. 2 may be implemented by the computer device in fig. 4.

Fig. 4 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present application. Computer device 400 includes at least one processor 401, a communication bus 402, a memory 403, and at least one communication interface 404.

Processor 401 may be a CPU, microprocessor, ASIC, or one or more integrated circuits for controlling the execution of the programs of the present application.

Communication bus 402 may include a path to transfer information between the aforementioned components.

The communication interface 404, using any transceiver or other device for communicating with other devices or communication networks, such as ethernet, radio access network (radio access network, RAN), wireless local area network (wireless local area networks, WLAN), etc., may be used to select a communication mode according to the actual application scenario, which is not limited by the present application.

The memory 403 may be, but is not limited to, a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, an EEPROM, a CD-ROM or other optical disk storage, optical disk storage (including compact disks, laser disks, optical disks, digital versatile disks, blu-ray disks, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory may be stand alone and coupled to the processor via a bus. The memory may also be integrated with the processor.

Wherein the memory 403 is used for storing application program codes for executing the inventive arrangements and is controlled to be executed by the processor 401. The processor 401 is configured to execute application code stored in the memory 403, thereby implementing a method for data management provided in the embodiments of the present application described below.

In a particular implementation, processor 401 may include one or more CPUs, such as CPU0 and CPU1 of FIG. 4, as an embodiment.

In a particular implementation, as one embodiment, computer device 400 may include multiple processors, such as processor 401 and processor 407 in FIG. 4. Each of these processors may be a single-core processor or a multi-core processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In a specific implementation, computer device 400 may also include an output device 405 and an input device 406, as one embodiment. The output device 405 communicates with the processor 401 and may display information in a variety of ways. For example, the output device 405 may be a liquid crystal display (liquid crystal display, LCD), a light emitting diode (light emitting diode, LED) display device, a Cathode Ray Tube (CRT) display device, or a projector (projector), or the like. The input device 406 is in communication with the processor 401 and may accept user input in a variety of ways. For example, the input device 406 may be a mouse, keyboard, touch screen device, or sensing device, among others.

The computer device 400 may be a general purpose computer device or a special purpose computer device. In particular implementations, computer device 400 may be a desktop, a laptop, a web server, an embedded device, or a device having a similar structure as in FIG. 4, or the like. Embodiments of the application are not limited to the type of computer device 400.

The data management method provided by the embodiment of the present application will be specifically described with reference to fig. 2 to 4.

As shown in fig. 5, a process of adding data in the data management method according to the embodiment of the present application includes the following steps:

s510, initializing the storage device.

Before the data management method provided by the embodiment of the application is executed, a storage device in a data management system needs to be initialized, and a space in the storage device is divided into a part for storing management information and a part for storing data. Specifically, the portion storing management information is divided into spaces storing three types of management information of sub-database index information, data unit state information, and data management information, respectively, and the three types of management information are set as initial values, and the portion storing data is divided into a plurality of data units which are smaller in space and independent of each other.

It should be noted that, the initializing step in step S510 is only performed when the data management system is first started or the data management system needs to be formatted, rather than performing the initializing step each time the sub-database is added or deleted.

S520, inquiring management information, and acquiring available data units for adding data.

When adding data in the sub-database, most importantly, the location of the available data unit of the added data is found, and there may be multiple methods for obtaining the available data unit according to the different data currently stored, and fig. 6 shows a flow for obtaining the available data unit.

As shown in fig. 6, when adding data, it is first necessary to query the sub-database index information to determine whether the sub-database has been added to the storage device. When the index information of the sub-database is inquired, two results exist, namely, the index information of the sub-database is stored in a storage device to mark the data added to the sub-database, and the index information of the sub-database is recorded with the information of the data unit of the last data of the sub-database at the moment, so that the position of the data unit to be added can be quickly indexed; and secondly, index information corresponding to the sub-database is not stored in a memory, and the state information of the data unit is required to be queried to acquire a new data unit as an available data unit, and the data of the sub-database is stored in the acquired data unit.

In addition, when adding the data of the sub-database, when updating the index information of the sub-database in the storage device, there may be a case that the data unit used for adding the data of the sub-database last time is full, and at this time, it is also necessary to query the status information of the data unit and acquire a new and available data unit.

As another implementation, when initializing the storage device, the management information of the storage device may not include the sub-database index information, and at this time, when adding the data of the sub-database, the available data units for adding the data may be acquired by traversing the data management information and the data unit state information.

Specifically, the data management information can be queried according to the identification of the sub-database to be added, and the data unit of the sub-database data added last time can be found. Although there may be a plurality of data units storing sub-database data, the sub-database data is stored using a new data unit only if the storage space of a certain data unit is full, so that the data unit to which the sub-database data is added last time refers to a data unit, the storage space of which is not full, in the data units storing the sub-database data to be added, and the data unit is taken as an available data unit.

When the data management unit is queried, no storage unit storing the sub-database data to be added is found (i.e. the sub-database data to be added is added to the storage device for the first time), or no data unit with a not-full storage space is found in the data units storing the sub-database data to be added (i.e. the storage space of the data unit of the last time the sub-database data to be added is full), the state information of the data units needs to be queried again, and a new data unit is acquired as an available data unit to be added with the data of the sub-database to be added.

S530, writing the data of the sub-database into the corresponding data unit.

When writing the data in the sub-database, in step S520, the available data units where the data in the sub-database can be written are obtained by query management, and the data to be stored can be written into the available data units.

S540, updating management information.

The data management information of the data unit includes information such as which sub-database the data unit stores belongs to, the position in the sub-database, and the number of data entries stored. After writing data into the available data units, the data management information corresponding to the available data units needs to be updated. After the data of the sub-database is written into the available data units, the corresponding sub-database index information needs to be updated, and the data unit information of the sub-database stored for the last time is written into the sub-database index information, so that the position of the writable data unit can be conveniently and quickly positioned when the data of the sub-database is stored for the next time. In addition, if the space of the data unit written with the data after the data is written with the data can not accommodate other data, the state information of the data unit corresponding to the data unit needs to be updated, and the state of the data unit is marked as a state with full storage space.

If the sub database is added for the first time, the index information of the sub database is also required to be created, and then the information of the data unit which is written into the sub database data for the last time is written into the index information. It should be noted that, the index information of the new sub-database may be created before the new data unit is acquired in step S520, and the embodiment of the present application does not limit the time for creating the index information of the sub-database.

As an implementation, if the management information does not include sub-database index information, only the data management information of the available data units needs to be updated at this time.

Fig. 7 is a schematic flow diagram of a data management method according to an embodiment of the present application when adding data from the interaction point of each management information and data unit.

As shown in fig. 7, when adding a sub-database, first, index information and data unit status information of the sub-database are queried to obtain an available data unit location for writing the sub-database data, and a specific method for obtaining the available data unit location is described in fig. 6, which is not repeated herein. After determining the available data unit, writing the data to be stored into the data unit, and updating management information corresponding to the data unit, including information of the data unit in the sub-database index information, data unit state information corresponding to the available data unit and data management information.

Fig. 8 shows a storage space utilization condition of storing data by using the data management method according to the embodiment of the present application, where an initialization mode is adopted in the embodiment that data unit status information, data management information and data units are set to be in one-to-one correspondence. As can be seen from fig. 8, the method according to the embodiment of the present application divides the storage space in the storage device into a plurality of data units with smaller space, the same sub-database may store a plurality of data units, and a new data unit may be used after one data unit is full, so that the storage space may be more fully utilized. In addition, as the information of the sub-databases and the data units is recorded in the management information, when a plurality of sub-databases are stored simultaneously, the data units for storing the data of the sub-databases can be discontinuous, and the data units adopted by each sub-database can be crossed without worrying about data confusion, so that the data storage is more flexible.

After the data of each sub-database is stored, when the data of a certain sub-database is required to be searched, the data management information is searched according to the label information or other ID information of the sub-database to be searched, the data unit storing the data of the sub-database to be searched is found, then the data in the data unit storing the data of the sub-database to be searched is sequentially searched, the data meeting the condition is found, the search result is returned, the returned result after the search is completed comprises but is not limited to the information such as the sub-database ID and the data offset, and the position of the search result in the sub-database can be determined.

According to the data management method provided by the embodiment of the application, the storage space is divided into a large number of independent data units with smaller capacity, and three management information are adopted to manage the data units and the data stored in the data units, so that a plurality of sub-databases can be continuously added in a staggered manner without worrying about the mutual confusion of the data of each sub-database. In the scheme, only less storage space is needed to store management information, so that the waste of storage space caused by setting fixed storage space for each sub-database in the prior art is avoided, and the utilization rate of the storage space is improved.

As shown in fig. 9, a process of deleting data in the data management method according to the embodiment of the present application includes the following steps:

s901, inquiring index information of the sub-database.

Before executing the operation of deleting the sub-database, the index information of the sub-database needs to be queried first to confirm that the sub-database to be deleted is stored in the storage device.

S902, clearing data management information corresponding to the sub-database to be deleted.

Inquiring data management information according to the identification information of the sub-database to be deleted, finding out the data management information corresponding to the data units storing the data of the sub-database to be deleted, and clearing the data management information, wherein the identification information can be the serial number of the sub-database. The data management information of the data unit is cleared, so that the data stored in the data unit lacks management information to be meaningless pure data, namely, the effect of deleting the data stored in the data unit is achieved.

S903, updating the status information of the data unit storing the sub-database data to be deleted.

After the data management information corresponding to the sub-database to be deleted is cleared, the data written in the corresponding data unit is virtually meaningless pure data due to the lack of management information, but the state of the data unit is still the state of the written data, and the data unit cannot be used at this time. Therefore, after the data management information corresponding to the sub-database to be deleted is cleared, the state information of the data unit needs to be further updated, and the state of the corresponding data unit is marked as an available state, so that new data can be written into the data unit.

S904, deleting sub-database index information corresponding to the sub-database to be deleted.

After steps S902 and S903 are performed, that is, the data unit storing the data of the sub-database to be deleted cannot be found in the storage device, the sub-database to be deleted is already "deleted" from the memory, and at this time, the index information of the sub-database to be deleted is also meaningless, and the index information of the sub-database to be deleted needs to be deleted according to the identification information of the sub-database to be deleted.

It should be noted that, the embodiment of the present application is not limited to the execution sequence of step S904 and step S902, and after the identification information of the sub-database to be deleted is obtained, the index information of the sub-database to be deleted may be selected to be deleted first, or the data management information corresponding to the sub-database to be deleted may be selected to be deleted first.

It should be noted that, although the data of the sub-database to be deleted actually stored in the data unit is not deleted in the above process, alternatively, after the data unit storing the data of the sub-database to be deleted is found, the data of the sub-database to be deleted actually stored in the data unit may be deleted, that is, the data stored in the data unit storing the data of the sub-database to be deleted may be deleted after step S902.

Fig. 10 shows how various management information is manipulated when deleting a sub database. As shown in fig. 10, the sub-database index information first needs to be queried to determine whether the sub-database to be deleted is stored or not before deleting the sub-database. When the sub-database to be deleted is stored in the memory, the data management information is firstly required to be queried to determine the data unit storing the data of the sub-database to be deleted, and the data management information corresponding to the sub-database to be deleted is cleared, namely the data unit is marked as not writing data. After the data management information corresponding to the sub-database to be deleted is cleared, the state information of the data unit corresponding to the data unit needs to be updated, the state of the data unit is marked as available, namely, the corresponding data unit is classified as available data unit, and the data unit can be used when the sub-database is stored next time.

The method for deleting data provided by the embodiment of the application does not need to delete the data actually stored in the data unit when deleting the sub-database, only needs to modify the management information, clear the corresponding data management information and update the state of the data unit, and re-put the data unit into the available resource pool. At this time, although the data stored in the data unit is not deleted, since the management information has been deleted, it is impossible to determine or identify which type of data or which structure the data stored in the data unit is, the data stored in the data unit becomes meaningless pure data so that it can be overwritten with new data. The method for deleting the data provided by the embodiment of the application only needs to modify the corresponding management information, does not need to operate the data unit, and improves the deleting efficiency.

The above description describes the method provided by the embodiment of the present application mainly from the interaction between the data management device and the storage device and the relationship between the management information and the data units in the storage device. It will be appreciated that the data management device and the storage device described above include corresponding hardware structures and/or software modules that perform the functions described above. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The embodiment of the application can divide the functional modules of the data management device according to the embodiment of the method, for example, each functional module can be divided for each function, two or more functions can be integrated into one processing module, and the integrated modules can be realized in a form of hardware or a form of a software functional module. It should be noted that, in the embodiment of the present application, the division of the modules is schematic, which is merely a logic function division, and other division manners may be implemented in actual implementation.

For example, fig. 11 shows a possible configuration diagram of the data management apparatus in the case where respective functional blocks are divided with respect to respective functions. The data management device includes a reading module 1110, a processing module 1120, and a writing module 1130. The reading module 1110 is configured to read storage device management information, where the management information includes sub-database index information, data unit status information, and data management information. The processing module 1120 is configured to determine an available data unit of the data of the sub-database to be added according to the management information, and the writing module 1130 is configured to write the data of the sub-database to be added into the available data unit, and update the management information in the storage device.

The data management apparatus 1100 further includes an initialization module 1140, where the initialization module 1140 is configured to initialize the storage apparatus, that is, divide a storable space of the storage apparatus into a space for storing management information and a space for storing data, where the space for storing data is divided into a plurality of data units.

The data management device provided by the embodiment of the application divides the storage space in the storage device into a plurality of data units with smaller space, manages the states of the data units and the stored data by using the management information, can store the same sub-database by using a plurality of data units, adopts new data units after one data unit is fully stored, and can more fully utilize the storage space. In addition, as the information of the sub-databases and the data units is recorded in the management information, when a plurality of sub-databases are stored simultaneously, the data units storing the data of the same sub-database can be discontinuous, and the data units adopted by all the sub-databases can be crossed without worrying about data confusion, so that the data storage is more flexible.

Optionally, the processing module 1120 is configured to query the sub-database index information, and when the sub-database index information includes the sub-database to be added, that is, the data of the sub-database to be added has been added in the storage device, take the data unit of the last added data in the index information of the sub-database to be added as the available data unit.

Optionally, in some cases, the storage space of the data unit used for adding the sub-database data to be added last time is full, and at this time, the processing module 1120 is further configured to query the status information of the data unit, and obtain an unused data unit as an available data unit.

Optionally, when the index information of the sub-database to be added does not include the index information of the sub-database to be added, the processing module 1120 needs to query the status information of the data unit to obtain an unused data unit as the available data unit.

According to the method provided by the embodiment of the application, the management information needs to be updated after each data addition, and the writing module 1130 needs to update the data management information corresponding to the available data units and the index information of the sub-database to be added. And the writing module updates information of data stored in the available data units according to the added data, namely, updates data management information corresponding to the available data units. The writing module 1130 also needs to update the index information of the sub-database to be added, such as the identifier of the data unit added last time, and the remaining storage space of the data unit, so as to be convenient for quickly finding out a suitable available data unit when adding next time.

Optionally, when the storage space is full after the available data unit is written into the data to be added, the writing module 1130 needs to update the state information of the data unit corresponding to the available data unit, and marks the state of the available data unit as the storage space is full.

When the sub-database needs to be deleted, the processing module 1120 is configured to query the data management unit according to the identifier of the sub-database to be deleted, obtain data management information to be removed, where the data management information to be removed is data management information corresponding to the data unit storing the information of the sub-database to be deleted, that is, the data management information to be removed includes the identifier of the sub-database to be deleted; the writing module 1130 is configured to clear the data management information to be cleared, update the state information of the data unit, that is, mark the state of the data unit corresponding to the data management information to be cleared as an available state, and delete the index information of the sub-database to be deleted. By using the data management device provided by the embodiment of the application, the data actually stored in the data unit is not required to be deleted when the sub-database is deleted, and only the corresponding management information is required to be modified, so that the deletion efficiency is obviously improved.

The specific implementation manner and the beneficial effects of the embodiment of the device correspond to those of the embodiment of the method, and refer to the related description of the embodiment of the method.

In the present embodiment, the data management apparatus is presented in the form of dividing each functional module corresponding to each function, or the data management system is presented in the form of dividing each functional module in an integrated manner. "module" as used herein may refer to an Application-specific integrated circuit (ASIC), a circuit, a processor and memory that execute one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the described functionality. In a simple embodiment, one skilled in the art will recognize that the data management system 110 may take the form shown in FIG. 4. For example, the read module 1110, the processing module 1120, the write module 1130, and the initialization module in fig. 11 may be implemented by the processor 401 and the memory 403 in fig. 4. In particular, the read module 1110, the processing module 1120, the write module 1130, and the initialization module may be executed by the processor 401 to invoke application code stored in the memory 403, which is not limiting in any way by the embodiments of the present application.

The embodiment of the application also provides a chip, which is connected with the memory, wherein the memory stores a computer program, and the chip is used for reading and executing the computer program stored in the memory so as to realize the functions of the data management device shown in fig. 2.

The embodiment of the present application also provides a computer storage medium storing program code which, when executed by a processor, is used to implement a method of the data management apparatus in the flow as shown in fig. 5 or fig. 9 in the present application.

Embodiments of the present application also provide a computer program product. The computer program product comprises computer software instructions that can be loaded by a processor to implement the method of the data management device in the flow shown in fig. 5 or fig. 9 of the present application.

Although the application is described herein in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus (device), or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects all generally referred to herein as a "module" or "system. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. A computer program may be stored/distributed on a suitable medium supplied together with or as part of other hardware, but may also take other forms, such as via the Internet or other wired or wireless telecommunication systems.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

Although the application has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, the specification and drawings are merely exemplary illustrations of the present application as defined in the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the application. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method of data management, the method comprising:

reading management information in a storage device, wherein the management information comprises sub-database index information, data unit state information and data management information; the data unit status information includes an available status of each data unit in the storage device; the data management information is used for identifying the data in each data unit;

querying the index information of the sub database;

when the index information of the sub-database comprises a sub-database to be added, taking a data unit of the last added data in the index information of the sub-database to be added as an available data unit; or alternatively, the first and second heat exchangers may be,

when the sub-database to be added is not included in the sub-database index information, inquiring the state information of the data unit, and acquiring an unused data unit as an available data unit;

writing the data of the sub-database to be added into the available data unit;

and updating the management information.

2. The method of claim 1, wherein prior to said reading the management information, the method further comprises dividing the storage device into a portion storing management information and a portion storing data, the portion storing data comprising a plurality of data units.

3. The method of claim 1, wherein when the storage space of the data unit to which the sub-database data to be added is added last time is insufficient, the method further comprises querying the data unit status information to obtain an unused data unit as the available data unit.

4. A method according to any one of claims 1 to 3, wherein updating the management information comprises:

updating data management information corresponding to the available data units;

and updating the index information of the sub database to be added.

5. The method of claim 4, wherein the method further comprises:

and after writing the data of the sub-database to be added, the storage space of the available data unit is full, updating the state information of the data unit corresponding to the available data unit, and marking the state of the available data unit as the full storage space.

6. A method according to any one of claims 1 to 3, wherein when deleting a sub-database, the method further comprises:

inquiring the data management information according to the identification of the sub-database to be deleted to obtain data management information to be cleared, wherein the data management information to be cleared comprises the identification of the sub-database to be deleted;

Clearing the data management information to be cleared, updating the state information of the data units, and marking the state of the data units corresponding to the data management information to be cleared as an available state;

and deleting the index information of the sub-database to be deleted.

7. A data management device, characterized in that the data management device comprises a reading module, a processing module and a writing module, wherein:

the reading module is used for reading management information in the storage device, wherein the management information comprises sub-database index information, data unit state information and data management information; the data unit status information includes an available status of each data unit in the storage device; the data management information is used for identifying the data in each data unit;

when the sub-database index information comprises a sub-database to be added, the processing module is used for taking a data unit of the last added data in the index information of the sub-database to be added as an available data unit; or alternatively, the first and second heat exchangers may be,

when the sub-database index information comprises the sub-database to be added, the processing module is used for taking a data unit of the last added data in the index information of the sub-database to be added as an available data unit;

The writing module is used for writing the data of the sub-database to be added into the available data unit;

the writing module is also used for updating the management information.

8. The apparatus of claim 7, further comprising an initialization module to divide the storage apparatus into a portion storing management information and a portion storing data, the portion storing data comprising a plurality of data units.

9. The apparatus of claim 7, wherein when the storage space of the data unit to which the sub-database data to be added is added last time is insufficient, the processing module is further configured to query the data unit status information to obtain an unused data unit as the available data unit.

10. The apparatus according to any one of claims 7 to 9, wherein:

the writing module is used for updating the data management information corresponding to the available data units;

the writing module is also used for updating the index information of the sub-database to be added.

11. The apparatus of claim 7, wherein the available data unit storage space is full after writing the sub-database data to be added, the writing module is further configured to update data unit status information corresponding to the available data unit, and mark the status of the available data unit as full storage space.

12. The apparatus according to any one of claims 7 to 9, wherein:

the processing module is used for inquiring the data management information according to the identification of the sub-database to be deleted to obtain data management information to be cleared, wherein the data management information to be cleared comprises the identification of the sub-database to be deleted;

the writing module is also used for clearing the data management information to be cleared, updating the state information of the data units and marking the state of the data units corresponding to the data management information to be cleared as an available state;

the writing module is also used for deleting the index information of the sub-database to be deleted.

13. A data processing apparatus, comprising: a processor and a memory;

the memory is used for storing computer-executable instructions, the processor and the memory are used for storing computer-executable instructions, the processor is connected with the memory through a bus, and when the data processing device runs, the processor executes the computer-executable instructions stored in the memory so as to enable the data processing device to execute the data management method according to any one of claims 1 to 6.

14. A data management system, characterized in that the system comprises a data management device according to any one of claims 7 to 12, and a storage device; or,

the system comprising the data processing device of claim 13, and a storage device.