WO2016155669A1 - Data storage method and device - Google Patents

Abstract

A data storage method and device. Data comprises a webpage abstract and index information about the webpage abstract. The method comprises: when a webpage abstract is updated, determining an update field in the webpage abstract and a field storage zone corresponding to the update field (101); and adding an update storage zone to the field storage zone, and storing the field data of the update field after the update at the present time and the index information about the field data in the update storage zone (102). In the technical solution, during storage of a webpage abstract, only an update field and corresponding index information are incrementally updated without the need to incrementally update the data of all fields, thereby greatly reducing the data volume stored during updating at a single time, avoiding an overly large newly added data volume and the occurrence of a full update caused thereby, saving time and overheads of storage space and improving storage efficiency.

Description

Data storage method and device Technical field

The present invention relates to the field of computer technologies, and in particular, to a data storage method and apparatus.

Background technique

When the search engine provides the search service for the user, the search information of the inverted index and the positive index is used to obtain the relevant information of the target webpage and provide the information to the user. The inverted index is an index structure from a keyword mapping to a plurality of sorted web pages, and the front index is an index structure that maps from a specific web page to summary information of the web page. When searching, the search engine first determines the search keyword according to the search sentence input by the user, and then retrieves related target web pages according to the search keyword and the inverted index, and sorts the target web pages, and then summarizes the webpage according to the positive index. And the original address of the webpage is provided to the user, and the user further decides whether to click the original address of the webpage to further browse the webpage according to the webpage summary.

The update speed of webpages on the Internet is very fast, and there are constantly updated webpage generations (in this article, newly generated webpages, webpages to be deleted, and webpages with updates are collectively referred to as update webpages). In order to provide users with more real-time and accurate search services, the search engine will use the crawler software to continuously search for updated web pages in the background, and store the summary of the updated webpages, and incrementally update the existing inverted index and the positive index, that is, in the original Some index structures are newly created based on the inverted index from the keyword to the updated web page, and the positive index from the updated web page to the corresponding web page summary.

The positive index is updated incrementally along with an incremental update of the stored page summary. A page summary consists of several fields, such as author, keyword, title, creation time, update time, page hits, and more. When any of the fields change, you need to re-save a full snippet containing all the fields and re-stor the corresponding index of the page. Usually, after several incremental updates of the positive index, the search efficiency is reduced due to the large amount of new data, so the full update is performed, that is, the entire data structure of the positive index is updated.

It can be seen that the existing web page summary storage method needs to incrementally store all the fields included in the web page summary when the web page summary is incrementally updated, so the amount of data stored in each incremental update is large. In addition, due to the faster frequency of web page updates, after several incremental updates, the full update of the web page summary and the positive index has to be performed, and the full amount of data involved is larger, which takes up a lot of time and Equipment resources.

Summary of the invention

The embodiment of the invention provides a data storage method and device, which solves the technical problem that the webpage digest storage method of the prior art has to be fully updated after several incremental updates, resulting in occupying a large amount of time and device resources. .

In order to solve the above technical problem, the embodiment of the present invention discloses the following technical solutions:

In one aspect, a data storage method is provided, the data including a webpage digest and index information of the webpage digest, the method comprising: determining an update field in the webpage digest when the webpage digest is updated, and the Updating a field storage area corresponding to the field; newly adding an update storage area in the field storage area, and storing, in the update storage area, field data of the update field after the current update and index information of the field data.

Further, the update storage area includes a data storage area and a corresponding index storage area, and the updated field data is stored in the data storage area, and the index information of the field data is stored in the index storage area. .

Further, the storing the index information of the field data in the index storage area includes: storing, in the index storage area, a webpage identifier corresponding to the field data, and the field data in the data storage area Store location information.

Further, the method further includes: newly adding a webpage index table, storing, in the webpage index table, a webpage identifier corresponding to the current update, and storage location information of the webpage identifier in the index storage area.

Further, the storing the webpage identifier corresponding to the current update in the webpage index table includes: setting 2 ^N index sub-tables in the webpage index table, and setting a corresponding N-bit binary table for each indexword table. A value of N is an integer that is greater than or equal to 1; a binary value corresponding to the identifier of the webpage is obtained, and the webpage identifier is stored in an index sub-table of the corresponding table value according to the first N digits of the binary value.

Further, the method further includes: presetting a plurality of field storage areas, and respectively designating one or more corresponding fields for each of the field storage areas.

Further, the specifying one or more fields for each field area respectively includes: updating an update frequency of each field included in the summary webpage, and specifying one or more corresponding fields for each field storage area according to the update frequency Fields.

Further, specifying one or more fields corresponding to each field storage area according to the update frequency includes: dividing fields with the same or similar update frequency into the same field storage area.

Further, the method further includes: determining whether there is a webpage to be deleted, if yes, setting a valid time of the webpage to be deleted in the newly added update storage area; and when the valid time is reached, the method is to be The deleted field data is marked as invalid in the field data and corresponding index information stored at each update.

Further, the method further includes: marking the history field data corresponding to the update field and the corresponding history index information in the history update storage area as invalid.

Further, the method further includes: merging a plurality of update storage areas included in the field storage area, and deleting field data marked as invalid and corresponding index information in the merged new update storage area.

Further, the merging the field storage area includes a plurality of update storage areas, including: storing in the field Selecting, in the area, a plurality of update storage areas to be merged; respectively calculating a sum of the quantity of valid field data included in the update storage area to be merged; and if the sum of the quantity is less than a first preset threshold, combining the to-be-combined Update store.

Further, the selecting, by the field storage area, a plurality of update storage areas to be merged comprises: separately calculating a quantity of valid field data included in each update storage area; and selecting the quantity of the valid field data from the field storage area The least number of update storage areas serve as the update storage area to be merged.

Further, the selecting, by the field storage area, the plurality of update storage areas to be merged comprises: respectively calculating a ratio of the number of valid field data included in the update storage area to the total field data quantity included in the update storage area; Selecting, in the field storage area, a plurality of update storage areas having the lowest ratio as the update storage area to be merged.

In another aspect, the present invention provides a data storage device, the data including a webpage digest and index information of the webpage digest, the apparatus comprising: a determining unit, configured to determine the webpage digest when the webpage digest is updated An update field in the field, and a field storage area corresponding to the update field; a first storage unit, configured to newly add an update storage area in the field storage area, and storing the update field in the update storage area in the current update Subsequent field data and index information of the field data.

Further, the update storage area includes a data storage area and a corresponding index storage area; the first storage unit includes: a data storage subunit and an index storage subunit, and the data storage subunit is configured to be in the data The storage area stores the field data of the current update; the index storage subunit is configured to store index information of the field data in the index storage area.

Further, the index storage subunit is configured to store, in the index storage area, a webpage identifier corresponding to the field data, and storage location information of the field data in the data storage area.

Further, the device further includes: a second storage unit, configured to newly add a webpage index table, and store, in the webpage index table, a webpage identifier corresponding to the current update, and the webpage identifier is in the index storage area Storage location information in .

Further, the second storage unit includes: a setting subunit, configured to set 2 ^N index sub-tables in the webpage index table, and set a corresponding N-bit binary table value for each index word table, where N is a pre- An integer of greater than or equal to 1 is set; a webpage storage subunit is configured to obtain a binary value corresponding to the identifier of the webpage, and store the webpage identifier into an index subtable of the corresponding table value according to the first N digits of the binary value.

Further, the device further includes: a setting unit, configured to preset a plurality of field storage areas, and respectively specify one or more corresponding fields for each field storage area.

Further, the setting unit is configured to: collect an update frequency of each field included in the webpage summary, and specify a corresponding one or more fields for each field storage area according to the update frequency.

Further, the device further includes: a determining setting unit, configured to determine whether there is a webpage to be deleted, and if yes, setting a valid time of the webpage to be deleted in the newly added update storage area; the first marking unit, After the valid time is reached, the field data and corresponding index information stored in the webpage to be deleted are marked as invalid.

Further, the device further includes: a second marking unit, configured to mark the historical field data corresponding to the update field and the corresponding historical index information in the history update storage area as invalid.

Further, the apparatus further includes: a merging unit, configured to merge the plurality of update storage areas included in the field storage area; and a deleting unit, configured to: in the merged new update storage area, the first marking unit and the first The two tag units are marked as invalid field data and index information is deleted.

Further, the merging unit includes: a first selecting subunit, configured to select, in the field storage area, a plurality of update storage areas to be merged; and a first calculating subunit, configured to separately calculate the update storage to be merged The sum of the number of valid field data included in the area; the first merge subunit, configured to merge the update storage areas to be merged if the sum of the quantities is less than a first preset threshold.

Further, the first selection subunit includes: a second calculation subunit, configured to separately calculate a quantity of valid field data included in each update storage area; and a second selection subunit, configured to select from the field storage area The plurality of update storage areas with the least number of valid field data are used as the update storage area to be merged.

Further, the first selection subunit includes: a third calculation subunit, configured to respectively calculate a ratio of the number of valid field data included in the update storage area to the total field data quantity included in the update storage area; And selecting a subunit, configured to select, in the field storage area, the plurality of update storage areas with the lowest ratio as the update storage area to be merged.

The foregoing technical solution provides a data storage method and apparatus, and when an update of a webpage is updated, determining an update field in the webpage summary and a field storage area corresponding to the update field; and newly adding an update storage area in the field storage area, The update storage area stores field data of the update field after the current update and index information of the field data.

Further, the setting unit is further configured to: divide the fields with the same or similar update frequency into the same field storage area.

The application also provides a computer terminal for executing the program code of the steps provided by the above data storage method.

The application also provides a storage medium for storing program code executed by the above data storage method.

When the webpage summary is stored, the technical solution only needs to update the update field and the corresponding index information in an incremental manner without incrementally updating the data of all the fields, thereby greatly reducing the number of times stored in a single update. According to the quantity, the amount of new data is too large, and the resulting full update occurs, saving time and storage space overhead and improving storage efficiency.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it will be apparent to those skilled in the art that In other words, other drawings can be obtained based on these drawings without paying for creative labor.

FIG. 1 is a schematic flowchart diagram of an embodiment of a data storage method according to the present invention;

2 is a schematic flowchart diagram of another embodiment of a data storage method according to the present invention;

FIG. 3 is a schematic diagram of a data storage structure corresponding to a data storage method according to the present invention; FIG.

4 is a schematic flowchart diagram of another embodiment of a data storage method according to the present invention;

FIG. 5 is a schematic structural diagram of an embodiment of a data storage device according to the present invention; FIG.

FIG. 6 is a schematic structural diagram of another embodiment of a data storage device according to the present invention; FIG.

FIG. 7 is a schematic structural diagram of another embodiment of a data storage device according to the present invention; FIG.

FIG. 8 is a schematic structural diagram of another embodiment of a data storage device according to the present invention; FIG.

FIG. 9 is a schematic structural diagram of an embodiment of a merging unit of a data storage device according to the present invention.

detailed description

First, the embodiment provided by the data storage method of the present invention is described. Referring to FIG. 1 , it is a schematic flowchart of an embodiment of the data storage method provided by the present invention. The embodiment includes the following steps:

Step 101: Determine, when the webpage summary is updated, an update field in the webpage summary, and a field storage area corresponding to the update field.

Referring to the related description of the background art, a web page summary usually includes multiple fields such as an author, a keyword, a body, a title, a creation time, an update time, and a web page click. In general, when a web page is updated, not all of the fields it contains are updated. For example, the possibility of updating the fields such as author, creation time, etc. is very small, and the possibility of updating the fields such as page hits and visitors is large, so You need to identify the updated fields in the snippet. For newly created pages and pages to be deleted, all fields contained in their snippet are considered to be update fields.

Step 102: Add an update storage area in the field storage area, and store, in the update storage area, field data of the update field after the current update and index information of the field data.

Each field storage area contains several update storage areas, and each update storage area is in each update page. When needed, the update store is added to the field store corresponding to the field in which the update occurred.

The index information is the positive index information from the webpage to the webpage summary field data. In the search, the inverted index is used to retrieve a plurality of target webpages related to the search keyword, and then the target is obtained according to the positive index information. Summary field data for the page.

In an actual application, the update storage period of the webpage summary may be preset, for example, may be set to one day, and at each preset update time, all the updated webpages in the previous day are counted, and the summary field in which the update occurs is The corresponding index information is stored.

The technical solution of the foregoing embodiment provides a data storage method and apparatus, and when an update of a webpage is updated, determining an update field in the webpage summary and a field storage area corresponding to the update field; and newly adding an update storage in the field storage area And storing, in the update storage area, field data of the update field after the current update and index information of the field data.

In this embodiment, when the webpage digest is stored, only the update field and the corresponding index information are incrementally updated, and the data of all the fields need not be incrementally updated, thereby greatly reducing the data stored in the single update. The amount, so as to avoid the excessive amount of new data, and the resulting full amount of updates, saving time and storage space overhead, improving storage efficiency.

Optionally, in other embodiments of the present invention, a plurality of field storage areas may be pre-established, and one or more fields are respectively designated for each field storage area.

Preferably, the update frequency of each field included in the webpage summary is pre-stated, and one or more fields corresponding to each field storage area are respectively specified according to the update frequency, and the fields with the same or similar update frequency may be divided into the same field. In the storage area, when the web page summary is updated, the update storage area can be newly added only in the field storage area where the updated field is located.

For example, according to the update frequency of each field, it can be divided into three field storage areas: stable storage area, non-changeable storage area, and variable storage area. Among them, the stable storage area corresponds to the author, keyword, creation time and other relatively stable fields, and it is not easy to change the storage area corresponding to the text, the title and the like, and the changeable time, the page click amount, etc. are more likely to occur. Changed field.

Those skilled in the art can also divide each field into different field storage areas according to experience or statistical data in actual operation to obtain higher update and storage efficiency. The division manner may be fixed, or may be periodically adjusted dynamically by statistical data within a certain period of time.

In another embodiment of the data storage method of the present invention, the update storage area may be specifically divided into two parts: a data storage area and a corresponding index storage area, and the updated field is stored in the data storage area. Data, storing index information of the field data in the index storage area.

The index information may specifically include a webpage identifier corresponding to the field data, and storage location information of the field data in the data storage area. Since the number of web pages updated each time is generally large, the number of field data stored in the data storage area is also large. When a certain field data is acquired, the field can be read from the corresponding index storage area. The index information of the data, thereby locating the piece of field data in the data storage area.

FIG. 2 is a schematic flowchart diagram of another embodiment of a data storage method according to the present invention. The embodiment includes the following steps 201 to 204:

Step 201: Determine, when the webpage summary is updated, an update field in the webpage summary, and a field storage area corresponding to the update field.

Step 202: Add an update storage area to the field storage area, where the update storage area includes a data storage area and a corresponding index storage area.

Step 203: Store the updated field data in the data storage area, store the webpage identifier corresponding to the field data in the index storage area, and store the field data in the data storage area. location information.

Step 204: The webpage index table is newly added, and the webpage identifier corresponding to the current update is stored in the webpage index table, and the storage location information of the webpage identifier in the index storage area.

The webpage identifier corresponding to the current update is stored in the newly added webpage index table, and the webpage identifier may be a URL address of the webpage, or other information that can be used to identify the webpage. The webpage storage table further stores storage location information of the webpage identifier in the index storage area, and is used to locate a target webpage identifier in the index storage area.

The order of execution of the above steps 203, 204 is not limited.

The step 204 may store the webpage identifier corresponding to the field data according to the following steps a) and b):

Step a): setting 2 ^N index sub-tables in the webpage index table, and setting a corresponding N-bit binary table value for each index word table, where N is an integer preset to be greater than or equal to 1.

Step b): Obtain a binary value corresponding to the webpage identifier, and store the webpage identifier into an index sub-table corresponding to the table value according to the first N digits of the binary value.

Since the number of webpages involved in each update is large, and the storage form of the webpage identifier in the computer is generally a binary number with a large number of digits, in order to quickly find the logo of the target webpage in the webpage index table, the webpage may be The index table is divided into 2 ^N index sub-tables, and each index word table corresponds to an N-bit binary table value. When the webpage identifier is stored, the webpage identifier may be stored in the index word table corresponding to the table value according to the first N-bit binary value of the webpage identifier. In this way, when searching for the webpage identifier, it is only necessary to search according to the first N-bit binary value of the webpage identifier to the index word table corresponding to the table value, which greatly saves the search time.

FIG. 3 is a schematic diagram of a data storage structure established by using the foregoing embodiment of the present invention. The number of web pages included in the Internet is large. We generally store the digest and index information of web pages with the same or similar hash values in the same index fragment. In Figure 3, a separate index fragment is used as an example. The index fragment includes a webpage index table, a field storage area, and a version information table.

The webpage index table stores a list of webpage identifiers involved in the current update, and storage location information of each webpage identifier in the newly added update storage area. The web page index table i is a newly added web page index table at the time of the i-th update. In an actual application, the update time information may be added to the name of the webpage index table. On the one hand, each different webpage index table may be distinguished, and on the other hand, it is convenient to immediately distinguish which one is the latest webpage index table from the name.

In FIG. 3, three field storage areas are included, which are a stable storage area, a non-volatile storage area, and a volatile storage area. Each field store contains several update buckets. For example, the update storage area i is an update storage area newly added in the non-changeable storage area at the ith update, and includes two parts of the data storage area i and the corresponding index storage area i. In practical applications, the update time information may also be added to the name of the update storage area, on the one hand, the different update storage areas may be distinguished, and on the other hand, it is convenient to immediately distinguish which one is the latest update storage area from the name.

The latest version information is recorded in the version information table, such as which web index table is included in the current index fragment, and which update storage area is included in each field storage area, so as to facilitate version management to ensure that the latest index can be updated. The index information gets the latest summary field data. The version information table i is a newly added version information table at the time of the ith update. In an actual application, the update time information may be added to the name of the version information table, and on the one hand, different version information tables may be distinguished, and another It is also convenient to immediately identify which one is the latest version information table from the name.

FIG. 4 is a schematic flowchart diagram of another embodiment of a data storage method according to the present invention. In the embodiment, steps 201 to 204 refer to descriptions of corresponding steps in the foregoing embodiment, and the embodiment is described. Also included are the following steps 205 to 208:

Step 205: Determine whether there is a webpage to be deleted, and if yes, set a valid time of the webpage to be deleted in the newly added update storage area.

Step 206: After the valid time is reached, the field data and the corresponding index information stored in the webpage to be deleted are marked as invalid.

In the technical solution of the present invention, the update of the webpage summary and the index is an incremental update, and is also applicable to the webpage to be deleted. If the webpage that needs to be deleted is detected during the update, all the fields of the webpage are considered to be update fields, and the update storage area is also added in the corresponding field storage area.

The updated field data is empty data. Therefore, in the data storage area, the updated field data may be replaced by a preset identifier, and the preset identifier is stored in the index storage area. Location letter Information and web page identification.

The inverted index and the positive index may have problems with the update time difference in actual use, such as the webpage to be deleted in this update, if each summary field of the webpage to be deleted is updated after the webpage summary and the corresponding positive index are updated. And the index information is deleted immediately, or marked as invalid, since the inverted index may not be completely updated at this time, that is, the webpage identifier to be deleted has not been deleted in the webpage list used by the inverted index, then it may be There will be an index requirement for the summary field of the web page to be deleted.

Therefore, in the embodiment, the “effective time” attribute is set for the webpage to be deleted, so that after the webpage summary and the corresponding index information are updated, the field data and the corresponding index information of the webpage to be deleted are still retained. After the expiration of the valid time, the field data and the corresponding index information stored in the webpage to be deleted are marked as invalid, so that the webpage list used in the inverted index can be guaranteed. The deleted webpage identifier has been deleted, and there is no longer an index requirement for the digest field of the webpage to be deleted, thereby solving the problem of the update time difference between the inverted index and the positive index.

The "valid time" attribute may be stored in the index storage area together with the web page identifier to be deleted. In order to keep the storage format consistent, for the web pages that do not need to be deleted, the "valid time" attribute may be retained in each update storage area, and the attribute value is set to be invalid, or an infinitely long time is set until it is detected. When the page needs to be deleted, the value of the attribute is actually set in the newly added update store.

Step 207: Mark the history field data corresponding to the update field and the corresponding history index information in the history update storage area as invalid.

Since the update of the webpage summary field of the present invention is an incremental update, it may happen that the plurality of update storage areas contain field data and corresponding index information updated by the same webpage at different times. At this time, the newly updated update storage area should be used as the valid update storage area corresponding to the webpage, and the field data and the corresponding historical index information included in the valid update storage area are valid field data and index information. The history field data corresponding to the update field and the corresponding history index information in the past history update storage area are all marked as invalid.

In practical applications, the update time information may be included in the file name corresponding to the update storage area, so that the file name of each update storage area can quickly determine which update storage area is a valid update storage area.

Step 208: Combine the plurality of update storage areas included in the field storage area, and delete the field data marked as invalid and the corresponding index information in the merged new update storage area.

As the web page summary is continuously updated, the number of update storage areas included in each field storage area is increasing, and the number of invalid field data and index information contained in each update storage area is also increasing. Excessive amount of update storage reduces the efficiency of the retrieval service, and invalid field data and index information wastes a lot of storage space. If the full amount of updates is performed, the amount of data involved is large. Therefore, in this embodiment, a plurality of update storage areas are merged, the number of updated storage areas is effectively reduced, and invalid field data and corresponding index information contained therein are removed, thereby saving Storage space, so that incremental updates continue, avoiding the time and equipment overhead caused by full updates.

Specifically, the foregoing step 208 may specifically include the following steps c), d), e):

Step c): selecting a plurality of update storage areas to be merged in the field storage area.

Step d): respectively calculating the sum of the number of valid field data included in the update storage area to be merged.

The number of valid field data referred to herein may be the number of valid fields or the memory value occupied by the valid field data.

Step e): If the sum of the quantities is less than the first preset threshold, merge the update storage areas to be merged.

Specifically, the upper limit threshold of the field data and/or the index information included in the update storage area may be set for each update storage area, for example, the field data corresponding to the maximum of 100 fields in each update storage area is set. Assuming that two update storage areas to be merged are selected, and the two update storage areas to be merged contain valid fields of 55 and 60, respectively, the number of valid fields included in the two update storage areas to be merged And more than 100, so the two update storage areas to be merged cannot be merged.

Optionally, the foregoing step 2081 may specifically include the following sub-steps:

Calculate the number of valid field data contained in each update store separately.

Selecting, from the field storage area, a plurality of update storage areas with the least number of valid field data as the update storage area to be merged.

When the update storage area to be merged is selected, a plurality of update storage areas with the least number of valid field data may be selected in a targeted manner, so that the selected update storage area to be merged is more likely to meet the merge condition of the above step 2083.

Optionally, the foregoing step 2081 may specifically include the following sub-steps:

The selecting, from the field storage area, a plurality of update storage areas to be merged.

Calculating, respectively, a ratio of the number of valid field data included in the update storage area to the total number of field data included in the update storage area.

Selecting, in the field storage area, a plurality of update storage areas having the lowest ratio as the update storage area to be merged.

When selecting the update storage area to be merged, a plurality of update storage areas with the least number of valid field data may be selected, or a plurality of update storage areas with the least proportion of valid field data may be selected, so that the selected update to be merged is selected. The storage area is more likely to meet the merge condition of the above step 2083, which saves time and space overhead.

The foregoing technical solution provides an embodiment of a data storage method, where an update field in the webpage summary and a field storage area corresponding to the update field are determined when the webpage summary is updated; and the update storage is newly added in the field storage area. a storage area in which the field data of the update field after the current update and the index information of the field data are stored in the update storage area.

When the webpage summary is stored, the technical solution only needs to incrementally update the update field and the corresponding index information, without incrementally updating the data of all the fields, thereby greatly reducing the data stored in a single update. The amount, so as to avoid the excessive amount of new data, and the resulting full amount of updates, saving time and storage space overhead, improving storage efficiency.

Corresponding to the embodiment provided by the data storage method of the present invention, the present invention further provides an embodiment of a data storage device, as shown in FIG. 5, which is a structure of an embodiment of a data storage device provided by the present invention. Schematically, the device comprises:

The determining unit 501 is configured to: when the webpage summary is updated, determine an update field in the webpage summary, and a field storage area corresponding to the update field;

The first storage unit 502 is configured to newly add an update storage area in the field storage area, and store, in the update storage area, field data of the update field after the current update and index information of the field data.

Optionally, the update storage area includes a data storage area and a corresponding index storage area;

The first storage unit 502 includes: a data storage subunit 5021 and an index storage subunit 5022;

The data storage subunit 5021 is specifically configured to store, in the data storage area, the field data after the current update;

The index storage subunit 5022 is specifically configured to store index information of the field data in the index storage area.

Optionally, the index storage unit 5022 is configured to store, in the index storage area, a webpage identifier corresponding to the field data, and storage location information of the field data in the data storage area.

FIG. 6 is a schematic structural diagram of another embodiment of a data storage device according to the present invention. The device further includes:

The second storage unit 503 is configured to newly add a webpage index table, and store, in the webpage index table, a webpage identifier corresponding to the current update, and storage location information of the webpage identifier in the index storage area.

Optionally, the second storage unit 503 includes:

The setting subunit 5031 is configured to set 2 ^N index sub-tables in the webpage index table, and set a corresponding N-bit binary table value for each index word table, where N is an integer preset to be greater than or equal to 1;

The webpage storage subunit 5032 is configured to obtain a binary value corresponding to the identifier of the webpage, and store the webpage identifier into an index subtable of the corresponding table value according to the first N digits of the binary value.

FIG. 7 is a schematic structural diagram of another embodiment of a data storage device according to the present invention. The device further includes:

The setting unit 504 is configured to preset a plurality of field storage areas, and respectively specify one or more corresponding fields for each field storage area.

Optionally, the setting unit 504 is specifically configured to:

The update frequency of each field included in the statistical webpage summary, and corresponding one or more fields are respectively designated for each field storage area according to the update frequency.

Optionally, as shown in FIG. 7, the apparatus further includes:

a determining setting unit 505, configured to determine whether there is a webpage to be deleted, and if yes, setting a valid time of the webpage to be deleted in the newly added update storage area;

The first marking unit 506 is configured to mark, when the valid time is reached, the field data and the corresponding index information stored in the webpage to be deleted at each update as invalid.

Optionally, as shown in FIG. 7, the apparatus further includes:

The second marking unit 507 is configured to mark the history field data corresponding to the update field and the corresponding history index information in the history update storage area as invalid.

FIG. 8 is a schematic structural diagram of another embodiment of a data storage device according to the present invention. The device further includes:

a merging unit 508, configured to merge the plurality of update storage areas included in the field storage area;

The deleting unit 509 is configured to delete the field data and the index information marked as invalid by the first marking unit and the second marking unit in the merged new update storage area.

FIG. 9 is a schematic structural diagram of an embodiment of a merging unit 508 of a data storage device according to the present invention. The merging unit 508 includes:

a first selection subunit 5081, configured to select, in the field storage area, a plurality of update storage areas to be merged;

a first calculating subunit 5082, configured to separately calculate a sum of the number of valid field data included in the update storage area to be merged;

The first merging sub-unit 5083 is configured to merge the update storage areas to be merged if the sum of the quantities is less than a first preset threshold.

Optionally, the first selection subunit 5081 includes:

a second calculating subunit 50811, configured to separately calculate a quantity of valid field data included in each update storage area;

The second selection sub-unit 50812 is configured to select, from the field storage area, a plurality of update storage areas with the least number of valid field data as the update storage area to be merged.

Optionally, the first selection subunit 5081 may also include:

a third calculating subunit (not shown) for respectively calculating a ratio of the number of valid field data included in the update storage area to the total field data quantity included in the update storage area;

And a third selection subunit (not shown) for selecting, in the field storage area, the plurality of update storage areas with the lowest ratio as the update storage area to be merged.

Here, it should be noted that each functional unit or subunit or module in each of the above embodiments may be operated in a computer terminal as part of the apparatus, and the above unit or subunit or module may be executed by a processor in the computer terminal. The functions realized, the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a mobile Internet device (MID), a PAD, and the like.

The embodiment of the data storage device provided by the present invention is essentially the same as the embodiment of the data storage method described above, and therefore is not explained in detail. For the related parts, reference may be made to the corresponding part of the embodiment of the data storage method.

The embodiment of the data storage device provided by the foregoing technical solution, when the webpage summary is updated, determining an update field in the webpage summary, and a field storage area corresponding to the update field; adding an update storage area in the field storage area And storing, in the update storage area, field data of the update field after the current update and index information of the field data.

In this embodiment, when the webpage digest is stored, only the update field and the corresponding index information are incrementally updated, and the data of all the fields need not be incrementally updated, thereby greatly reducing the data stored in the single update. The amount, so as to avoid the excessive amount of new data, and the resulting full amount of updates, saving time and storage space overhead, improving storage efficiency.

The various functional modules provided by the embodiments of the present application may be run in a mobile terminal, a computer terminal, or the like, or may be stored as part of a storage medium.

Thus, embodiments of the present invention may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.

In this embodiment, the computer terminal may execute the program code of the following steps in the data storage method: when the webpage summary is updated, determining an update field in the webpage summary, and storing a field corresponding to the update field And updating the storage area in the field storage area, and storing the field data of the update field after the current update and the index information of the field data in the update storage area.

Optionally, the computer terminal can include: one or more processors, memory, and transmission means.

The memory can be used to store software programs and modules, such as the data storage method and the program instructions/modules corresponding to the device in the embodiment of the present invention. The processor executes various functional applications by running software programs and modules stored in the memory. And data processing, that is, the above data storage method is implemented. The memory may include a high speed random access memory, and may also include non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, the memory can further include memory remotely located relative to the processor, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The above transmission device is for receiving or transmitting data via a network. Specific examples of the above network may include a wired network and a wireless network. In one example, the transmission device includes a Network Interface Controller (NIC) that can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In one example, the transmission device is a Radio Frequency (RF) module for communicating with the Internet wirelessly.

Specifically, the memory is used to store preset action conditions and information of the preset rights user, and an application.

The processor can call the memory stored information and the application by the transmitting device to execute the program code of the method steps of each of the alternative or preferred embodiments of the above method embodiments.

A person skilled in the art can understand that the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (MID), a PAD, and the like.

A person of ordinary skill in the art may understand that all or part of the steps of the foregoing embodiments may be completed by a program to instruct terminal device related hardware, and the program may be stored in a computer readable storage medium, and the storage medium may be Including: flash disk, read-only memory (ROM), random access memory (RAM), disk or optical disk.

Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the foregoing storage medium may be used to save program code executed by the data storage method provided by the foregoing method embodiment and the device embodiment.

Optionally, in this embodiment, the foregoing storage medium may be located in any one of the computer terminal groups in the computer network, or in any one of the mobile terminal groups.

Optionally, in the embodiment, the storage medium is arranged to store program code for performing the following steps: And updating, in the webpage summary, an update field in the webpage summary, and a field storage area corresponding to the update field; adding an update storage area in the field storage area, and storing the update field in the update storage area The field data after the update and the index information of the field data.

Alternatively, in the present embodiment, the storage medium may also be configured to store program code of various preferred or optional method steps provided by the data storage method.

The data storage method and apparatus according to the present invention are described above by way of example with reference to the accompanying drawings. However, it should be understood by those skilled in the art that various improvements can be made to the method and apparatus for obtaining a three-dimensional heat map proposed by the present invention without departing from the scope of the present invention. Therefore, the scope of the invention should be determined by the content of the appended claims.

It will be apparent to those skilled in the art that the technology in the embodiments of the present invention can be implemented by means of software plus necessary general hardware including general-purpose integrated circuits, general-purpose CPUs, general-purpose memories, general-purpose components, and the like. It can be implemented by dedicated hardware including an application specific integrated circuit, a dedicated CPU, a dedicated memory, a dedicated component, etc., but in many cases the former is a better implementation. Based on such understanding, the technical solution in the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product, which may be stored in a storage medium such as a read-only memory. (ROM, Read-Only Memory), Random Access Memory (RAM), disk, CD, etc., including a number of instructions to make a computer device (can be a personal computer, server, or network device, etc.) The methods described in various embodiments of the invention or in certain portions of the embodiments are performed.

The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the device and system embodiments, the description is relatively simple, since it is substantially similar to the method embodiment, and the relevant portions of the method embodiments can be referred to.

The embodiments of the invention described above are not intended to limit the scope of the invention. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (30)

1. A data storage method, the data comprising a webpage digest and index information of the webpage digest, wherein the method comprises:

   Determining, in the webpage summary update, an update field in the webpage summary, and a field storage area corresponding to the update field;

   Adding an update storage area to the field storage area;

   And storing, in the update storage area, field data of the update field after the current update and index information of the field data.
2. The method according to claim 1, wherein said update storage area comprises a data storage area and a corresponding index storage area, and said updated field data is stored in said data storage area, said index The storage area stores index information of the field data.
3. The method according to claim 2, wherein storing the index information of the field data in the index storage area comprises:

   And storing, in the index storage area, a webpage identifier corresponding to the field data, and storage location information of the field data in the data storage area.
4. The method of claim 3, wherein the method further comprises:

   A webpage index table is newly added, and the webpage identifier corresponding to the current update is stored in the webpage index table, and the storage location information of the webpage identifier in the index storage area.
5. The method according to claim 4, wherein storing the webpage identifier corresponding to the current update in the webpage index table comprises:

   Setting 2 ^N index sub-tables in the webpage index table, and setting a corresponding N-bit binary table value for each index word table, where N is an integer preset to be greater than or equal to 1;

   Obtaining a binary value corresponding to the webpage identifier;

   The web page identifier is stored in an index sub-table corresponding to the table value according to the first N bits of the binary value.
6. The method according to any one of claims 1 to 5, further comprising:

   A plurality of field storage areas are preset, and one or more fields are respectively designated for each field storage area.
7. The method of claim 6 wherein assigning each of the field regions a corresponding one or more fields comprises:

   The update frequency of each field included in the statistical webpage summary, and corresponding one or more fields are respectively designated for each field storage area according to the update frequency.
8. The method of claim 6 wherein the method further comprises:

   Determine whether there is a web page to be deleted;

   If yes, setting a valid time of the webpage to be deleted in the newly added update storage area;

   After the valid time is reached, the field data and the corresponding index information stored in the webpage to be deleted are marked as invalid.
9. The method according to any one of claims 1 to 5 or 7 to 8, wherein the method further comprises:

   The history field data corresponding to the update field and the corresponding history index information in the history update storage area are marked as invalid.
10. The method of claim 9 wherein the method further comprises:

    Merging a plurality of update storage areas included in the field storage area;

    Field data and index information marked as invalid are deleted in the merged new update store.
11. The method according to claim 10, wherein merging the plurality of update storage areas included in the field storage area comprises:

    Selecting a plurality of update storage areas to be merged in the field storage area;

    Calculating, respectively, a sum of the number of valid field data included in the update storage area to be merged;

    If the sum of the quantities is less than the first preset threshold, the update storage areas to be merged are merged.
12. The method according to claim 11, wherein selecting a plurality of update storage areas to be merged from the field storage area comprises:

    Calculating the number of valid field data included in each update storage area separately;

    Selecting, from the field storage area, a plurality of update storage areas with the least number of valid field data as the update storage area to be merged.
13. The method according to claim 11, wherein selecting a plurality of update storage areas to be merged from the field storage area comprises:

    Calculating, respectively, a ratio of a quantity of valid field data included in the update storage area to a total number of field data included in the update storage area;

    Selecting, in the field storage area, a plurality of update storage areas having the lowest ratio as the update storage area to be merged.
14. The method according to claim 7, wherein the specifying one or more fields for each field storage area according to the update frequency includes:

    Fields with the same or similar update frequency are divided into the same field storage area.
15. A data storage device, the data comprising a webpage digest and index information of the webpage digest, wherein the apparatus comprises:

    a determining unit, configured to determine an update field in the webpage summary and a field storage area corresponding to the update field when the webpage summary is updated;

    The first storage unit is configured to newly add an update storage area in the field storage area, and store, in the update storage area, field data of the update field after the current update and index information of the field data.
16. The device according to claim 15, wherein the update storage area comprises a data storage area and a corresponding index storage area, wherein the first storage unit comprises:

    The data storage subunit is configured to store, in the data storage area, the field data after the current update;

    The index storage subunit is configured to store index information of the field data in the index storage area.
17. The device according to claim 16, wherein the index storage subunit is configured to store, in the index storage area, a webpage identifier corresponding to the field data, and the field data in the data storage area. Store location information.
18. The device according to claim 17, wherein the device further comprises:

    a second storage unit, configured to newly add a webpage index table, and store, in the webpage index table, a webpage identifier corresponding to the current update, and storage location information of the webpage identifier in the index storage area.
19. The device of claim 18, wherein the second storage unit comprises:

    a setting subunit, configured to set 2 ^N index sub-tables in the webpage index table, and set a corresponding N-bit binary table value for each index word table, where N is an integer preset to be greater than or equal to 1;

    The webpage storage subunit is configured to obtain a binary value corresponding to the identifier of the webpage, and store the webpage identifier into an index subtable of the corresponding table value according to the first N digits of the binary value.
20. The device according to any one of claims 15 to 19, wherein the device further comprises:

    The setting unit is configured to preset a plurality of field storage areas, and respectively specify one or more corresponding fields for each field storage area.
21. The device according to claim 20, wherein the setting unit is further configured to:

    The update frequency of each field included in the statistical webpage summary, and corresponding one or more fields are respectively designated for each field storage area according to the update frequency.
22. The device of claim 21, wherein the device further comprises:

    a judging setting unit, configured to determine whether there is a webpage to be deleted, and if yes, setting a valid time of the webpage to be deleted in the newly added update storage area;

    The first marking unit is configured to mark, when the valid time is reached, the field data and the corresponding index information stored by the to-be-deleted webpage at each update as invalid.
23. Apparatus according to any one of claims 15 to 20 or 21 to 22, wherein said said The device also includes:

    And a second marking unit, configured to mark the historical field data corresponding to the update field and the corresponding historical index information in the history update storage area as invalid.
24. The device of claim 23, wherein the device further comprises:

    a merging unit, configured to merge a plurality of update storage areas included in the field storage area;

    And a deleting unit, configured to delete the field data and the index information marked as invalid by the first marking unit and the second marking unit in the merged new update storage area.
25. The apparatus according to claim 24, wherein said merging unit comprises:

    a first selection subunit, configured to select, in the field storage area, a plurality of update storage areas to be merged;

    a first calculating subunit, configured to separately calculate a sum of the quantity of valid field data included in the update storage area to be merged;

    a first merging subunit, configured to merge the update storage areas to be merged if the sum of the quantities is less than a first preset threshold.
26. The apparatus according to claim 25, wherein said first selection subunit comprises:

    a second calculating subunit, configured to separately calculate a quantity of valid field data included in each update storage area;

    And a second selection subunit, configured to select, from the field storage area, a plurality of update storage areas with the least number of valid field data as the update storage area to be merged.
27. The apparatus according to claim 25, wherein said first selection subunit comprises:

    a third calculating subunit, configured to separately calculate a ratio of a quantity of valid field data included in the update storage area to a total number of field data included in the update storage area;

    And a third selection subunit, configured to select, in the field storage area, a plurality of update storage areas with the lowest ratio as the update storage area to be merged.
28. The device according to claim 21, wherein the setting unit is further configured to:

    Fields with the same or similar update frequency are divided into the same field storage area.
29. A computer terminal for executing the program code of the steps provided by the data storage method of any one of claims 1 to 14.
30. A storage medium for storing program code executed by the data storage method according to any one of claims 1 to 14.

Images