CN107545011B - Data reading method and device - Google Patents
Data reading method and device Download PDFInfo
- Publication number
- CN107545011B CN107545011B CN201610499749.4A CN201610499749A CN107545011B CN 107545011 B CN107545011 B CN 107545011B CN 201610499749 A CN201610499749 A CN 201610499749A CN 107545011 B CN107545011 B CN 107545011B
- Authority
- CN
- China
- Prior art keywords
- subscript
- data
- reading
- traversed
- subscripts
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The embodiment of the invention discloses a data reading method and a data reading device, wherein subscripts in a subscript set of data to be read are traversed, and the subscripts in the subscript set are ordered from small to large; according to the technical scheme, when the traversed subscript does not meet the preset whole block reading condition, only the data to be read corresponding to the traversed subscript is read, and the useless data between the data to be read are not read, so that the reading time of the useless data is reduced, and the data reading efficiency is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data reading method and device.
Background
An array, that is, a collection of data (elements) of the same data type arranged in a certain order, is to name a limited number of variables of the same data type by a name, and then distinguish their collection of variables by numbers, where the name is called an array name and the number is called a subscript. In the prior art, when data in an array is read, a whole block reading mode is adopted. For example, the subscripts of the data to be read are 1000 and 100001, and the conventional way of reading the whole block is to read the data between the subscript 1000 and the subscript 100001 as a whole data block.
Therefore, when the subscripts of the data to be acquired are not continuous, according to a data reading method in the prior art, data corresponding to all the subscripts between the discontinuous subscripts are read, and all the read data located between the data to be acquired are useless data, the data reading method in the prior art reads a large amount of useless data, and the reading time of the useless data is inevitably increased, so that the data reading efficiency is reduced.
Disclosure of Invention
In view of this, embodiments of the present invention provide an efficient data reading method and apparatus.
In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:
a method of data reading, the method comprising:
traversing subscripts in a subscript set of data to be read, wherein the subscripts in the subscript set are ordered from small to large;
and judging whether the traversed subscript meets a preset whole block reading condition, if so, reading whole block data corresponding to the traversed subscript, and if not, reading data corresponding to the subscript.
Preferably, the judging whether the traversed subscript meets the preset whole block reading condition specifically includes:
judging whether the difference value of the currently traversed subscript and the smallest subscript in the traversed subscript exceeds a preset data length threshold, if so, judging whether the number of the traversed subscripts is greater than a preset subscript number threshold, wherein the traversed subscript does not comprise the currently traversed subscript;
and if the number of the traversed subscripts is larger than the preset whole block reading condition, the traversed subscripts are the subscripts meeting the preset whole block reading condition.
Preferably, the method further comprises:
and deleting the traversed subscript from the subscript set, judging whether the subscript set has the subscript which is not traversed or not, and returning to the step of traversing the subscript in the subscript set of the data to be read if the subscript set has the subscript which is not traversed.
Preferably, the reading of the whole block of data corresponding to the traversed subscript specifically includes:
and reading data corresponding to all indexes from the minimum index to the maximum index which are traversed.
Preferably, the traversing the subscript in the subscript set of the data to be read specifically includes:
and traversing the subscripts in the subscript set of the data to be read, grouping the subscripts in the subscript set to obtain a subscript subset, wherein the difference value between the maximum subscript and the minimum subscript in the subscript subset does not exceed a preset data length threshold, and the subscripts in the subscript set are sorted in the order from small to large.
Preferably, the judging whether the traversed subscript meets the preset whole block reading condition specifically includes:
judging whether the number of subscripts in the subscript subset is greater than a preset subscript number threshold;
and if the index is larger than the preset reading condition, the index in the index subset set is the index meeting the preset whole block reading condition.
Preferably, the reading of the whole block of data corresponding to the traversed subscript specifically includes:
and reading data corresponding to all subscripts from the minimum subscript to the maximum subscript in the subscript subset.
Preferably, the process of determining the preset data length threshold includes:
determining an upper limit value of an occupied memory in the process of reading data to be read;
determining a data length value of the data to be read;
and determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
A data reading apparatus, the apparatus comprising:
the subscript traversal module is used for traversing the subscripts in a subscript set of data to be read, and the subscripts in the subscript set are ordered from small to large;
the whole block reading condition judging module is used for judging whether the traversed subscript meets the preset whole block reading condition;
the whole block data reading module is used for reading the whole block data corresponding to the traversed subscript under the condition that the whole block reading condition judging module judges that the traversed subscript meets the preset whole block reading condition;
and the subscript data reading module is used for reading data corresponding to the subscript under the condition that the whole block reading condition judging module judges that the traversed subscript does not meet the preset whole block reading condition.
Preferably, the whole block reading condition determining module includes:
the difference value judging module is used for judging whether the difference value of the currently traversed subscript and the smallest subscript in the traversed subscript exceeds a preset data length threshold value or not;
the first subscript number judging module is used for judging whether the number of the traversed subscripts is larger than a preset subscript number threshold or not under the condition that the difference value judging module judges that the difference value between the currently traversed subscript and the smallest subscript in the traversed subscripts exceeds a preset data length threshold, wherein the traversed subscripts do not comprise the currently traversed subscripts;
and the first subscript determining module is used for determining that the traversed subscript meets the preset whole block reading condition under the condition that the subscript number judging module judges that the number of the traversed subscripts is greater than the preset subscript number threshold.
Preferably, the whole block data reading module includes:
and the first whole block data reading submodule is used for reading data corresponding to all subscripts from the traversed minimum subscript to the traversed maximum subscript.
Preferably, the subscript traversal module comprises:
the subscript traversal submodule is used for traversing the subscripts in the subscript set of the data to be read, and the subscripts in the subscript set are ordered from small to large;
and the grouping module is used for grouping the subscripts in the subscript set to obtain a subscript subset, and the difference value between the maximum subscript and the minimum subscript in the subscript subset does not exceed a preset data length threshold.
Preferably, the whole block reading condition determining module includes:
the second subscript number judging module is used for judging whether the number of subscripts in the subscript subset is larger than a preset subscript number threshold value or not;
and the second subscript determining module is used for determining that the subscript in the subscript subset is the subscript meeting the preset whole block reading condition under the condition that the second subscript number judging module judges that the number of the subscripts in the subscript subset is greater than the preset subscript number threshold.
Preferably, the whole block data reading module includes:
and the second whole data reading submodule is used for reading data corresponding to all subscripts from the minimum subscript to the maximum subscript in the subscript subset.
Preferably, the method further comprises the following steps: a data length threshold determination module, configured to determine a preset data length threshold, where the data length threshold determination module includes:
the memory upper limit value determining module is used for determining the upper limit value of the occupied memory in the process of reading the data to be read;
a data length value determining module, configured to determine a data length value of the data to be read;
and the data length threshold determining submodule is used for determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
Based on the technical scheme, the embodiment of the invention discloses a data reading method and a data reading device, wherein subscripts in a subscript set of data to be read are traversed, and the subscripts in the subscript set are ordered from small to large; according to the technical scheme, when the traversed subscript does not meet the preset whole block reading condition, only the data to be read corresponding to the traversed subscript is read, and the useless data between the data to be read are not read, so that the reading time of the useless data is reduced, and the data reading efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data reading method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data reading method according to an embodiment of the present invention;
FIG. 3 is a flowchart of another data reading method according to an embodiment of the present invention;
fig. 4 is a block diagram of a data reading apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of a whole block reading condition determining module according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The inventor of the present invention finds that, when subscripts of data to be acquired are not consecutive, according to a data reading method in the prior art, data corresponding to all subscripts between the discontinuous subscripts are read, and all the read data located between the data to be acquired are useless data, the data reading method in the prior art reads a large amount of useless data, and the reading time of the useless data is inevitably increased, so that the data reading efficiency is reduced.
Therefore, how to provide an efficient data reading method is an urgent problem to be solved at present.
In order to solve the above problems, the technical solution disclosed in the embodiments of the present invention provides a data reading method, in which subscripts in a subscript set of data to be read are traversed, and the subscripts in the subscript set are ordered from small to large; according to the technical scheme, when the traversed subscript does not meet the preset whole block reading condition, only the data to be read corresponding to the traversed subscript is read, and the useless data between the data to be read are not read, so that the reading time of the useless data is reduced, and the data reading efficiency is improved.
Fig. 1 is a flowchart of a data reading method according to an embodiment of the present invention, where the method may include:
s101: traversing subscripts in a subscript set of data to be read;
it should be noted that the subscript of the data to be acquired represents the array number of the data to be acquired, and the subscripts in the subscript set are sorted in the order from small to large.
S102: judging whether the traversed subscript meets a preset whole block reading condition, if so, executing S103; otherwise, executing S104;
wherein, the preset whole data reading condition is specifically as follows: the difference value between the maximum subscript and the minimum subscript in the data block read in the whole block each time does not exceed a preset data length threshold, and the number of the subscripts in the data block read in the whole block is larger than a preset subscript number threshold. Two specific implementation schemes for determining whether the preset whole block data reading condition is met are provided in the present invention, and will be described in the following two embodiments.
S103: reading whole data corresponding to the traversed subscript;
it should be noted that, in reading the whole block of data corresponding to the traversed subscript, that is, in one reading process, reading the data corresponding to all the subscripts between the traversed minimum subscript and the traversed maximum subscript, that is, reading one whole block of data at one time, compared with a mode of reading data corresponding to one subscript at a time, all the data to be read can be obtained only through one reading operation, and the data reading efficiency is improved.
Optionally, in the embodiment of the present invention, after reading the whole block of data corresponding to the traversed subscript, the method further includes: and deleting the data with subscripts not corresponding to the subscripts in the group of subscript sets from the read whole data to obtain the data to be acquired, so that the occupation of the memory is reduced.
S104: and reading data corresponding to the subscript.
It should be noted that, in this step, only the data corresponding to the subscript in the subscript set is read, and the reading mode is a single data reading mode, that is, the data to be read corresponding to one subscript is read at a time until all the data to be read are obtained, so that it can be ensured that, in the case that the proportion of the data amount to be read in all the data amounts is small, that is, in the case that the data amount to be read is small, the data to be read corresponding to the subscript in the group is read only by using a single data reading mode, and all the data do not need to be read, thereby reducing waste on the memory.
According to the technical scheme, the subscripts in the subscript set of the data to be read are traversed, and the subscripts in the subscript set are ordered from small to large; according to the technical scheme, when the traversed subscript does not meet the preset whole block reading condition, only the data to be read corresponding to the traversed subscript is read, and the useless data between the data to be read are not read, so that the reading time of the useless data is reduced, and the data reading efficiency is improved.
Optionally, fig. 2 shows a flowchart of another data reading method provided in an embodiment of the present invention, where the method may include:
s201: traversing subscripts in a subscript set of data to be read;
the subscripts in the subscript set are sorted in the order from small to large;
s202: judging whether the difference value of the currently traversed subscript and the minimum subscript in the traversed subscript exceeds a preset data length threshold value or not; if yes, go to S203;
the preset data length threshold is the maximum data length which can be read in one data reading process.
The preset data length threshold is determined according to an upper limit value of an occupied memory in the data reading process and a data length value of data to be read, and the specific process is as follows: determining an upper limit value of an occupied memory in the process of reading data to be read; determining a data length value of the data to be read; and determining the preset data length threshold according to the upper limit value of the memory occupied in the process of reading the data to be read and the data length value of the data to be read, wherein the upper limit value of the memory occupied in the data reading process in the embodiment of the invention refers to the upper limit of the memory occupied by a data reading module for reading the data. The specific implementation of determining the data length threshold herein applies to all embodiments of the present invention.
For example, since the data reading process is limited by the upper limit of the memory of the data reading module, the size of the preset data length threshold needs to be dynamically determined according to the upper limit of the memory of the data reading module and the data length value of the data to be read, and it is assumed that the upper limit of the memory usage of the data reading module is 1Mb (i.e., 1024 × 1024 — 1048576 bytes); if the size of the data to be read is 100 bytes, the preset data length threshold value is 1048576/100 10485.76, which is approximately 10000, and the preset data length threshold value is 10000 at this time; similarly, assuming that the upper limit value of the memory occupied by reading the data to be acquired is 2Mb, if the size of the data to be read is 500 bytes, the preset data length threshold is 1024 × 2/500 — 4194.304, and the preset data length threshold takes the value of 4000 at this time.
S203: judging whether the number of traversed subscripts is greater than a preset subscript number threshold; if yes, executing S204; otherwise, executing S205;
the traversed subscripts do not include the currently traversed subscript.
It should be noted that, the way of determining whether the number of traversed subscripts is greater than the preset subscript number threshold is used to determine whether to read the whole block of data or the single data, so that the size of the preset subscript number threshold nMin is related to the size of the preset data length threshold nMax, and when the data is read according to the whole block, the average reading time of each data is related to the multiple of the reading time of reading the data corresponding to one subscript at a time. Assuming that the reading time for reading one data at a time is w times of the average reading time of each data in the whole block reading, the reading time for reading one data at a time is t, and the average reading time of each data in the whole block reading is (t/w); then nMin ═ (nMax ═ t/w))/t; the simplification is as follows: nMin is nMax/w.
S204: reading data corresponding to all subscripts from the traversed minimum subscript to the traversed maximum subscript;
s205: reading data corresponding to the traversed subscript;
s206: deleting the traversed subscript from the set of subscripts;
s207: and judging whether the subscript set has a subscript which is not traversed, if so, returning to the step S201, and otherwise, ending the process.
The following describes in detail a specific process of the above data reading method provided by the embodiment of the present invention by using a specific example.
For example, there is a set of data in which 100 pieces of data are recorded. From these 100 ten thousand pieces of data, a subscript set of data to be read is now determined for reading the required data. Assuming that the determined subscript set of the data to be read is: {1000, 1010, 1011, 1050, 1100, 1101, 1102, 1105, 1107, 1200, 1250, 2000, 3000, 3010, 200000, 201000, 500000, 500002, 500010, 500011, 500012, 500100, 500200, 500201, 501000, 501002, 502000, 700001, 700005, 700010}, which contain 30 subscripts, the subscripts in the set of subscripts being arranged in order of decreasing size to increasing size.
Presetting a preset data length threshold nMax to 5000 in the initialization process; the preset subscript number threshold is nMin-10;
setting the ith subscript nStart in the subscript set as i, where i is greater than or equal to 0, and is less than or equal to 29, and 29 is the total number of subscripts in the subscript set, where the start subscript vnDataIdx [ nStart ] of the data reading of this time is 1000, that is, the first element value of the data subscript set to be read is the start subscript of the data read for the first time;
sequentially traversing each subscript vnDataIdx [ i ] in the subscript set, judging whether the difference value between the currently traversed subscript and the smallest subscript in the traversed subscript exceeds nMax, judging that when the currently traversed subscript is vnDataIdx [14] ═ 200000, the difference value between the currently traversed subscript and the smallest subscript vnDataIdx [0] ═ 1000 in the traversed subscript exceeds nMax, judging that the number 14 of the traversed subscripts is greater than nMin, reading data corresponding to all the subscripts between the data subscripts corresponding to the 1 st to 14 th elements, namely reading data corresponding to all 2011 subscripts between the subscripts 1000 to the subscript 3010;
as the subscript set still has the non-traversed subscript, a new traversal judgment process is restarted, when the currently traversed subscript is vnDataIdx [16] ═ 500000, the difference between the currently traversed subscript and the smallest subscript vnDataIdx [14] ═ 200000 in the traversed subscripts exceeds nMax, and the number 2 of the traversed subscripts is judged to be less than nMin, that is, the data corresponding to the 15 th to 16 th element data subscripts are obtained, that is, the data corresponding to the 200000 and 201000 subscripts are read;
by analogy, data corresponding to all subscripts between data subscripts corresponding to 17 th to 27 th elements is obtained for the third time, that is, data corresponding to all 2001 subscripts between the subscript 500000 and the subscript 502000 are read; the data corresponding to the 28 th to 30 th element data indexes, that is, the data corresponding to the 700001, 700005, and 700010 th indexes are read, are acquired for the fourth time.
Optionally, fig. 3 shows a flowchart of another data reading method provided in an embodiment of the present invention, where the method may include:
s301: traversing subscripts in a subscript set of data to be read, and grouping the subscripts in the subscript set to obtain a subscript subset, wherein the subscripts in the subscript set are ordered from small to large;
the difference value between the maximum subscript and the minimum subscript in the subscript subset does not exceed a preset data length threshold, wherein the preset length threshold is the maximum data length which can be read in one data reading process.
In the embodiment of the present invention, the process of grouping the subscripts in the subscript set to obtain a subscript subset includes:
extracting the subscript ordered at the first position in the subscript set as a target subscript;
sequentially calculating whether the difference value between other subscripts arranged behind the target subscript and the target subscript is smaller than or equal to a preset data length threshold value or not according to the arrangement sequence of the subscripts in the subscript set, and taking the previous subscript of the currently traversed subscript as a termination subscript under the condition that the difference value between the currently traversed subscript and the target subscript is larger than the preset data length threshold value;
grouping all subscripts from the target subscript to the termination subscript in the set of subscripts as a set of subscript subsets;
judging whether the subscript set has the remaining subscripts which are not divided into the subscript subset, if so, extracting the subscript which is sorted at the first position in the remaining subscripts as a new target subscript, and returning to execute the step of calculating whether the difference value between the other subscripts which are arranged behind the target subscript and the target subscript is less than or equal to a first preset value in sequence according to the arrangement sequence of the subscripts in the subscript set; if not, the flow is ended.
In the embodiment of the invention, after the subscripts in the subscript set are grouped, the data to be read corresponding to the subscripts in the subscript set are read according to the groups, so that the upper limit of the memory occupied by the data reading process cannot be exceeded in the process of reading each group of data, and the data reading process cannot be limited by the memory.
S302: judging whether the number of subscripts in the subscript subset is greater than a preset subscript number threshold, if so, executing S303, otherwise, executing S304;
s303: reading data corresponding to all subscripts from the minimum subscript to the maximum subscript in the subscript subset;
s304: and reading data corresponding to the subscripts in the subscript subset.
The following describes in detail a specific process of the above data reading method provided by the embodiment of the present invention by using a specific example.
For example, there is a set of data in which 100 pieces of data are recorded. From these 100 ten thousand pieces of data, a subscript set of data to be read is now determined for reading the required data. Assuming that the determined subscript set of the data to be read is: {1000, 1010, 1011, 1050, 1100, 1101, 1102, 1105, 1107, 1200, 1250, 2000, 3000, 3010, 200000, 201000, 500000, 500002, 500010, 500011, 500012, 500100, 500200, 500201, 501000, 501002, 502000, 700001, 700005, 700010}, which contain 30 subscripts, the subscripts in the set of subscripts being arranged in order of decreasing size to increasing size.
Presetting a preset data length threshold nMax to 5000 in the initialization process; the preset subscript number threshold is nMin-10;
setting the ith subscript nStart in the subscript set to be greater than or equal to 0, i being less than or equal to 29, where the starting subscript vnDataIdx [ nStart ] of the data reading of this time is 1000, that is, the first element value of the data subscript set to be read is the starting subscript of the data read for the first time;
traversing each subscript vnDataIdx [ i ] in the subscript set in sequence, grouping the subscripts in the subscript set to obtain a subscript subset, wherein the difference between the largest subscript and the smallest subscript in the subscript subset does not exceed nMax, and it can be seen that when i ═ 13 (i.e. the 14 th subscript in the subscript set), vnDataIdx [ i ] -vnDataIdx [ nStart ] is less than nMax, i.e. 3010-. Then the 1 st element to the 14 th element are divided into the same subscript subset, and so on, and the obtained subscript subset is as follows: {1000, 1010, 1011, 1050, 1100, 1101, 1102, 1105, 1107, 1200, 1250, 2000, 3000, 3010}, {200000, 201000}, {500000, 500002, 500010, 500011, 500012, 500100, 500200, 500201, 501000, 501002, 502000}, {700001, 700005, 700010 };
judging whether the number of the subscripts in each subscript subset is greater than nMIN, if so, reading data corresponding to all the subscripts between the minimum subscript and the maximum subscript in the subscript subset; otherwise, reading data corresponding to the subscript in the subscript subset. The judgment shows that the number of the subscripts in the first subscript subset is 14, which is greater than nMin, and data corresponding to all the subscripts between the data subscripts corresponding to the 1 st element and the 14 th element are read for the first time, that is, data corresponding to all 2011 subscripts between the subscripts 1000 and the subscript 3010 are read;
the number of the subscripts in the second subscript subset is 2 and is less than nMin, and data corresponding to the 15 th to 16 th element data subscripts are acquired for the second time, namely data corresponding to the 200000 and 201000 subscripts are read;
the number of the subscripts in the third subscript subset is 11 and is greater than nMin, and data corresponding to all the subscripts between the data subscripts corresponding to the 17 th element and the 27 th element is acquired for the third time, namely data corresponding to all 2001 subscripts between the subscripts 500000 and the subscript 502000 are read;
the number of subscripts in the fourth subscript subset is 3, which is smaller than nMin, and the data corresponding to the 28 th to 30 th element data subscripts, that is, the data corresponding to the reading 700001, 700005, and 700010 subscripts, are acquired for the fourth time.
In the above process of reading data, if n units (where n is a larger value, such as more than 1000) of data are read each time, the average reading time of each unit of data is about five percent of the reading time of one unit of data at a time, and assuming that the reading time of one unit of data at a time is 1t, the average reading time of one unit of data is 0.002t when the data is read in a whole block. Then when nMax is 5000, the maximum read time to read the entire block of data once is 0.002t 5000 ═ 10t, then for the index array of data to be acquired:
1. according to the method of reading one unit of data each time, the reading time is as follows: 30 x 1t ═ 30 t;
2. according to the method for reading data four times, the reading time is as follows: 10t 2+ 5t 1t 25 t;
therefore, the method for reading data four times disclosed by the embodiment of the invention has higher efficiency compared with the method for reading one unit of data each time: (30 t-25 t)/30t ═ 16.67%. Therefore, the data reading method adopted by the embodiment of the invention has higher efficiency than the method for reading one unit of data at a time.
The technical scheme in the embodiment of the invention is that subscripts in a subscript set of data to be read are traversed, and the subscripts in the subscript set are ordered from small to large; according to the technical scheme, when the traversed subscript does not meet the preset whole block reading condition, only the data to be read corresponding to the traversed subscript is read, and the useless data between the data to be read are not read, so that the reading time of the useless data is reduced, and the data reading efficiency is improved.
The method is described in detail in the embodiments disclosed above, and the method of the present invention can be implemented by various types of apparatuses, so that the present invention also discloses an apparatus, and the following detailed description will be given of specific embodiments.
Fig. 4 is a block diagram of a data reading apparatus according to an embodiment of the present invention, where the apparatus may include:
the subscript traversing module 11 is configured to traverse subscripts in a subscript set of data to be read, where the subscripts in the subscript set are sorted in a descending order;
the whole block reading condition judging module 12 is used for judging whether the traversed subscript meets the preset whole block reading condition;
the whole block data reading module 13 is configured to read the whole block data corresponding to the traversed subscript when the whole block reading condition determining module determines that the traversed subscript meets the preset whole block reading condition;
and the subscript data reading module 14 is configured to read data corresponding to the subscript when the whole block reading condition determining module determines that the traversed subscript does not meet the preset whole block reading condition.
Based on the above data reading device embodiment, the present invention may further select a suitable unit from the following units to form a new data reading device, and the specific forming manner may be determined according to the related description in the method embodiment, which is not described in detail in this embodiment.
As shown in fig. 5, the whole block reading condition determining module 12 includes:
a difference value determining module 121, configured to determine whether a difference value between the currently traversed subscript and a smallest subscript in the traversed subscripts exceeds a preset data length threshold;
a first subscript number judging module 122, configured to, when the difference judging module judges that the difference between the currently traversed subscript and the smallest subscript in the traversed subscript exceeds the preset data length threshold, judge whether the number of the traversed subscripts is greater than a preset subscript number threshold, where the traversed subscript does not include the currently traversed subscript;
a first subscript determining module 123, configured to determine that the traversed subscript is a subscript that meets a preset whole block reading condition when the subscript number judging module judges that the number of the traversed subscripts is greater than a preset subscript number threshold.
The whole block data reading module comprises:
and the first whole block data reading submodule is used for reading data corresponding to all subscripts from the traversed minimum subscript to the traversed maximum subscript.
The subscript traversal module comprises:
the subscript traversal submodule is used for traversing the subscripts in the subscript set of the data to be read, and the subscripts in the subscript set are ordered from small to large;
and the grouping module is used for grouping the subscripts in the subscript set to obtain a subscript subset, and the difference value between the maximum subscript and the minimum subscript in the subscript subset does not exceed a preset data length threshold.
The whole block reading condition judgment module comprises:
the second subscript number judging module is used for judging whether the number of subscripts in the subscript subset is larger than a preset subscript number threshold value or not;
and the second subscript determining module is used for determining that the subscript in the subscript subset is the subscript meeting the preset whole block reading condition under the condition that the second subscript number judging module judges that the number of the subscripts in the subscript subset is greater than the preset subscript number threshold.
The whole block data reading module comprises:
and the second whole data reading submodule is used for reading data corresponding to all subscripts from the minimum subscript to the maximum subscript in the subscript subset.
Optionally, the method further includes: a data length threshold determination module, configured to determine a preset data length threshold, where the data length threshold determination module includes:
the memory upper limit value determining module is used for determining the upper limit value of the occupied memory in the process of reading the data to be read;
a data length value determining module, configured to determine a data length value of the data to be read;
and the data length threshold determining submodule is used for determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
In summary, the following steps:
the embodiment of the invention discloses a data reading method and a data reading device, wherein subscripts in a subscript set of data to be read are traversed, and the subscripts in the subscript set are ordered from small to large; according to the technical scheme, when the traversed subscript does not meet the preset whole block reading condition, only the data to be read corresponding to the traversed subscript is read, and the useless data between the data to be read are not read, so that the reading time of the useless data is reduced, and the data reading efficiency is improved.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. A method of reading data, the method comprising:
traversing subscripts in a subscript set of data to be read, wherein the subscripts in the subscript set are ordered from small to large;
judging whether the traversed subscript meets a preset whole block reading condition, if so, reading whole block data corresponding to the traversed subscript, and if not, reading data corresponding to the subscript;
deleting the traversed subscript from the subscript set, judging whether the subscript set has the subscript which is not traversed or not, if so, returning to the step of traversing the subscript in the subscript set of the data to be read,
the step of judging whether the traversed subscript meets the preset whole block reading condition specifically comprises the following steps:
judging whether the difference value of the currently traversed subscript and the smallest subscript in the traversed subscript exceeds a preset data length threshold, if so, judging whether the number of the traversed subscripts is greater than a preset subscript number threshold, wherein the traversed subscript does not comprise the currently traversed subscript;
and if the number of the traversed subscripts is larger than the preset whole block reading condition, the traversed subscripts are the subscripts meeting the preset whole block reading condition.
2. The method of claim 1, wherein reading the entire block of data corresponding to the traversed index specifically comprises:
and reading data corresponding to all indexes from the minimum index to the maximum index which are traversed.
3. The method of claim 1, wherein the determining of the preset data length threshold comprises:
determining an upper limit value of an occupied memory in the process of reading data to be read;
determining a data length value of the data to be read;
and determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
4. A method of reading data, the method comprising:
traversing subscripts in a subscript set of data to be read, wherein the subscripts in the subscript set are ordered from small to large;
judging whether the traversed subscript meets the preset whole block reading condition, if so, reading the whole block data corresponding to the traversed subscript, if not, reading the data corresponding to the subscript,
the traversing subscripts in the subscript set of the data to be read specifically includes:
traversing subscripts in a subscript set of data to be read, grouping the subscripts in the subscript set to obtain a subscript subset, wherein the difference value between the maximum subscript and the minimum subscript in the subscript subset does not exceed a preset data length threshold value, the subscripts in the subscript set are sorted in the order from small to large,
the step of judging whether the traversed subscript meets the preset whole block reading condition specifically comprises the following steps:
judging whether the number of subscripts in the subscript subset is greater than a preset subscript number threshold;
and if the index is larger than the preset reading condition, the index in the index subset set is the index meeting the preset whole block reading condition.
5. The method of claim 4, wherein reading the entire block of data corresponding to the traversed index specifically comprises:
and reading data corresponding to all subscripts from the minimum subscript to the maximum subscript in the subscript subset.
6. The method of claim 4, wherein the determining of the preset data length threshold comprises:
determining an upper limit value of an occupied memory in the process of reading data to be read;
determining a data length value of the data to be read;
and determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
7. A data reading apparatus, characterized in that the apparatus comprises:
the subscript traversal module is used for traversing the subscripts in a subscript set of data to be read, and the subscripts in the subscript set are ordered from small to large;
the whole block reading condition judging module is used for judging whether the traversed subscript meets the preset whole block reading condition;
the whole block data reading module is used for reading the whole block data corresponding to the traversed subscript under the condition that the whole block reading condition judging module judges that the traversed subscript meets the preset whole block reading condition;
the subscript data reading module is used for reading data corresponding to the subscript under the condition that the whole block reading condition judging module judges that the traversed subscript does not meet the preset whole block reading condition;
a deleting module, configured to delete the traversed subscript from the subscript set, determine whether there is any non-traversed subscript in the subscript set, and if yes, instruct the subscript traversing module to traverse the subscript in the subscript set of the data to be read,
the whole block reading condition judgment module comprises:
the difference value judging module is used for judging whether the difference value of the currently traversed subscript and the smallest subscript in the traversed subscript exceeds a preset data length threshold value or not;
the first subscript number judging module is used for judging whether the number of the traversed subscripts is larger than a preset subscript number threshold or not under the condition that the difference value judging module judges that the difference value between the currently traversed subscript and the smallest subscript in the traversed subscripts exceeds a preset data length threshold, wherein the traversed subscripts do not comprise the currently traversed subscripts;
and the first subscript determining module is used for determining that the traversed subscript meets the preset whole block reading condition under the condition that the subscript number judging module judges that the number of the traversed subscripts is greater than the preset subscript number threshold.
8. The apparatus of claim 7, wherein the entire block of data reading module comprises:
and the first whole block data reading submodule is used for reading data corresponding to all subscripts from the traversed minimum subscript to the traversed maximum subscript.
9. The apparatus of claim 7, further comprising: a data length threshold determination module, configured to determine a preset data length threshold, where the data length threshold determination module includes:
the memory upper limit value determining module is used for determining the upper limit value of the occupied memory in the process of reading the data to be read;
a data length value determining module, configured to determine a data length value of the data to be read;
and the data length threshold determining submodule is used for determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
10. A data reading apparatus, characterized in that the apparatus comprises:
the subscript traversal module is used for traversing the subscripts in a subscript set of data to be read, and the subscripts in the subscript set are ordered from small to large;
the whole block reading condition judging module is used for judging whether the traversed subscript meets the preset whole block reading condition;
the whole block data reading module is used for reading the whole block data corresponding to the traversed subscript under the condition that the whole block reading condition judging module judges that the traversed subscript meets the preset whole block reading condition;
the subscript data reading module is used for reading data corresponding to the subscript under the condition that the whole block reading condition judging module judges that the traversed subscript does not meet the preset whole block reading condition;
the subscript traversal module comprises:
the subscript traversal submodule is used for traversing the subscripts in the subscript set of the data to be read, and the subscripts in the subscript set are ordered from small to large;
a grouping module for grouping the subscripts in the subscript set to obtain a subscript subset, wherein the difference value between the maximum subscript and the minimum subscript in the subscript subset does not exceed a preset data length threshold,
the whole block reading condition judgment module comprises:
the second subscript number judging module is used for judging whether the number of subscripts in the subscript subset is larger than a preset subscript number threshold value or not;
and the second subscript determining module is used for determining that the subscript in the subscript subset is the subscript meeting the preset whole block reading condition under the condition that the second subscript number judging module judges that the number of the subscripts in the subscript subset is greater than the preset subscript number threshold.
11. The apparatus of claim 10, wherein the entire block of data reading module comprises:
and the second whole data reading submodule is used for reading data corresponding to all subscripts from the minimum subscript to the maximum subscript in the subscript subset.
12. The apparatus of claim 10, further comprising: a data length threshold determination module, configured to determine a preset data length threshold, where the data length threshold determination module includes:
the memory upper limit value determining module is used for determining the upper limit value of the occupied memory in the process of reading the data to be read;
a data length value determining module, configured to determine a data length value of the data to be read;
and the data length threshold determining submodule is used for determining the preset data length threshold according to the upper limit value of the occupied memory in the process of reading the data to be read and the data length value of the data to be read.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610499749.4A CN107545011B (en) | 2016-06-29 | 2016-06-29 | Data reading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610499749.4A CN107545011B (en) | 2016-06-29 | 2016-06-29 | Data reading method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107545011A CN107545011A (en) | 2018-01-05 |
| CN107545011B true CN107545011B (en) | 2020-04-10 |
Family
ID=60965914
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610499749.4A Active CN107545011B (en) | 2016-06-29 | 2016-06-29 | Data reading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107545011B (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8311119B2 (en) * | 2004-12-31 | 2012-11-13 | Microsoft Corporation | Adaptive coefficient scan order |
| CN102043795B (en) * | 2009-10-13 | 2013-01-16 | 上海新华控制技术(集团)有限公司 | Establishing method for process control historical data file structure and data read-write method |
| CN101770515B (en) * | 2010-01-18 | 2012-01-11 | 杭州顺网科技股份有限公司 | Data block comparison based data updating method |
| GB2502620B (en) * | 2012-06-01 | 2020-04-22 | Advanced Risc Mach Ltd | A parallel parsing video decoder and method |
| CN103677655B (en) * | 2012-09-26 | 2016-12-21 | 北京信威通信技术股份有限公司 | A kind of two-dimemsional number group data stream reading/writing method on a memory and device |
| CN105224532B (en) * | 2014-05-28 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN104461920B (en) * | 2014-12-09 | 2019-04-12 | 杭州华为数字技术有限公司 | A kind of method and device of storing data |
| CN104994340A (en) * | 2015-06-25 | 2015-10-21 | 广东工业大学 | Precise positioning playback method for audio/video storage mode |
-
2016
- 2016-06-29 CN CN201610499749.4A patent/CN107545011B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107545011A (en) | 2018-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106681934B (en) | Method and device for recycling garbage of storage device | |
| CN109815162A (en) | EMS memory management process, device, mobile terminal and storage medium | |
| CN111324303B (en) | SSD garbage recycling method, SSD garbage recycling device, computer equipment and storage medium | |
| CN102682086B (en) | Data segmentation method and data segmentation equipment | |
| CN106570025B (en) | Data filtering method and device | |
| CN105468642A (en) | Data storage method and apparatus | |
| CN110659151A (en) | Data verification method and device and storage medium | |
| CN110399096B (en) | Method, device and equipment for deleting metadata cache of distributed file system again | |
| CN110297810B (en) | Stream data processing method and device and electronic equipment | |
| CN103473343A (en) | File management method, device and terminal | |
| KR20180108810A (en) | Service parameter selection method and associated device | |
| CN103747276A (en) | CDN data deletion method and CDN server | |
| CN107480771B (en) | Deep learning-based activation function realization method and device | |
| CN110019184B (en) | Method for compressing and decompressing ordered integer array | |
| CN107545011B (en) | Data reading method and device | |
| CN109542799B (en) | Block memory splicing method, splicing module, storage device and field programmable gate array | |
| CN110018797B (en) | Data migration method, device and equipment and readable storage medium | |
| CN109063261B (en) | Method and device for judging aging trend of power battery | |
| CN111666150A (en) | Storage space allocation method and device, terminal and computer readable storage medium | |
| CN110647498A (en) | File storage method, device, equipment and computer readable storage medium | |
| CN107783990B (en) | Data compression method and terminal | |
| CN107273303B (en) | Flash memory data management system and method, flash memory chip and storage device | |
| CN105095382A (en) | Method and device for sample distributed clustering calculation | |
| CN103746968A (en) | CDN server removal method, CDN control center and system thereof | |
| CN106469174B (en) | Method for reading data and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200506 Address after: 310052 room 508, floor 5, building 4, No. 699, Wangshang Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province Patentee after: Alibaba (China) Co.,Ltd. Address before: 100080 Beijing City, Haidian District Suzhou Street No. 3 floor 16 room 2 Patentee before: AUTONAVI INFORMATION TECHNOLOGY Co.,Ltd. |