CN115563107A - Continuous grid data storage and reading method - Google Patents
Continuous grid data storage and reading method Download PDFInfo
- Publication number
- CN115563107A CN115563107A CN202211159258.7A CN202211159258A CN115563107A CN 115563107 A CN115563107 A CN 115563107A CN 202211159258 A CN202211159258 A CN 202211159258A CN 115563107 A CN115563107 A CN 115563107A
- Authority
- CN
- China
- Prior art keywords
- pixel
- data
- screening
- value
- column
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Remote Sensing (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for storing and reading continuous grid data, which comprises the following steps: repartitioning the rows and columns of the original classification raster Data: the method comprises the steps of re-dividing rows and columns of original raster Data consisting of K rows and L columns of pixels by taking a screening area as the minimum granularity to obtain a row a and a column b screening blocks; screening the blocks one by adopting an error control method to carry out dynamic pixel screening on the screening blocks in the row a and the column b, and storing candidate pixel marking data and candidate pixel values into a data file Xdata; recording the storage initial position of the screening area block of each row and the first column in the data file Xdata; recording metadata information of the original continuous raster Data. The invention carries out interpolation screening of multi-level row and column pixels according to the appointed characteristic point pixel value, and has better data compression efficiency.
Description
Technical Field
The invention relates to the field of raster data processing, in particular to a method for storing and reading continuous raster data.
Background
The grid data model divides the geographic space into a pixel array consisting of a plurality of rows and columns, the minimum unit of the pixel array is called as a pixel or a pixel, the position of each pixel is determined by a row-column number, and a certain type of geographic attribute value of the position is recorded through the value of the pixel. The raster data with the integer pixel value is generally used for storing various discrete data or classified data, such as land utilization type raster data, wherein each pixel stores an integer representing the land utilization category of the geographic space corresponding to the pixel, for example, the integer 1 represents cultivated land, the integer 2 represents forest land, the integer 3 represents grassland, and the like; one of the characteristics of the discrete grid data is that arithmetic calculation cannot be directly performed on pixel values, and for example, it is meaningless to perform mean calculation on two pixels with pixel values of 1 and 2 to obtain a pixel value of 1.5; the continuous type raster data is characterized in that each pixel value has continuity, such as DEM raster data, and any value between the maximum pixel value and the minimum pixel value can be used as a certain pixel value.
Assuming that a certain continuous type grid data is composed of K rows and L columns of grid pixels, and a numerical value corresponding to each grid pixel is a double type floating point value, according to a common practice of a current grid data model, K × L × 8 bytes are needed, and when a coverage area of the grid data is large and a resolution is high (for example, each pixel corresponds to a 1-meter long and wide area), the number K × L of rows of the grid is large, and a required storage space is very large, which is a great challenge to a storage device cost and a network transmission efficiency of a system.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a method for storing and reading continuous grid data, which can effectively reduce the data storage capacity, save the cost of storage equipment and improve the network transmission efficiency.
A storage method of continuous raster Data stores original classified raster Data composed of K lines and L columns of pixels, and comprises the following steps:
s1, re-dividing rows and columns of original classification raster Data:
the method comprises the steps of re-dividing rows and columns of original raster Data consisting of K rows and L columns of pixels by taking a screening area as the minimum granularity to obtain a row a and a column b screening blocks;
when K/s 1 When it is an integer, a = K/s 1 ；
When K/s 1 When not an integer, a = [ K/s ] 1 ]+1；
When L/s is 2 When it is an integer, b = L/s 2 ；
When L/s is 2 When not an integer, b = [ L/s ] 2 ]+1；
Wherein the screening region is s 1 Line s 2 Grid data matrix of columns, s 1 、s 2 Is a constant, s 1 >＝3，s 2 >＝3；
S2, screening the blocks one by adopting an error control method for the screening blocks in the row a and the column b to carry out pixel dynamic screening, and storing candidate pixel marking data and candidate pixel values into a data file Xdata;
s3, recording the storage initial position of the screening block in the first column of each row in the data file Xdata;
and S4, recording metadata information of the original continuous raster Data.
Further, in S2, for the screening blocks in row a and column b, the dynamic screening of the pixels is performed by using error control to screen the blocks one by one, which specifically includes the following steps:
selecting a plurality of feature points from a row a and a column b screening blocks as candidate pixels;
determining a maximum absolute error threshold value e parameter value;
and performing interpolation calculation layer by layer according to the candidate pixels by adopting the maximum absolute error threshold value e parameter value, thereby performing dynamic pixel screening.
Further, the dynamic pixel screening specifically comprises the following steps:
selecting an intermediate pixel from every two candidate pixels, calculating a calculated value of an intermediate point pixel, judging whether the error between the calculated value of the intermediate pixel and the true value of the intermediate pixel is within a maximum error threshold value e, if so, not taking the intermediate pixel as a screening candidate pixel, otherwise, taking the intermediate pixel as a candidate pixel; and screening the pixels row by row and column by column until all the pixels are screened.
Further, in the step 2, candidate image element marking data and candidate image element values are stored in a data file Xdata; recording the size of bytes required by the storage of the current screening block by using a numerical value Ts, and initially setting the size to be 0;
marking each pixel screening condition of the screening block row by row and column by column, if the pixel is taken as a candidate pixel, setting marking data B [ i ] [ j ] =1, otherwise, setting marking data B [ i ] [ j ] =0;
if s 1 *s 2 Is a multiple of 8 Ts value of s 1 *s 2 8; if s 1 *s 2 Not a multiple of 8, i.e., the marking data B2][]Cannot be completely divided into an integer number of bytes, bit with value of 0 is supplemented to the complete one byte at the end, and the value of Ts is [ s ] 1 *s 2 /8]+1；
Writing candidate pixel mark data B [ ] [ ] into the data file Xdata by taking bytes as a unit, wherein Ts bytes are written into the data file Xdata; and writing the screened candidate pixel values into a data file Xdata in sequence.
Further, the step 3 of recording the storage start position of the screening area block of the first column of each row in the data file Xdata includes the following steps: the accumulated value sumTs of the storage space size Ts of each column filter block in each row is recorded and written into the data file rowSize.
Further, in step 4, the metadata information includes: the line number K and the column number L of the original continuous grid Data; screening area s 1 、s 2 Value, maximum absolute error threshold value, value of parameter.
Also provided is a method for reading continuous raster Data, which is used for reading raster Data stored according to the above method for storing continuous raster Data, and if raster unit Data D (r, c) of pixel value of the r row and c column of original classified raster Data is to be read, wherein 1< = r < = K,1< = c < = L, the method comprises the following steps:
step 1, reading a Data file meta, obtaining metadata information of original continuous raster Data, and obtaining the line number K and the column number L of the original continuous raster Data; screening area s 1 、s 2 Calculating a value, namely a maximum absolute error threshold value e parameter value, and calculating a screening block with a row and b columns in the data file Xdata;
step 2, calculating the screening block number of the pixel of the row r and the column c, and assuming that the row number of the screening block of the pixel of the request is v and the column number is z;
when r/s 1 When it is an integer, v = r/s 1 ；
When r/s 1 Is not thatWhen integer, v = [ r/s 1 ]+1；
When c/s 2 When it is an integer, z = c/s 2 ；
When c/s 2 When not an integer, z = [ c/s = 2 ]+1；
Step 3, reading the data file rowSize to obtain a integer values e1, e2,. And ea;
when v =1, the data of the first column screening block of the 1 st row starts at 0 byte of the data file Xdata, where offset =0;
when v >1, the data of the first column screening block of the v-th row starts at e (v-1) byte of the data file Xdata, when offset = e (v-1);
and reading the data file Xdata from the byte of the offset, and analyzing the information of the screening block one by one so as to calculate and obtain the pixel value of the screening block in which the requested pixel is located.
Further, for analyzing the information of the screening block one by one in step 3, the information of the screening block includes reading candidate pel marking data and reading candidate pel value data.
Further, analyzing the information of the filtering block to calculate the pixel value of the filtering block in which the requested pixel is located, specifically comprising the following steps:
if the current read is the jth screening block, 1< = j < = z;
reading candidate pixel mark data:
if s 1 *s 2 Is multiple of 8, reads s from the data file Xdata 1 *s 2 The/8 bytes are candidate pixel marking data; if s 1 *s 2 Reading s from data file Xdata if not multiple of 8 1 *s 2 /8]+1 byte, take the first s 1 *s 2 The bit data is candidate pixel marking data;
reading candidate pixel value data:
accumulation s 1 *s 2 Marking the number m of bits with the median value of 1 in data by using the bit candidate pixel, reading m numerical values v1, v2, v3, aIdentify s in the corresponding screening block 1 *s 2 M candidate pixel values;
if j = = z, performing interpolation calculation according to the m candidate pixel values to obtain all pixel values in the screening block;
if j! And = z, the requested pixel is not in the screening block, and the pixel value in the screening block does not need to be analyzed, and the processing of the screening block data of the next column is continued.
Further, the interpolation calculation to obtain all pixel values in the filtering block includes the following steps:
before the interpolation of the intermediate pixel of every two candidate pixels, judging whether the intermediate pixel value belongs to the candidate pixel, if so, directly determining the intermediate pixel value as the candidate pixel value without performing interpolation calculation, and if not, taking the interpolation calculation result as the intermediate pixel value; carrying out interpolation processing for multiple times line by line to obtain all pixel values in the screening block;
and then according to the row and column numbers r and c of the request pixel, obtaining the pixel position in the screening block, further determining the pixel value, and ending the reading.
Compared with the prior art, the invention has the following technical effects:
the invention carries out interpolation screening of multi-level row and column pixels according to the specified characteristic point pixel value, has data compression effect as long as the pixel values in a certain row and a certain column are partially close, does not depend on the initially specified characteristic point pixel value, and has better data compression efficiency.
Drawings
Fig. 1 is a schematic diagram of the overall structure of the system of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth to illustrate, but are not to be construed to limit the scope of the invention.
Referring to fig. 1, a method for storing continuous raster Data stores original classification raster Data composed of K rows and L columns of pixels, and for convenience of description, the row and column numbers of the pixels and blocks of the original classification raster Data are counted from 1. In the present application, the symbol [ x ] denotes the largest integer not greater than x.
The storage method of the continuous grid data comprises the following steps:
s1, repartitioning the rows and columns of original classification raster Data:
the method comprises the steps of re-dividing rows and columns of original raster Data consisting of K rows and L columns of pixels by taking a screening area as the minimum granularity to obtain a row a and a column b screening blocks;
when K/s 1 When it is an integer, a = K/s 1 ；
When K/s 1 When not an integer, a = [ K/s ] 1 ]+1；
When L/s is 2 When it is an integer, b = L/s 2 ；
When L/s is 2 When not an integer, b = [ L/s = 2 ]+1；
Wherein the screening region is s 1 Line s 2 Grid data matrix of columns, s 1 、s 2 Is a constant, s 1 >＝3，s 2 >＝3。
In a particular embodiment, the number of rows s of picture elements of the screening area 1 Sum column number s 2 May be unequal or equal. If s 1 ＝s 2 And = s, the screening area consists of s rows and s columns of pixels, and the original classification raster Data is blocked from top to bottom and from left to right by taking the screening area as the minimum granularity to obtain a row and b columns of screening blocks.
When K/s is an integer, a = K/s;
a = [ K/s ] +1 when K/s is not an integer; at this time, if there are less than s rows of pixels in the screening block of the a-th row, the pixels are supplemented to s rows of pixels with the pixel value of 0.
B = L/s when L/s is an integer;
when L/s is not an integer, b = [ L/s ] +1; at this time, if there are less than s rows of pixels in the b-th row screening block, the pixel value is 0 and the pixel value is supplemented to s rows of pixels.
S2, for the screening blocks in the row a and the column b, adopting a maximum absolute error threshold value e parameter value to screen the blocks one by one to carry out pixel dynamic screening, and storing candidate pixel marking data and candidate pixel values into a data file Xdata.
And (3) a dynamic screening process of screening blocks in the j row and q column, wherein 1< = j < = a,1< = q < = b: recording the size of the byte required by the storage of the current screening block by using the numerical value Ts, wherein the initial size is 0, and the method specifically comprises the following steps:
s2-1, selecting a plurality of feature points from a row a and a column b screening blocks as candidate pixels;
in particular embodiments, the screened regions have s in common 1 *s 2 The pixel comprises a 1 st row, a 1 st column, a 1 st row, and an s 2 Column pixel B, s 1 Line 1, column C, s 1 Line(s) 2 And taking four pixels of the column pixel D as feature points to directly select candidate pixel points. Of course, other pixels can be selected as feature points, and epitaxial interpolation calculation is needed when calculating certain pixel values.
S2-2, determining a maximum absolute error threshold value e parameter value;
the parameter value e may be set empirically for the maximum absolute error threshold.
And S2-3, performing layer-by-layer interpolation calculation by adopting a maximum absolute error threshold value e parameter value according to the candidate pixels, and performing dynamic pixel screening.
Selecting an intermediate pixel from every two candidate pixels, calculating a calculated value of an intermediate point pixel, judging whether the error between the calculated value of the intermediate pixel and the true value of the intermediate pixel is within a maximum error threshold value e, if so, not taking the intermediate pixel as a screening candidate pixel, otherwise, taking the intermediate pixel as a candidate pixel; and screening row by row and column by column until all pixels are screened.
In a specific embodiment, for the filtering of row 1 pels in the filtering block:
the pixels in the row 1 are interpolated according to the candidate pixels A and B to calculate a calculation value VM of an intermediate point pixel M, whether the error between the comparison value VM and the true value PM of the pixel M is within a maximum absolute error threshold value e is judged, and if yes, the intermediate point pixel M is not used as a screening candidate pixel; otherwise, the pixel M is used as a candidate pixel; the interpolation method is not particularly limited, and may be consistent with the interpolation calculation in the subsequent reading analysis, and it is preferable to use efficient linear interpolation.
For the pixels between the pixel A and the pixel M, calculating a calculation value VS of the pixel S at the intermediate point by the same interpolation, comparing whether the error between the value VS and the true value PS of the pixel S is within a maximum absolute error threshold e, and if so, not taking the pixel S at the intermediate point as a candidate pixel; otherwise, the pixel S is used as a candidate pixel;
for the pixels between the pixel M and the pixel B, the calculated value VT of the pixel T at the intermediate point is calculated by interpolation in the same way, whether the error between the comparison value VT and the true value PT of the pixel T is within the maximum absolute error threshold value e or not is judged, and if yes, the pixel T at the intermediate point is not taken as a candidate pixel; otherwise, the pixel T is used as a candidate pixel;
and continuing the same process to process the pixels between the pixel A and the pixel S, the pixels between the pixel S and the pixel M, the pixels between the pixel M and the pixel T and the pixels between the pixel T and the pixel B until all the pixels in the row are screened.
And (3) screening the 1 st column of pixels in the screening block: pixels between the pixel A and the pixel C in the 1 st row are processed in a similar way to the pixel screening process in the 1 st row, a calculated value VM of an intermediate point pixel M is calculated according to interpolation of the pixel A and the pixel C, whether the error between the compared value VM and the true value PM of the pixel M is within a maximum absolute error threshold value e or not is judged, and if yes, the intermediate point pixel M is not taken as a candidate pixel; otherwise, the pixel M is used as a candidate pixel; and (4) screening the pixels between the pixel A and the pixel M and between the pixel M and the pixel C according to the similar method until all the pixels in the row are screened.
Screening the pixels in the s-th row in the screening block: and the processing process of the pixels between the pixel B and the pixel D in the s-th column is similar to the screening process of the pixel in the 1 st column until all the pixels in the column are screened.
And (3) screening pixels from the 2 nd row to the s th row in the screening block: at this time, the pixel values of the 1 st column and the s th column from the 2 nd row to the s th row are determined, and pixel screening is performed row by adopting the method in the step A.
For storing candidate pel marking data and candidate pel values into a data file Xdata, the method comprises the following steps:
distribution s 1 *s 2 Candidate pixel marking data B [2] of bit storage space][]；B[][]Is a two-dimensional array having s 1 Line s 2 And the rows respectively correspond to whether each pixel in the screening block is a candidate pixel or not.
Marking each pixel screening condition of the screening blocks from top to bottom and from left to right in sequence, and setting marking data B [ i ] if the pixel is taken as a candidate pixel][j]=1, otherwise set flag data B [ i][j]=0; if s is 1 *s 2 A value of Ts which is a multiple of 8 is s/8; if s is 1 *s 2 Is not a multiple of 8, i.e., the label data B [ ]][]Cannot be completely divided into an integer number of bytes, then the bit with 0 is supplemented to the complete one byte at the end, and the value of Ts is [ s S/8%]+1；
Writing candidate pixel mark data B [ ] [ ] into the data file Xdata by taking bytes as a unit, wherein Ts bytes are written into the data file Xdata; writing the screened candidate pixel values into a data file Xdata from left to right from top to bottom in sequence; suppose there are m candidate pixels, each pixel value is of double type, and the Ts value is increased by 8 × m bytes.
In an embodiment, a certain filtering block s =7 and the maximum absolute error threshold e =1.0 of the continuous grid data, and the pixel values of the filtering block are as shown in table 1 below:
TABLE 1
| 562.3 | 563.1 | 568.7 | 564.6 | 564.8 | 565.7 | 566.3 |
| 662.3 | 663.1 | 663.7 | 664.6 | 674.8 | 655.7 | 666.3 |
When the screening block is dynamically screened, the pixel value of the ith row and the jth column in the screening block is represented by p [ i ] [ j ], and the pixel value of the ith row and the jth column in the screening block is represented by Vi [ i ] [ j ];
in the 1 st row of pixels, interpolation is carried out according to the pixel p [1] [1] and the pixel p [1] [7] to calculate a calculated value VM = (V [1] [1] + V [1] [7 ])/2.0 =564.3 of the intermediate point pixel p [1] [4], whether an error delt =0.3 of a true value PM =564.6 of the comparison value VM and the pixel p [1] [4] is within a maximum absolute error threshold e or not is judged, and the intermediate point pixel p [1] [4] is not used as a screening candidate pixel due to the delt < e; then, interpolating and calculating a calculated value VM = V [1] [1] + (V [1] [4] -V [1] [1 ])/3 =563.06 of the middle point pixel p [1] [2] according to the pixel p [1] [1] and the pixel p [1] [4], wherein the pixel p [1] [2] is not the middle position of the pixel p [1] [1] and the pixel p [1] [4], an error delt =0.04 of a real value PM =563.1 of a comparison value VM and the pixel p [1] [2], and the middle point pixel p [1] [2] is not used as a screening candidate pixel due to the delt < e; then, interpolating and calculating a calculated value VM = (V [1] [2] + V [1] [4 ])/2.0 =563.85 of the intermediate point pixel p [1] [3] according to the pixel p [1] [2] and the pixel p [1] [4], comparing the error delt =4.85 of the actual value PM =568.7 of the value VM and the pixel p [1] [3], and taking the intermediate point pixel p [1] [3] as a screening candidate pixel due to the fact that the delt > e; the interpolation screening is carried out on the rest pixels in the same way, and it can be known that 3 pixels in the first row are taken as candidate pixels, and are respectively p 1, p 1 and p 1 7; similarly, all the pixels in the 1 st and 7 th columns are screened, and then all the pixels in the 2 nd to 7 th columns are screened to obtain 4 pixels in the 7 th column as candidate pixels, which are p 7 1, p 75, p 7 6 and p 7; the candidate pixel marking data B [ ] [ ] of 49 bit storage spaces are allocated, and the value of each flag bit is shown in the following table 2:
TABLE 2
| 1 | 0 | 1 | 0 | 0 | 0 | 1 |
| 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| 1 | 0 | 0 | 0 | 1 | 1 | 1 |
When candidate pixel mark data B [ ] [ ] is written into the data file Xdata in byte unit, at this time, 49 bits are not multiples of 8, and then 7 bits with value of 0 are supplemented to a complete byte at the end, which totally requires 7 bytes of storage space, and the following mark data are obtained, as shown in table 3:
TABLE 3
| 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | ... | 1 | 1 | 1 | 0000000 |
Writing 7 bytes of storage space in the table into the data file Xdata one by one according to bytes from left to right; then writing the screened candidate pixel values into a data file Xdata from left to right from top to bottom in sequence, namely, pixels p [1] [1], p [1] [3] and p [1] [7]; for each block, the values of p 7, 6 and p 7 are written into the data file Xdata in sequence; assuming that there are m =12 candidate pixels, ts =7+12 + 8=103 bytes; at this point, the dynamic screening process of the screening block is finished.
And S3, recording the storage initial position of the screening block in the first column of each row in the data file Xdata, so as to improve the file reading and analyzing efficiency.
It can be seen from the above steps that the storage space values used when the data file Xdata is dynamically screened and written into each screening block may not be the same, and a problem arises after the data of all the screening blocks is stored into the data file Xdata according to the above sequence of steps: for example, the data of the screening block in row 2 and column 1 starts at which position in the data file Xdata or the data of the screening block in row 2 and column 1 starts from the several bytes in the data file Xdata. If the reading and analyzing efficiency of the blocks which are screened one by one from the first row and the first column is low; if the starting point position of the data in the first column of each row of screening blocks is known, the information of any screening block can be quickly read, and only the data of a certain column of screening blocks needs to be analyzed one by one from the first column of screening blocks in the row.
Therefore, the accumulated value sumTs of the storage space size Ts of each column filter block in each row needs to be recorded and written into the data file rowSize. For a row a and a column b screening blocks in the S1, processing the screening blocks according to the S2 row by row to obtain a integer values e1, e2, and ea, wherein e1 represents that the storage space of the screening blocks of all columns in the first row is e1 bytes, and ea represents that the storage space of the screening blocks of all columns in the last row is ea bytes; these a integer values e1, e2, ·, ea are written into the data file rowSize.
And S4, recording the metadata information of the original continuous raster Data.
The metadata information includes: the line number K and the column number L of the original continuous grid Data; screening area s 1 、s 2 Values, maximum absolute error threshold e parameter values, etc.; the metadata information is written into the data file meta.
In the method, multi-level row and column pixel interpolation screening is carried out according to the appointed four characteristic point pixel values, a data compression effect is achieved as long as pixel values in a certain row and a certain column are partially close to each other, the method does not depend on the originally appointed four characteristic point pixel values, and the rule that local numerical similarity exists among most pixel values except the four characteristic point pixels is fully utilized, so that the method has better data compression efficiency.
Based on the storage method of the continuous grid data, the invention also provides a reading method of the continuous grid data. The specific reading process comprises the following steps:
supposing that the pixel value of the r row and c column of original continuous raster Data is to be read; 1< = r < = K,1< = c < = L;
step 1, reading a Data file meta, obtaining metadata information of original continuous raster Data, and obtaining the line number K and the column number L of the original continuous raster Data; and (4) calculating a screening area S value, a maximum absolute error threshold value e parameter value and the like according to the S1, wherein a row and b columns of screening blocks are shared in the data file Xdata.
And 2, calculating the screening block number of the pixel of the row r and the column c, wherein the row number of the screening block in which the requested pixel is located is assumed to be v, and the column number of the screening block in which the requested pixel is located is assumed to be z. According to the partitioning method in S1, it is possible to obtain:
when r/s 1 When it is an integer, v = r/s 1 ；
When r/s 1 When not an integer, v = [ r/s ] 1 ]+1；
When c/s 2 When it is an integer, z = c/s 2 ；
When c/s 2 When not an integer, z = [ c/s ] 2 ]+1。
In a particular embodiment, s 1 ＝s 2 = s; that is to say that the first and second electrodes,
when r/s is an integer, v = r/s;
when r/s is not an integer, v = [ r/s ] +1;
when c/s is an integer, z = c/s;
z = [ c/s ] +1 when c/s is not an integer;
step 3, reading the data file rowSize to obtain a integer values e1, e2,. And ea;
when v =1, the data of the first column screening block of the 1 st row starts at 0 byte of the data file Xdata, where offset =0;
when v >1, the data of the first column screening block of the v-th row starts at e (v-1) byte of the data file Xdata, when offset = e (v-1);
the data file Xdata is read from the offset byte, that is, the information of the screening blocks in the v-th row and the 1 st column is analyzed one by one.
Specifically, if the current read is the jth screening block, 1< = j < = z;
and 3-1, reading the candidate pixel mark data.
If s 1 *s 2 Is multiple of 8, reads s from the data file Xdata 1 *s 2 The/8 bytes are candidate pixel marking data; if s 1 *s 2 Reading s from data file Xdata if not multiple of 8 1 *s 2 /8]+1 byte, take the first s 1 *s 2 And the bit data is candidate pixel marking data.
In an embodiment, if s 1 ＝s 2 S, if s is a multiple of 8, reading s/8 bytes from the data file Xdata as candidate pixel marking data;reading [ s/8 ] from the data file Xdata if s is not a multiple of 8]+1 bytes; and taking the first s bits of data as candidate pixel marking data.
And 3-2, reading candidate pixel value data.
Sum s 1 *s 2 Marking the number m of bits with the median value of 1 in data by using the bit candidate pixel, reading the values v1, v2, v3, the 1 *s 2 M candidate pel values of pixels.
In an embodiment, if s 1 ＝s 2 And (5) accumulating the bit number m of which the median value in the s bit candidate pixel marking data is 1, reading the values v1, v2, v3,. And vm of m double types from the data file Xdata, and respectively corresponding to the values of s candidate pixels in the screening block.
And 3-3, calculating to obtain the pixel value of the screening block where the request pixel is located.
If j = = z, interpolation calculation needs to be performed according to the m candidate pixel values to obtain all pixel values in the screening block, and the specific steps are as follows:
before the interpolation of the intermediate pixel of every two candidate pixels, judging whether the intermediate pixel value belongs to the candidate pixel, if so, directly determining the intermediate pixel value as the candidate pixel value without performing interpolation calculation, and if not, taking the interpolation calculation result as the intermediate pixel value; performing interpolation processing for multiple times line by line to obtain all pixel values in the screening block; and then according to the row and column numbers r and c of the requested pixel element, obtaining the pixel position in the screening block, further determining the pixel value, and finishing reading.
In a specific embodiment, before interpolating an intermediate pixel C of a pixel A and a pixel B, judging whether the pixel C value belongs to a candidate pixel, if so, directly determining the pixel C value as the candidate pixel value without performing interpolation calculation, and if not, taking the interpolation calculation result as the pixel C value; and performing interpolation processing for multiple times line by line to obtain all pixel values in the screening block. And then according to the column number r of the request pixel, obtaining the pixel position in the screening block, further determining the pixel value, and finishing reading.
In a specific embodiment, a 7-byte storage space is read from the xx byte in the data file Xdata to obtain candidate tag data of a certain filter block as follows, as shown in table 4:
TABLE 4
| 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | ... | 1 | 1 | 1 | 0000000 |
Taking the first 49 bit data as candidate pixel marking data, accumulating the number of bits with the median value of 1 in the candidate pixel marking data to be 12, reading the values v1, v2, v3, a.
TABLE 5
| v1 | 0 | v2 | 0 | 0 | 0 | V3 |
| 0 | 0 | 0 | V4 | 0 | 0 | 0 |
| V9 | 0 | 0 | 0 | V10 | V11 | V12 |
Performing interpolation calculation according to the 12 candidate pixel values to obtain all pixel values in the screening block; before interpolating the intermediate pixel p 1< 3 > of the pixel p 1< 1 > and the pixel p 1< 7 >, whether the pixel p 1< 3 > belongs to the candidate pixel is judged, and the pixel p 1< 3 > is determined by interpolation calculation because the pixel p 1< 3 > is not the candidate pixel; in the same way, the value of the pixel p 1 is calculated and determined by interpolating the pixel p 1 and the pixel p 1 3; the pixel p [1] [3] is a candidate pixel, so that interpolation calculation is not needed, and the value is directly determined as a v2 value; and so on may determine all pixel values.
If j! And = z, the requested pixel is not in the screening block, and the pixel value in the screening block does not need to be analyzed, and the processing of the screening block data of the next column is continued.
Detailed description of the preferred embodiment
Taking national DEM (Digital Elevation Model) data as an example, under the condition of 30 m resolution, the data is composed of K =134724 rows and L =161360 grid pixel data columns in a traditional storage mode, 8-byte double-type numerical value storage is adopted, and the total required storage space is about 161.9GB.
By applying the method, a screening area of 15x15 pixels is divided to obtain M =8982, N =10758 divided screening blocks, then the M × N screening blocks are subjected to the dynamic screening method, and candidate pixel marking data and candidate pixel values are stored in the data file Xdata under the condition that the maximum absolute error threshold value e =3.0 meters. Through statistics, the storage size of the obtained data file Xdata is about 8.09GB and about 5% of the storage size of the original data; the storage space of the data file rowSize is 8982 × 8 bytes and is approximately 70KB, and the storage space of the metadata information of the original DEM data can be ignored, so that the method can greatly save the storage space, and the data compression effect is obviously better than that of the existing other methods.
The method for storing and reading the continuous grid data has the following advantages:
the method fully utilizes the rule that most pixel values except for four feature point pixels in the screening block have local numerical similarity, does not require that most pixel values in the screening block are the same as or similar to the grid pixel values of a certain fixed position, and is suitable for continuous grid data with spatial similarity, such as DEM (digital elevation model) and gradient data, so that the method has wider applicability and better data compression efficiency.
The dynamic screening process and the data file reading and analyzing process in the screening block only involve simple interpolation calculation, the processing is simple, the operation efficiency is high, the storage space can be saved for the storage, backup and network transmission of massive raster data, the system cost is reduced, and the method has obvious economic value. The method is particularly suitable for the display system of the three-dimensional terrain data in the large-scene real-time simulation (under the conditions of large DEM data volume and low precision requirement), and can have good data compression efficiency on the DEM data, thereby improving the response performance of the system.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A storage method of continuous raster Data stores original classified raster Data composed of K lines and L columns of pixels, and is characterized by comprising the following steps:
s1, re-dividing rows and columns of original classification raster Data:
the method comprises the steps of re-dividing rows and columns of original raster Data consisting of K rows and L columns of pixels by taking a screening area as the minimum granularity to obtain a row a and a column b screening blocks;
when K/s 1 When it is an integer, a = K/s 1 ；
When K/s 1 When not an integer, a = [ K/s ] 1 ]+1；
When L/s is 2 When it is an integer, b = L/s 2 ；
When L/s is 2 When not an integer, b = [ L/s = 2 ]+1；
Wherein the screening region is s 1 Line s 2 Grid data matrix of columns, s 1 、s 2 Is a constant number s 1 >＝3，s 2 >＝3；
S2, screening the blocks one by adopting an error control method for the screening blocks in the row a and the column b to carry out pixel dynamic screening, and storing candidate pixel marking data and candidate pixel values into a data file Xdata;
s3, recording the storage initial position of the screening block in the first column of each row in the data file Xdata;
and S4, recording the metadata information of the original continuous raster Data.
2. The method as claimed in claim 1, wherein in S2, the dynamic pixel selection is performed on the a row and b column selection blocks one by using error control, and the method comprises the following steps:
selecting a plurality of feature points from a row a and a column b screening blocks as candidate pixels;
determining a maximum absolute error threshold value e parameter value;
and performing interpolation calculation layer by adopting the parameter value e of the maximum absolute error threshold value according to the candidate pixels, thereby performing dynamic pixel screening.
3. The method for storing continuous grid data according to claim 2, wherein the dynamic pixel selection comprises the following steps:
selecting an intermediate pixel from every two candidate pixels, calculating a calculated value of an intermediate point pixel, judging whether the error between the calculated value of the intermediate pixel and the true value of the intermediate pixel is within a maximum error threshold value e, if so, not taking the intermediate pixel as a screening candidate pixel, otherwise, taking the intermediate pixel as a candidate pixel; and screening row by row and column by column until all pixels are screened.
4. The method for storing continuous grid data according to claim 3, wherein in step 2, the candidate pel flag data and the candidate pel value are stored in a data file Xdata; recording the size of bytes required by the storage of the current screening block by using a numerical value Ts, and initially setting the size to be 0;
marking each pixel screening condition of the screening block row by row and column by column, if the pixel is taken as a candidate pixel, setting marking data B [ i ] [ j ] =1, otherwise, setting marking data B [ i ] [ j ] =0;
if s 1 *s 2 Is a multiple of 8 Ts with a value of s 1 *s 2 8; if s 1 *s 2 Not a multiple of 8, i.e., the marking data B2][]Cannot be completely divided into an integer number of bytes, bit with value of 0 is supplemented to the complete one byte at the end, and the value of Ts is [ s ] 1 *s 2 /8]+1；
Writing candidate pixel mark data B [ ] [ ] into a data file Xdata by taking bytes as a unit, wherein the Ts bytes are written into the data file; and writing the screened candidate pixel values into a data file Xdata in sequence.
5. The method for storing continuous grid data according to claim 4, wherein the step 3 of recording the storage start position of the screening area block of the first column in each row in the data file Xdata comprises the following steps: the accumulated value sumTs of the storage space size Ts of each column filter block in each row is recorded and written into the data file rowSize.
6. The method according to claim 4, wherein in step 4, the metadata information includes: the line number K and the column number L of the original continuous grid Data; screening area s 1 、s 2 Value, maximum absolute error threshold value, value of parameter.
7. A method for reading continuous raster Data stored by the method for storing continuous raster Data according to any one of claims 5 to 6, wherein if raster unit Data D (r, c) of pixel values of the r row and c column of original classified raster Data is to be read, wherein 1< = r < = K,1< = c < = L, the method comprises the following steps:
step 1, reading a Data file meta, obtaining metadata information of original continuous raster Data, and obtaining the line number K and the column number L of the original continuous raster Data; screening area s 1 、s 2 Calculating a value, namely a maximum absolute error threshold value e parameter value, and calculating a screening block with a row and b columns in the data file Xdata;
step 2, calculating the screening block number of the pixel of the row r and the column c, and assuming that the row number of the screening block of the pixel of the request is v and the column number is z;
when r/s 1 When it is an integer, v = r/s 1 ；
When r/s 1 When not an integer, v = [ r/s = 1 ]+1；
When c/s 2 When it is an integer, z = c/s 2 ；
When c/s 2 When not an integer, z = [ c/s = 2 ]+1；
Step 3, reading a data file rowSize to obtain a integer values e1, e2, e.
When v =1, the data of the first column screening block of the 1 st row starts at 0 byte of the data file Xdata, where offset =0;
when v >1, the data of the first column screening block of the v-th row starts at e (v-1) byte of the data file Xdata, when offset = e (v-1);
and reading the data file Xdata from the byte of the offset, and analyzing the information of the screening block one by one so as to calculate and obtain the pixel value of the screening block in which the requested pixel is located.
8. The method as claimed in claim 7, wherein the information of the selected blocks is analyzed one by one in step 3, and the information of the selected blocks includes the read candidate pel flag data and the read candidate pel value data.
9. The method according to claim 8, wherein the step of analyzing the information of the filtering block to calculate the pixel value of the filtering block in which the requested pixel is located comprises the following steps:
if the current read is the jth screening block, 1< = j < = z;
reading candidate pel mark data:
if s 1 *s 2 If the value is multiple of 8, s is read from the data file Xdata 1 *s 2 The/8 bytes are candidate pixel marking data; if s 1 *s 2 Reading s from data file Xdata if not multiple of 8 1 *s 2 /8]+1 bytes, take the first s 1 *s 2 The bit data is candidate pixel marking data;
reading candidate pixel value data:
accumulation s 1 *s 2 Marking the number m of bits with the median value of 1 in data by using the bit candidate pixel, reading the values v1, v2, v3, the 1 *s 2 M candidate pixel values;
if j = = z, performing interpolation calculation according to the m candidate pixel values to obtain all pixel values in the screening block;
if j! And = z, the requested pixel is not in the screening block, and the pixel value in the screening block does not need to be analyzed, and the processing of the screening block data of the next column is continued.
10. The method for reading continuous grid data according to claim 9, wherein the step of obtaining all pixel values in the filtered block by interpolation comprises the steps of:
before the interpolation of the intermediate pixel of every two candidate pixels, judging whether the intermediate pixel value belongs to the candidate pixel, if so, directly determining the intermediate pixel value as the candidate pixel value without performing interpolation calculation, and if not, taking the interpolation calculation result as the intermediate pixel value; performing interpolation processing for multiple times line by line to obtain all pixel values in the screening block;
and then according to the row and column numbers r and c of the requested pixel element, obtaining the pixel position in the screening block, further determining the pixel value, and finishing reading.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211159258.7A CN115563107A (en) | 2022-09-22 | 2022-09-22 | Continuous grid data storage and reading method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211159258.7A CN115563107A (en) | 2022-09-22 | 2022-09-22 | Continuous grid data storage and reading method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115563107A true CN115563107A (en) | 2023-01-03 |
Family
ID=84741015
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211159258.7A Pending CN115563107A (en) | 2022-09-22 | 2022-09-22 | Continuous grid data storage and reading method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115563107A (en) |
-
2022
- 2022-09-22 CN CN202211159258.7A patent/CN115563107A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101154291B (en) | Image data compression method, image display method and corresponding device | |
| CN111683269B (en) | Video processing method, video processing device, computer equipment and storage medium | |
| CN102025959B (en) | The System and method for of high definition video is produced from low definition video | |
| WO1993017519A1 (en) | Fractal coding of data | |
| CN107480203A (en) | It is a kind of to be directed to identical and similar pictures duplicate removal view data cleaning method | |
| CN101300576A (en) | Image comparison | |
| US20160239597A1 (en) | Apparatus and method for performing finite element computation | |
| US11055816B2 (en) | Image processing device, image processing method, and image processing program | |
| CN104504121A (en) | Video retrieval method and device | |
| KR20150114950A (en) | Increasing frame rate of an image stream | |
| US20080212883A1 (en) | Processing Method of Data Structure for Real-Time Image Processing | |
| Morovic et al. | A fast, non-iterative and exact histogram matching algorithm | |
| CN116431857B (en) | Video processing method and system for unmanned scene | |
| CN115563107A (en) | Continuous grid data storage and reading method | |
| CN111652976A (en) | Through-viewing analysis method based on DEM raster data | |
| CN101087436A (en) | Temporal noise analysis of a video signal | |
| CN111047654A (en) | High-definition high-speed video background modeling method based on color information | |
| Peng et al. | Volatility-Based Measurements Allocation for Distributed Data Storage in Mobile Crowd Sensing | |
| JPH0528266A (en) | Method and device for analyzing texture | |
| CN117710235B (en) | Image target enhancement method, device, computer equipment and storage medium | |
| CN110991361A (en) | Multi-channel multi-modal background modeling method for high-definition high-speed video | |
| CN106331720B (en) | Video decoding related information storage method and device | |
| CN116527908B (en) | Motion field estimation method, motion field estimation device, computer device and storage medium | |
| CN115049873A (en) | Storage and reading method of classified raster data | |
| CN116883261A (en) | Image denoising method, image denoising device, computer equipment, storage medium and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |