CN112699102A - Processing method of lattice point data and related equipment thereof - Google Patents

Processing method of lattice point data and related equipment thereof Download PDF

Info

Publication number
CN112699102A
CN112699102A CN202110305003.6A CN202110305003A CN112699102A CN 112699102 A CN112699102 A CN 112699102A CN 202110305003 A CN202110305003 A CN 202110305003A CN 112699102 A CN112699102 A CN 112699102A
Authority
CN
China
Prior art keywords
data
lattice
point data
lattice point
grid
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110305003.6A
Other languages
Chinese (zh)
Other versions
CN112699102B (en
Inventor
张宝才
高瑞翔
于强
彭乘风
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jianju Technology Co ltd
Original Assignee
Beijing Jianju Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jianju Technology Co ltd filed Critical Beijing Jianju Technology Co ltd
Priority to CN202110305003.6A priority Critical patent/CN112699102B/en
Publication of CN112699102A publication Critical patent/CN112699102A/en
Application granted granted Critical
Publication of CN112699102B publication Critical patent/CN112699102B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present disclosure provides a method for processing lattice data, including: creating a database table in a database according to a preset lattice point data structure; wherein the database table comprises: a grid data field and at least one attribute field; reading a plurality of groups of lattice point data to be processed and at least one item of attribute information corresponding to each group of lattice point data from a data set; packaging the lattice point data into a lattice point data object according to a preset lattice point data object structure aiming at each group of lattice point data; and aiming at each group of lattice data, writing the lattice data object into a lattice data field of one data record in the database table, and respectively writing the at least one item of attribute information into the attribute field corresponding to the at least one item of attribute information in the data record. Corresponding to the processing method of the lattice point data, the disclosure also provides a processing device of the lattice point data, an electronic device and a computer readable storage medium.

Description

Processing method of lattice point data and related equipment thereof
Technical Field
One or more embodiments of the present disclosure relate to a data processing technology, and in particular, to a method for processing grid point data, an apparatus for processing grid point data, an electronic device, and a computer-readable storage medium.
Background
With the increase of meteorological observation data types and observation frequency, the meteorological data amount is larger and larger. The meteorological data is typical big data and has the characteristics of large data volume, high timeliness, rich data types and the like. At present, many meteorological data are stored in the form of grid point data, which is also called meteorological grid point data, such as precipitation data or temperature data of a certain region. Weather grid point data is generally obtained by dividing a space into a plurality of grids according to longitude and latitude, each grid is called a grid point, the position of an entity represented by each grid point is defined by the row number and the column number of the entity, and each grid point is assigned with a corresponding value to represent the numerical value of the entity represented by the grid point. Briefly, the grid data may be viewed simply as a two-dimensional matrix, with each grid corresponding to a position and a value in the matrix, and this portion of the data may be referred to as the data portion of the grid data. The above-mentioned lattice data includes, in addition to the above-mentioned data part, description information describing the above-mentioned data part. For example, the description information describes the information such as the number of rows and columns of the data portion, the resolution of the rows and columns, and the longitude and latitude corresponding to the starting row and column. In addition, each grid point in the grid point data may also correspond to one or more source data describing the attributes of the grid point data. The source data may include attribute information of one or more items of lattice data. For example, these attribute information may include: time information, altitude information or sampling values corresponding to the lattice point data are information of other dimensions except the geographic space. In general, the meteorological grid point data may include a plurality of grid point data files, each meteorological grid point data file storing a type of meteorological data, and each grid point in each meteorological grid point data corresponds to one or more attribute information. For example, the precipitation in a certain area on a certain day may be stored in a lattice point data file, and the row number of each lattice point in the lattice point data file corresponds to a latitude value, and the column number of each lattice point corresponds to a longitude value, so that the entity position represented by each lattice point may be determined by the longitude and latitude corresponding to the row and column of the lattice point. Moreover, the value of each grid point is the precipitation at that location on the day. It can be seen that the precipitation grid point data records the precipitation of each geographical position determined by different longitudes and latitudes in a certain area of a certain day. For another example, the average day temperature in a certain area on a certain day may be stored in another grid point data file, the row number of each grid point corresponds to a latitude value, and the column number of each grid point corresponds to a longitude value, so that the entity position represented by each grid point may be determined by the longitude and latitude corresponding to the row and column of the grid point. Also, the value of each grid point is the average daily temperature at that location on that day. It can be seen that the daily temperature grid data records the average daily temperature of each geographical location determined by different latitudes and longitudes in a certain area of the day. Of course, it should be noted that each grid point in one grid point data file may correspond to different time information.
At present, the storage and real-time query of meteorological grid point data are usually realized by adopting a relational database and file system. However, the method of adding a file system to a relational database is often difficult to bear the increase of the number of files with large data volume, and the tree structure of the file directory does not well satisfy the characteristic of sequentially accessing data, and when the meteorological lattice point data is queried or analyzed, the file system needs to download all the meteorological lattice point data from the relational database first and then analyze and process the meteorological lattice point data, so that the required processing time is very long. Therefore, this model does not meet the processing requirements of meteorological grid point data well.
Disclosure of Invention
In view of this, one or more embodiments of the present disclosure provide a method for processing lattice data. Based on the lattice point data method disclosed by the invention, the rapid real-time query and real-time operation of lattice point data can be realized.
The processing method of lattice point data according to one or more embodiments of the present disclosure may include: creating a database table in a database according to a preset lattice point data structure; wherein the database table comprises: a grid data field and at least one attribute field; reading a plurality of groups of lattice point data to be processed and at least one item of attribute information corresponding to each group of lattice point data from a data set; packaging the lattice point data into a lattice point data object according to a preset lattice point data object structure aiming at each group of lattice point data; and aiming at each group of lattice data, writing the lattice data object into a lattice data field of one data record in the database table, and respectively writing the at least one item of attribute information into the attribute field corresponding to the at least one item of attribute information in the data record.
Wherein, the encapsulating the lattice point data into a lattice point data object according to a preset lattice point data object structure comprises: extracting description information and a data part from the lattice point data; respectively processing the description information and the data part according to a preset grid point data object structure; and according to a preset grid point data object structure, packaging the processed description information and the data part into a grid point data object.
The above method may further comprise: receiving a query condition for one or more items of attribute information; the query condition comprises the value or value range of one or more items of attribute information; selecting data records meeting the query conditions from the database table according to the query conditions; and outputting the grid point data object recorded by the grid point data field in the data record.
The above method may further comprise: receiving spatial information to be inquired; for each record in the database table, reading a first lattice point data object from the lattice point data field, and extracting a second lattice point data object from the first lattice point data object according to the spatial information; and outputting the second lattice data object.
Reading a first grid data object from the grid data field, and extracting a second grid data object from the first grid data object according to the spatial information includes: reading a first data part and first description information in the first lattice data object from the lattice data field; extracting a second data part corresponding to the spatial information from the first data part; updating the first description information according to the spatial information to generate second description information corresponding to the second data part; and encapsulating the second description information and the second data portion into the second lattice data object according to the lattice data object structure.
The above method may further comprise: determining the type and parameters of operation to be performed; extracting a data part of a lattice point data object to be operated; respectively operating the values on the grid points of the extracted data part according to the operation type and the operation parameters; and returning the lattice point data object obtained after the operation.
The above method may further comprise: determining the type of operation to be performed; extracting a data part of a lattice point data object to be operated; calculating the values of each grid point of the extracted data part according to the operation type; and returning the lattice point data object obtained after the operation.
The above method may further comprise: determining the type of operation to be performed; extracting data parts of two or more lattice point data objects to be operated; respectively operating the values on the corresponding grid points of the two or more extracted data parts according to the operation type; and returning the lattice point data object obtained after the operation.
Corresponding to the above method, one or more embodiments of the present disclosure further disclose a device for processing lattice data, including:
the database table generating module is used for creating a database table in the database according to a preset lattice point data structure; wherein the database table comprises: a grid data field and at least one attribute field;
the data reading module is used for reading a plurality of groups of lattice point data to be processed and at least one item of attribute information corresponding to each group of lattice point data from the data set;
the data encapsulation module is used for encapsulating the grid point data into a grid point data object according to the grid point data object structure aiming at each group of grid point data; and
and the data writing module is used for writing the lattice point data object into a lattice point data field of one data record of the database table aiming at each group of lattice point data, and respectively writing the at least one item of attribute information into the attribute field corresponding to the data record.
One or more embodiments of the present disclosure also provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the above-described method.
It can be seen that in the embodiment of the present disclosure, the lattice point data can be migrated from the data set to one database table of the database, so that the query and operation capabilities of the database to the data can be fully utilized, real-time query and fast analysis and processing of the lattice point data can be realized without any data download, the required processing time is greatly shortened, and the data processing efficiency is improved.
Especially for massive meteorological grid point data, after the meteorological grid point data are migrated to a database table of a database in the mode, the data in the database can be directly inquired and operated without downloading any meteorological grid point data, so that the processing time required for inquiring and analyzing the meteorological grid point data is greatly shortened, and the data processing efficiency is improved.
Drawings
In order to more clearly illustrate one or more embodiments or prior art solutions of the present disclosure, reference will now be made briefly to the attached drawings, which are used in the description of the embodiments or prior art, and it should be apparent that the attached drawings in the description below are only one or more embodiments of the present disclosure, and that other drawings may be obtained by those skilled in the art without inventive effort.
Fig. 1 is a schematic flow chart illustrating an implementation of a processing method of lattice data according to one or more embodiments of the present disclosure;
fig. 2 is a schematic flowchart of a specific implementation method for encapsulating the grid data into a grid data object according to a preset grid data object structure according to one or more embodiments of the present disclosure;
fig. 3 is a flowchart illustrating a method for querying data according to one or more items of attribute information according to one or more embodiments of the present disclosure;
fig. 4 is a schematic flowchart of a method for querying data according to spatial information according to one or more embodiments of the present disclosure;
fig. 5 is a flowchart illustrating a method for reading a first lattice data object from a lattice data field and extracting a second lattice data object from the first lattice data object according to the spatial information according to one or more embodiments of the present disclosure;
fig. 6 is a schematic internal structure diagram of a device for processing lattice data according to one or more embodiments of the present disclosure; and
fig. 7 is a schematic internal structure diagram of an electronic device according to one or more embodiments of the present disclosure.
Detailed Description
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present disclosure should have the ordinary meaning as understood by one of ordinary skill in the art to which the present disclosure belongs. The use of "first," "second," and similar terms in one or more embodiments of the present disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.
In order to solve the problem that the conventional meteorological grid point data cannot be queried at high performance due to long access time, one or more embodiments of the present disclosure provide a method for storing and querying the meteorological grid point data, which can implement fast and real-time query of the meteorological grid point data.
In embodiments of the present disclosure, to implement fast real-time querying of grid point data, grid point data stored within a relational database plus file system may be migrated to a database. Specifically, in some embodiments of the present disclosure, the database for storing the lattice data may be a development database, and specifically may be a PostgreSql database, or the like.
In order to realize the storage of the lattice point data in the database, in the embodiment of the present disclosure, a storage structure of the lattice point data in the database may be set in advance. In the present disclosure, a storage structure of the preset lattice point data may be referred to as a lattice point data object structure. Therefore, when the grid point data is stored, the grid point data can be packaged into a grid point data object according to a preset grid point data object structure, and then the grid point data object is stored in a database table of the database. The design of the lattice data object structure is related to the specific content of the lattice data, and may generally include a plurality of data items, where one data item may be a data portion of the lattice data stored in a two-dimensional matrix form, and the other data items may be used to store one or more pieces of description information describing the data portion of the lattice data.
For example, in some embodiments of the present disclosure, when the grid data is meteorological grid data, the above-described grid data object structure may be arranged in a structure as shown in table 1 below.
Name of data item Meaning of data item
Data type Data type for recording data portions in a lattice data object structure
Whether or not there is an invalid value For recording whether data part in lattice point data object structure has invalid value
Whether it is an invalid value For recording whether the data parts in the grid data object structure are all invalid data
Position of X axis For recording the longitude of the upper left corner of the data part in the structure of the grid point data object
Y-axis position For recording the latitude of the lower left corner of the data part in the grid data object structure
Resolution of X-axis Resolution for recording X-axis (longitude)
Resolution in the Y-axis Resolution for recording Y-axis (latitude)
Coordinate information For recording coordinate information of data portions in a structure of a lattice data object
Number of rows For recording the number of columns of data parts in a structure of a data object of a grid
Line number For recording the number of rows of data portions in a lattice data object structure
Data part For recording data portions in a structure of a lattice data object (usually stored in the form of a two-dimensional matrix)
TABLE 1
In the example of the present disclosure, the types of the data items in the lattice data object structure shown in table 1 may be further set according to business needs, for example, one of the data items may be set to be one of unsigned integer type, signed integer type, double-precision floating point type, and untyped pointer type, etc. according to business needs.
In addition, in the example of the present disclosure, a plurality of lattice data object structures may be preset, or the lattice data object structures may be continuously updated as the demand of the business improves. In order to distinguish different lattice data object structures, a data item with a version number may be added to each lattice data object structure to specifically indicate the version corresponding to the lattice data structure.
In addition, one or more reserved data items can be further added to the lattice data object structure, so that specific data content borne by the lattice data object structure can be flexibly set according to actual service requirements, and the universality and the expandability of the lattice data object structure are improved.
After the structure of the lattice data object is defined, the processing method of the lattice data can be started.
Fig. 1 shows a flow of implementing a processing method of lattice data according to one or more embodiments of the present disclosure. By the method shown in fig. 1, multiple sets of lattice point data stored in a relational database plus file system manner can be migrated to the database. As shown in fig. 1, the method may include:
in step 102, a database table is created in the database according to the preset grid data structure.
In an embodiment of the present disclosure, the database table is primarily used for storing lattice data. Specifically, the database table may include: a grid data field and at least one attribute field. Wherein, the lattice point data field is used for bearing the lattice point data object; the at least one attribute field is respectively used for bearing at least one item of attribute information corresponding to the lattice point data.
For example, if the source data corresponding to the lattice point data to be processed includes time information and height information, the database table should include a time field and a height field in addition to the lattice point data field. If the source data corresponding to the to-be-processed lattice point data includes time information, height information and sampling information, the database table should include a time field, a height field and a sampling field in addition to the lattice point data field.
In addition, in other embodiments of the present disclosure, in order to identify each record in the database table, the database table may further include an identification field for recording an Identification (ID) of each record in the database table.
It will be appreciated that each of the records in the database tables described above may be used to store a set of lattice data.
Specifically, in some embodiments of the present disclosure, the above operation of creating a database table in a database based on at least one item of attribute information may be implemented using a Structured Query Language (SQL) statement in SQL that creates the table. For example, in one example of the present disclosure, a database table containing identification fields, data lattice fields, and time fields, named test1, may be created in a database by the SQL statement "create table if not existing public. Wherein id represents the above-mentioned identification field, and its type is serial number type (serial); data represents the above-mentioned grid data field, and its type is grid data object type (grid); time represents the above time field, the type of which is numeric type (numeric); test1 represents the name of the database table described above.
After the structure of the lattice point data object is defined and the database table is created in the database, the lattice point data stored in the file system can be migrated to the database table of the database, so that the real-time and rapid query of the lattice point data can be realized by utilizing the query function of the database.
In step 104, a plurality of sets of lattice point data to be processed and at least one item of attribute information corresponding to each set of lattice point data are read from the dataset.
In embodiments of the present disclosure, the data set generally refers to a data set storing data in an existing relational database plus file system. In an example of the present disclosure, the data set stores grid point data and corresponding source data. Specifically, the data set may include: GRIB, Network Common Data Format (NetCDF), and Hierarchical Data Format (HDF 5), among others. Among them, the GRIB format is a concise data format applied to the meteorological field, standardized by the world meteorological organization, and generally used for storing historical meteorological data. The NetCDF file, which was originally intended for storing data in meteorological science, has now become the format of the generation file for many data collection software. HDF5 was developed by the university of illinois, ebana champagne, usa, and is a common cross-platform data storage file that can store different types of images and data and can be transmitted on different types of machines, along with a library of functions that uniformly handle this file format.
Specifically, taking the NetCDF file storing the grid point data as an example, in some embodiments of the present disclosure, in step 104, first, the NetCDF file may be read from the NetCDF database by using the C language, and an array of longitudes, an array of latitudes, and an array of data itself of the NetCDF file are obtained. And taking the precipitation data as columns, reading a NetCDF file for storing the precipitation data from a NetCDF database, and acquiring a longitude array, a latitude array and a precipitation data array of the precipitation data. The resolution of the X axis and the Y axis, the coordinate information of the positions of the X axis and the Y axis, and the like in the grid point data object structure can be obtained through the longitude array and the latitude array. And judging whether the effective value exists or not through the precipitation array. After the data information is acquired, the information is written into the lattice data object structure in a binary form according to the structure sequence of the lattice data object structure.
It is understood that, in the embodiment of the present disclosure, the lattice point data to be processed may include multiple sets of lattice point data, where, as described above, each set of lattice point data may include a data portion of the lattice point data stored in a two-dimensional matrix and description information describing the data portion. In addition, the at least one item of attribute information corresponding to each set of lattice data may include: one or more of time information, height information, sampling information, and the like corresponding to the lattice data may also be referred to as source data of the lattice data.
In some embodiments of the present disclosure, the grid point data may be meteorological grid point data. For example, the grid point data may be daily precipitation grid point data. In this case, the daily precipitation grid point data may include a plurality of sets of daily precipitation grid point data, that is, daily precipitation grid point data for a plurality of days in a period of time. Wherein each group of daily precipitation grid point data corresponds to a date; the data part of each group of daily precipitation grid point data is the total precipitation of geographic positions determined by different longitudes and latitudes in a certain area in a corresponding date stored in a two-dimensional matrix form. The description information corresponding to the data portion is generally used to describe the geographic location of the area, for example, the latitude and longitude range of the area and the precision of each grid point. At this time, the at least one item of attribute information corresponding to each set of daily precipitation grid point data may include a date corresponding to the precipitation grid point data. For another example, the lattice point data may be average daily temperature lattice point data. In this case, the average daily temperature lattice data may include a plurality of sets of average daily temperature lattice data, where each set of average daily temperature lattice data corresponds to a date and a geographic height; the data part of each group of the average daily temperature lattice data is the average daily temperature of each geographical position determined by different longitudes and latitudes in a certain area stored in a two-dimensional matrix form. The description information corresponding to the data portion is generally used to describe the geographic location of the area, for example, the latitude and longitude range of the area and the precision of each grid point. At this time, the at least one item of attribute information corresponding to each group of average daily temperature lattice data may include a date and a geographic height corresponding to the average daily temperature lattice data.
In step 106, for each set of lattice data, the lattice data is encapsulated into a lattice data object according to a preset lattice data object structure.
In one or more embodiments of the present disclosure, in the step 106, a specific implementation method for encapsulating the grid data into a grid data object according to a preset grid data object structure may be as shown in fig. 2, and specifically includes:
in step 202, description information and data portions are extracted from the lattice point data.
Specifically, as described above, the above-mentioned lattice point data generally includes a data portion stored in a two-dimensional matrix and description information describing the data portion, and therefore, in this step, the description information and the data portion may be separated from the above-mentioned lattice point data according to the storage manner of the lattice point data.
In step 204, the description information and the data portion are processed according to a preset grid data object structure.
Specifically, in this step, the extracted description information and data part may be processed according to the meaning and type of each data item in the lattice data object structure, respectively, so as to meet the requirement of the lattice data object structure.
Furthermore, the values of all the lattice points in the lattice point data part can be preprocessed according to the service requirement. For example, typically in an existing dataset, each grid point data file corresponds to a particular date and a particular time. If the actual service requirement only needs data corresponding to a specific date, preprocessing (adding, averaging, etc.) the data in the grid point data files corresponding to the same date and different times can be performed to obtain the data meeting the actual service requirement.
In step 206, the processed description information and data portion are encapsulated into a grid data object according to a preset grid data object structure.
Specifically, in this step, the processed description information and the data part may be combined together according to the grid data object structure according to the arrangement order of the data items in the grid data object structure, so as to obtain the grid data object.
In an example of the present disclosure, the process shown in fig. 2 described above may be implemented by an SQL function. The function can be written in C language at the bottom. By which the lattice data can be read in bytes and converted into lattice data objects.
Still taking the meteorological grid point data as an example, if the meteorological grid point data to be processed is daily precipitation grid point data, the precipitation grid point data of each day can be respectively packaged into a grid point data object through the above step 106. If the weather lattice point data to be processed is the average daily temperature lattice point data, the average daily temperature lattice point data corresponding to the same geographical height every day may be respectively packaged as a lattice point data object through the above step 106.
In step 108, for each set of lattice data, the lattice data object is written into a lattice data field of a data record in the database table, and the at least one item of attribute information is written into an attribute field corresponding to the at least one item of attribute information in the data record.
In embodiments of the present disclosure, one data record may be generated for each set of lattice data.
Specifically, in this step, the SQL statement that inserts data in SQL may be used to implement the operations of writing the lattice data object into a lattice data field of a data record in the database table, and writing the at least one item of attribute information into the attribute field corresponding to the at least one item of attribute information in the data record, respectively. For example, in one example, data may be inserted in various fields of a database table named test1 in the database through the SQL statement "insert to test1(data, attri1, attri2 …) values (grid, value1, value2 …)"; wherein, the data represents the lattice point data object field in the database table; attri1, attri2, etc. represents at least one attribute field in the database table; grid represents a grid data object; the value1, value2, and the like represent at least one attribute information corresponding to the above-described lattice data.
Still taking the meteorological grid point data as an example, if the meteorological grid point data to be processed is daily precipitation grid point data, then through the above step 108, a plurality of records may be generated in the pre-established database table, where each record stores daily precipitation for one day. That is, each record stores the date corresponding to the record and the daily precipitation grid data object. If the weather grid point data to be processed is average japanese grid point data, then through step 108, a plurality of records may be generated in the pre-established database table, where each record stores the average day temperature of the next day at a certain geographic altitude. That is, each record stores the date and the geographic height corresponding to the record, and the average daily temperature lattice data object corresponding to the date and the geographic height.
It can be seen that, in the embodiment of the present disclosure, by the method shown in fig. 1, the lattice point data can be migrated from the data set to one database table of the database, so that the query and operation capabilities of the database to the data can be fully utilized, real-time query and fast analysis and processing of the lattice point data can be realized without downloading any data, the required processing time is greatly shortened, and the data processing efficiency is improved. Especially for massive meteorological grid point data, after the meteorological grid point data are migrated to a database table of a database in the mode, the data in the database can be directly inquired and operated without downloading any meteorological grid point data, so that the processing time required for inquiring and analyzing the meteorological grid point data is greatly shortened, and the data processing efficiency is improved.
The method for migrating the lattice point data from the data set to the database table is described below by taking the NOAA _ OLR long-wave radiation information as the lattice point data. The NOAA satellite is the third generation practical meteorological observation satellite of the national oceanographic agency of america. The OLR is generally called as emitting long-wave Radiation (Outgoing long wave Radiation), and emits long-wave Radiation information for a ground-gas system monitored by polar satellite remote sensing. Thus, the NOAA _ OLR is long-wave radiation information emitted by a ground-gas system monitored by NOAA satellite remote sensing, and may be referred to as NOAA _ OLR lattice point data in the present disclosure for short. The text file of the NOAA _ OLR lattice point data acquired from the data set records OLR radiation wavelength data acquired twice on the same day. In the method of the present disclosure, first, a text file of NOAA _ OLR lattice data corresponding to a certain day is read from an official website of the NOAA satellite; then, acquiring numerical values of OLR radiation wavelengths acquired twice in the day; and calculating the average value of the numerical values of the OLR radiation wavelengths acquired twice, and taking the calculated average value as the data part of the lattice data object. Then, obtaining the coordinates of the starting point, the resolution of the X axis and the Y axis, the row number and the column number and the like according to the space range and the longitude and latitude information given by the NOAA satellite official network; then, these pieces of information are written in binary form into the description portion of the above-mentioned lattice data object in the structural order of the above-mentioned lattice data object structure. Thus, a lattice data object is obtained. Thereafter, an identification of the lattice data object may be further generated; and taking the corresponding time as the time attribute information of the lattice data object, further generating a data record together with the lattice data object, and writing the data record into a database table named NOAA _ OLR. Thus, the generated database table of the NOAA _ OLR can contain a plurality of data records, wherein each data record corresponds to the long-wave radiation information emitted by the ground-gas system monitored by NOAA satellite remote sensing at a certain day.
On the basis of the embodiment of migrating the lattice point data from the file system to the database table shown in fig. 1, one or more embodiments of the present disclosure also disclose a data query method and an operation method for the lattice point data in the database table.
The following describes in detail a data query method for lattice data in a database table. In an embodiment of the present disclosure, the data query method may include data query according to one or more items of attribute information and spatial query according to spatial information. Of course, in the embodiment of the present disclosure, the joint query may also be performed according to one or more items of attribute information and the spatial information at the same time. Specifically, in the embodiment of the present disclosure, the spatial information specifically refers to position information of each lattice point to be queried in the lattice point data. The spatial information may specifically refer to longitude and latitude information of each grid point to be queried in the weather grid point data, corresponding to the weather grid point data.
FIG. 3 shows a flow chart of a method for querying data based on one or more items of attribute information. As shown in fig. 3, the method may include:
at step 302, a query for one or more items of attribute information is received.
In this step, the query condition may be a value or a value range of one or more items of attribute information. For example, for the meteorological site data, there may be a time value or time range of the meteorological site data to be queried, a height value or height range, a sampling value or sampling value range, and so on.
In step 304, data records satisfying the query condition are selected from the database table according to the received query condition for one or more items of attribute information.
At step 306, the grid data object recorded by the grid data field in the selected data record is output.
Specifically, in this step, the SQL statement for selecting data in SQL may be used to implement the operation of selecting the data records satisfying the query condition from the database table according to the received query condition for one or more items of attribute information and outputting the corresponding lattice data object. For example, in one example, the data query described above can be implemented by a select statement in SQL. For example, in one example of the present disclosure, the database table stores daily precipitation data for 1 month and each day of 2021 year within the territory of the people's republic of china. In addition, the query condition may be specifically 1/2021. Then, by the data query method shown in fig. 3, a grid point data object storing daily precipitation data of 2021 year, 1 month and 1 day in the region range of the people's republic of china can be obtained.
Therefore, the query method can be used for rapidly querying the grid point data according to one or more items of attribute information. In addition, the query operation is a direct operation on the data in the database, so that the query capability of the database is fully utilized, all data do not need to be downloaded to the local first and then queried, and the rapid query of the grid point data can be realized.
Fig. 4 shows a flow chart of a method for performing data query according to spatial information. As shown in fig. 4, the method may include:
at step 402, spatial information to be queried is received.
In this step, the spatial information is used as a query condition, and may be a coordinate range of an X axis and a Y axis corresponding to a data portion in the lattice point data to be queried. Taking meteorological data as an example, the spatial information generally refers to a longitude and latitude range of an area to be queried. In some embodiments of the present disclosure, the latitude and longitude range may be determined according to a numerical range input by a user, or may be determined according to a polygon drawn on a graphical user interface by the user. In some embodiments of the present disclosure, the spatial information may be carried by a javascript object notation (JSON) object.
In step 404, for each record in the database table, a first lattice data object is read from the lattice data field, and a second lattice data object is extracted from the first lattice data object according to the above-mentioned spatial information.
In step 406, the second lattice data object is output.
In an embodiment of the present disclosure, a specific implementation method of the step 404 may be as shown in fig. 5, and specifically includes the following steps:
at step 502, reading a first data portion and first description information in a first lattice data object from a lattice data field;
at step 504, extracting a second data portion corresponding to the spatial information from the first data portion;
in step 506, updating the first description information according to the spatial information, and generating second description information corresponding to a second data portion;
at step 508, the generated second description information and second data portion are encapsulated as a second lattice data object in accordance with the lattice data object structure.
The method for extracting the second data portion corresponding to the spatial information from the first data portion in step 504 is described in detail below with a specific example.
As previously described, in step 502 above, the first data portion and the first description information may be read from the lattice data field.
Step 504 may include:
in step 5042, the position (starting longitude and latitude) of the X axis and the Y axis corresponding to the first data portion, the resolution, and the number of rows and columns of the first data portion are obtained from the first description information.
In step 5044, the position (starting longitude and latitude) of the X-axis and the Y-axis corresponding to the first data portion, the resolution, and the number of rows and columns of the first data portion are calculated to obtain the spatial range of the first data portion.
Specifically, in an example of the present disclosure, the above calculation formula of the spatial range of the first data portion is as follows:
x-axis end longitude = X-axis start longitude + X-axis resolution X number of columns;
y-axis ending latitude = Y-axis starting latitude + Y-axis resolution x number of rows;
in step 5046, a row and column number of the first data portion corresponding to the spatial information is determined according to the spatial information.
Specifically, in an example of the present disclosure, the spatial information may be specifically latitude and longitude ranges maxlat, minlat, maxlon, and minlon input by the user; wherein maxlat represents the maximum value of the latitude in the latitude and longitude range; minlat represents the minimum value of the latitude in the latitude and longitude range; maxlon represents the maximum value of the longitude in the latitude and longitude range; and minlon represents the minimum value of the longitudes in the latitude and longitude range.
Specifically, in step 5046, a method of specifically determining the row/column number of the first data portion corresponding to the spatial information is different depending on whether the starting longitude of the X axis is the maximum longitude or the minimum longitude and the starting latitude of the Y axis is the maximum latitude or the minimum latitude.
For example, if the X-axis starting longitude is the maximum longitude, then the column number method for determining maxlon, minlon is as follows:
(1) it is determined whether maxlon is greater than the X-axis starting longitude. In response to determining that maxlon is greater than the X-axis starting longitude, maxlon is assigned to the X-axis starting longitude. In response to determining that maxlon is less than or equal to the X-axis starting longitude, maxlon is held constant.
(2) It is determined whether minlon is less than the X-axis end longitude. In response to determining minlon is less than the X-axis end longitude, assigning minlon to the X-axis end longitude. In response to determining that minlon is greater than or equal to the X-axis end longitude, minlon is held constant.
(3) Calculating the column numbers corresponding to maxlon and minlon, wherein the specific calculation formula is as follows:
column number of (X-axis starting longitude-minlon)/abs (X-axis resolution) = minlon;
column number of (X-axis starting longitude-maxlon)/abs (X-axis resolution) = maxlon;
wherein abs () represents an absolute value operation.
For another example, if the X-axis starting longitude is the minimum longitude, the column number method for determining maxlon and minlon is as follows:
(1) it is determined whether minlon is less than the X-axis starting longitude. In response to determining that minlon is less than the X-axis starting longitude, minlon is assigned to the X-axis starting longitude. In response to determining that minlon is greater than or equal to the X-axis starting longitude, minlon is held constant.
(2) It is determined whether maxlon is greater than the X-axis end longitude. In response to determining that maxlon is greater than the X-axis end longitude, maxlon is assigned an X-axis end longitude. In response to determining that maxlon is less than or equal to the X-axis end longitude, maxlon is held constant.
(3) Calculating the column numbers of the mapping of maxlon and minlon, wherein the specific calculation formula is as follows:
(minlon-x axis starting longitude)/abs (x axis resolution) = column number of minlon;
column number of (maxlon-x axis starting longitude)/abs (x axis resolution) = maxlon;
the method for determining the row number corresponding to maxlat and minlat is basically the same as the method for determining the column number corresponding to maxlon and minlon, and the description thereof will not be repeated. Thus, the row and column numbers of maxlat, minlat, maxlon and minlon range mapping are obtained by the above method.
In step 5048, according to the row and column numbers of the first data portion corresponding to the spatial information, data corresponding to the row and column numbers of the first data portion is extracted from the first data portion as the second data portion.
Next, in step 506, the number of rows and columns corresponding to the second data portion, and the starting longitude and latitude, resolution and other information of the X axis and the X axis in the number of rows and columns are determined again according to the row and column number of the first data portion corresponding to the spatial information, so as to generate the second description information. Next, at step 508, the generated second description information and the second data portion are encapsulated as a second lattice data object, which is returned to the user as a result of the spatial query.
Therefore, the query method can rapidly query the lattice point data according to the spatial information, namely, the spatial query of the lattice point data is realized. For example, in one example of the present disclosure, the first lattice point data object stores daily precipitation data of 1 month and 1 day of 2021 year in a regional scope of the people's republic of china; the first description information may include the initial longitude and latitude, the resolution, the number of rows and columns of daily precipitation data, and the like of the region range of the people's republic of china; the first data part comprises daily precipitation numerical values of all longitude and latitude grid points in the range described by the first description information. In addition, the spatial information to be queried may specifically be a longitude and latitude range of beijing. A second lattice point data object representing daily precipitation data of 1 month and 1 day of 2021 in beijing may be obtained by the above-described spatial query method shown in fig. 4 and 5. The second description information may include the initial longitude and latitude, resolution, number of rows and columns of daily precipitation data, and the like of the beijing city region range; the second data part comprises daily precipitation data corresponding to each longitude and latitude grid point in the range described by the second description information.
It can be seen that, by the spatial query method shown in fig. 4 and 5, a spatial clipping function for lattice data can be implemented, that is, spatial clipping is performed on lattice data according to given spatial information. Specifically, the spatial information of the lattice point data to be cut out may be used as the spatial information shown in fig. 4, so that the execution result of fig. 4, that is, the second lattice point data object, may be regarded as the result of spatially cutting out the lattice point data according to the given spatial information. Furthermore, a new database table can be created, and the second lattice point data object is written into the lattice point data field of the new database table, so that the new database table for storing the lattice point data subjected to the spatial cutting on the lattice point data in the source database table can be obtained.
In other embodiments of the present disclosure, a method for performing a joint query simultaneously according to one or more items of attribute information and spatial information may also be implemented by combining the methods shown in fig. 3, fig. 4, and fig. 5, and a specific process will not be described again here. In addition, the joint query method also fully utilizes the query capability of the database, and the data does not need to be downloaded to the local, so that the rapid joint query of the lattice point data can be realized.
After the database table is generated, in addition to the data query function, some embodiments of the present disclosure also provide an operation method for performing mathematical operations on the lattice data in the database table. In an embodiment of the present disclosure, the operation may include: addition, subtraction, multiplication, division, averaging, maximum, minimum, variance, and dimension reduction, among others.
From the viewpoint of the operation object, the operation may be an operation performed on a single lattice data object, or may be an operation performed on two or more lattice data objects.
In some embodiments of the present disclosure, the operation performed on one lattice data object may include: addition, subtraction, multiplication, division, and dimension reduction operations, among others. The above-described operation performed on one lattice data object may be further divided into operation operations including operation parameters, such as addition, subtraction, multiplication, division operation, and the like, and operation operations not including operation parameters, such as dimension reduction operation, and the like.
Specifically, in the embodiment of the present disclosure, the operation performed on one lattice data object and including the operation parameter may be implemented by the following method:
first, the type of operation and the operation parameters to be performed are determined.
Next, the data portion of the lattice data object to be computed is extracted.
In this step, the lattice point data object to be operated may be extracted from the established database table by the above query method, and then the data portion of the lattice point data object to be operated may be extracted therefrom.
Then, the values at the respective lattice points of the extracted data portion are respectively calculated based on the operation type and the operation parameter.
And finally, returning the lattice point data object obtained after the operation.
For example, for an operation of adding a set value, the operation type is addition and the operation parameter is the set value. In this case, the operation result of the above method is a lattice data object obtained by adding the set numerical value to the value of each lattice of one lattice data object data portion.
In an embodiment of the present disclosure, the operation performed on one lattice data object and not including the operation parameter may be implemented by the following method:
first, the type of operation that needs to be performed is determined.
Next, the data portion of the lattice data object to be computed is extracted.
In this step, the lattice point data object to be operated may be extracted from the established database table by the above query method, and then the data portion of the lattice point data object to be operated may be extracted therefrom.
Then, the values at the respective lattice points of the extracted data portion are operated on according to the operation type described above.
And finally, returning the lattice point data object obtained after the operation.
For example, for dimension reduction operations, no operational parameters are needed. Generally, the dimension reduction operation includes two categories of dimension reduction according to an X axis (longitude) and dimension reduction according to a Y axis (latitude), and specific operation types include summation, averaging, maximum value calculation, minimum value calculation, and the like. The dimension reduction according to the X axis means that a plurality of values in each column of the data portion are calculated (summed, averaged, maximized, or minimized, etc.) to obtain one value. Dimensionality reduction according to the Y-axis refers to the operation (summing, averaging, maximizing or minimizing, etc.) of multiple values in each row of the data portion to obtain one value. Through the dimension reduction operation, the dimension of the data part of each lattice point data object is reduced into a one-dimensional array by a two-dimensional matrix. That is, the data portion of the lattice point data object obtained after the operation becomes a one-dimensional array.
In other embodiments of the present disclosure, the operation performed on two or more lattice data objects includes: addition, subtraction, multiplication, division, averaging, maximum or minimum, etc. Such operations generally require that the two or more lattice data objects participating in the operation should have the same specification and precision, i.e., that the data portions in the two or more lattice data objects participating in the operation should have the same number of rows and columns and that the precision of the rows and columns should be the same.
Specifically, in the embodiment of the present disclosure, the operation performed on two or more grid data objects may be implemented by the following method:
first, the type of operation that needs to be performed is determined.
Next, data portions of two or more lattice data objects to be computed are extracted.
In this step, two or more lattice point data objects to be operated may be extracted from the established database table by the above query method, and then data portions of the two or more lattice point data objects to be operated may be extracted therefrom.
Then, the values at the respective corresponding lattice points of the extracted two or more data parts are respectively operated on according to the operation types described above.
And finally, returning the lattice point data object obtained after the operation.
For example, for an averaging operation, the operation type is averaging. In this case, the operation result of the above method is a lattice data object obtained by averaging the values of each corresponding lattice of the plurality of lattice data object data portions.
Taking daily precipitation grid point data as an example, if the total precipitation of one month needs to be obtained, extracting precipitation grid point data objects of days in one month to be counted from a database table; and adding the data parts of the precipitation amount grid point data objects of each day to obtain the grid point data object of the monthly precipitation amount to be counted for one month. Specifically, the value of each grid point of the data part is the total precipitation of the longitude and latitude position corresponding to the grid point in the month. If the average precipitation in one month is required to be calculated, extracting a precipitation grid point data object to be counted for each day in one month from a database table; and averaging the data part of the precipitation grid point data object of each day to obtain the grid point data object of the average precipitation of the month to be counted for one month. Specifically, the value of each grid point of the data part is the average precipitation of the longitude and latitude position corresponding to the grid point in the month.
In general, the above query and operation functions are often used in combination. That is, a query condition (including a query condition and/or spatial information for one or more items of attribute information) is given first, and after one or more lattice data objects satisfying the query condition are queried, a specified operation is performed on the one or more lattice data objects obtained through query, so as to obtain an operation result. The result of the operation may be one or more lattice data objects.
In an example of the present disclosure, the above-mentioned various query and operation functions may be implemented by SQL functions. These functions can be written in the C language at the bottom. The functions can complete the query of the lattice data according to the query conditions, or can perform corresponding operation on the lattice data according to the operation type and the operation parameters.
On the basis of the lattice point data query and operation functions, various statistical and analysis functions for lattice point data objects can be realized. And because the query and operation methods fully utilize the query and calculation capacity of the database, the data does not need to be downloaded to the local, and the fast operation, various statistics and analysis of the lattice point data can be realized. For example, for precipitation data, if the lattice point data processing method disclosed by the present disclosure is not used, when performing calculation functions such as spatial query, average calculation, difference calculation, and statistics functions such as maximum value and minimum value, the precipitation data needs to be downloaded and transmitted to the local, and then the precipitation data is processed by locally deployed calculation statistics services, and finally the calculation result can be sent to the user. Obviously, such a data processing method may cause redundant transmission of data, resulting in a reduction in efficiency of the entire data calculation process. After the data processing method disclosed by the invention is adopted, when data is queried, only the packaged calculation functions for grid point data calculation, such as query, average value calculation, maximum and minimum value calculation and the like, need to be called according to the SQL syntax specification. The method does not need to transmit the data to the local for calculation, but directly calculates in the database, and the database directly returns the result to the user, thereby improving the efficiency of the whole data calculation process.
As described above, the results of the above-mentioned lattice point data query and operation are all lattice point data objects, and if lattice point data files in other formats need to be returned according to the requirements of the service, the embodiments of the present disclosure may further repackage the lattice point data to be returned into other formats according to the standard requirements of other formats, and then return the lattice point data to the user.
It should be noted that, although the examples of the present disclosure are described by taking the meteorological grid point data as an example, the above methods provided by the present disclosure are also applicable to other grid point data besides the meteorological grid point data. It is understood that the difference between the meteorological lattice data and other lattice data is only in the difference of the description information and thus different lattice data object structures may be applicable, but the difference of the lattice data object structures does not affect the implementation of the method provided by the present disclosure. Thus, the methods provided by the present disclosure are equally applicable to other grid point data.
Based on the above method, one or more embodiments of the present disclosure further provide a processing apparatus for grid point data, an internal structure of which is shown in fig. 6, and the processing apparatus mainly includes:
the database table generating module 602 is configured to create a database table in a database according to a preset lattice data structure. Wherein, the database table may include: a grid data field and at least one attribute field.
A data reading module 604, configured to read multiple sets of to-be-processed lattice point data and at least one item of attribute information corresponding to each set of lattice point data from a data set;
a data encapsulation module 606, configured to encapsulate, for each group of lattice data, the lattice data into a lattice data object according to the lattice data object structure; and
a data writing module 608, configured to write the lattice data object into a lattice data field of a data record in the database table for each set of lattice data, and write the at least one item of attribute information into the attribute field corresponding to the at least one item of attribute information in the data record, respectively.
In some embodiments of the present disclosure, the apparatus for processing lattice data may further include a data query module, configured to receive a query condition, extract a corresponding lattice data object from the database table according to the query condition, and return the extracted lattice data object. Specifically, the query condition includes a query condition and/or spatial information for one or more items of attribute information.
In some embodiments of the disclosure, the processing apparatus of lattice data may further include a data operation module, configured to determine an operation type, perform an operation on one or more lattice data objects according to the operation type, and return the lattice data objects obtained through the operation.
The specific implementation manner of each module may refer to the method of the foregoing embodiment, and a description thereof is not repeated here.
It should be noted that the method of one or more embodiments of the present disclosure may be performed by a single device, such as a computer or server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the devices may only perform one or more steps of the method of one or more embodiments of the present disclosure, and the devices may interact with each other to complete the method described above.
The foregoing description of specific embodiments of the present disclosure has been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the various modules may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present disclosure.
The apparatus of the foregoing embodiment is used to implement the corresponding method in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Fig. 7 is a schematic diagram of a more specific hardware structure of an electronic device according to an embodiment of the present disclosure, where the electronic device may include: processor 710, memory 720, input/output interface 730, communication interface 740, and bus 750. Processor 710, memory 720, input/output interface 730, and communication interface 740 are communicatively coupled to each other within the device via bus 750.
The processor 710 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the method for Processing the lattice data provided by the embodiment of the present disclosure.
The Memory 720 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 720 may store an operating system and other application programs, and when the processing method of the lattice point data provided by the embodiments of the present disclosure is implemented by software or firmware, the relevant program codes are stored in the memory 720 and called by the processor 710 for execution.
The input/output interface 730 is used for connecting an input/output module to realize information input and output. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 740 is used for connecting a communication module (not shown in the figure) to implement communication interaction between the present device and other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
Bus 750 includes a path that transfers information between various components of the device, such as processor 710, memory 720, input/output interface 730, and communication interface 740.
It should be noted that although the above-described device only shows the processor 710, the memory 720, the input/output interface 730, the communication interface 740, and the bus 750, in a specific implementation, the device may also include other components necessary for normal operation. Moreover, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present disclosure, and need not include all of the components shown in the figures.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the disclosure as described above, which are not provided in detail for the sake of brevity, within the spirit of the disclosure.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring one or more embodiments of the present disclosure, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which one or more embodiments of the present disclosure are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
The one or more embodiments of the present disclosure are intended to embrace all such alternatives, modifications and variances that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the disclosure are intended to be included within the scope of the disclosure.

Claims (10)

1. A method for processing lattice point data comprises the following steps:
creating a database table in a database according to a preset lattice point data structure; wherein the database table comprises: a grid data field and at least one attribute field;
reading a plurality of groups of lattice point data to be processed and at least one item of attribute information corresponding to each group of lattice point data from a data set;
packaging the lattice point data into a lattice point data object according to a preset lattice point data object structure aiming at each group of lattice point data; and
and aiming at each group of lattice data, writing the lattice data object into a lattice data field of one data record in the database table, and respectively writing the at least one item of attribute information into the attribute field corresponding to the at least one item of attribute information in the data record.
2. The method of claim 1, wherein the packaging the lattice data into a lattice data object according to a preset lattice data object structure comprises:
extracting description information and a data part from the lattice point data;
respectively processing the description information and the data part according to a preset grid point data object structure; and
and according to a preset grid point data object structure, packaging the processed description information and the data part into a grid point data object.
3. The method of claim 1, further comprising:
receiving a query condition for one or more items of attribute information; the query condition comprises the value or value range of one or more items of attribute information;
selecting data records meeting the query conditions from the database table according to the query conditions;
and outputting the grid point data object recorded by the grid point data field in the data record.
4. The method of claim 1, further comprising:
receiving spatial information to be inquired;
for each record in the database table, reading a first lattice point data object from the lattice point data field, and extracting a second lattice point data object from the first lattice point data object according to the spatial information; and
and outputting the second lattice point data object.
5. The method of claim 4, wherein reading a first grid data object from the grid data field and extracting a second grid data object from the first grid data object according to the spatial information comprises:
reading a first data part and first description information in the first lattice data object from the lattice data field;
extracting a second data part corresponding to the spatial information from the first data part;
updating the first description information according to the spatial information to generate second description information corresponding to the second data part; and
and according to the grid point data object structure, packaging the second description information and the second data part into the second grid point data object.
6. The method of claim 1, further comprising:
determining the type and parameters of operation to be performed;
extracting a data part of a lattice point data object to be operated from the database table;
respectively operating the values on the grid points of the extracted data part according to the operation type and the operation parameters; and
and returning the lattice point data object obtained after the operation.
7. The method of claim 1, further comprising:
determining the type of operation to be performed;
extracting a data part of a lattice point data object to be operated from the database table;
calculating the values of each grid point of the extracted data part according to the operation type; and
and returning the lattice point data object obtained after the operation.
8. The method of claim 1, further comprising:
determining the type of operation to be performed;
extracting data parts of two or more lattice point data objects to be operated from the database table;
respectively operating the values on the corresponding grid points of the two or more extracted data parts according to the operation type; and
and returning the lattice point data object obtained after the operation.
9. An apparatus for processing lattice data, comprising:
the database table generating module is used for creating a database table in the database according to a preset lattice point data structure; wherein the database table comprises: a grid data field and at least one attribute field;
the data reading module is used for reading a plurality of groups of lattice point data to be processed and at least one item of attribute information corresponding to each group of lattice point data from the data set;
the data encapsulation module is used for encapsulating the grid point data into a grid point data object according to the grid point data object structure aiming at each group of grid point data; and
and the data writing module is used for writing the lattice point data object into a lattice point data field of one data record of the database table aiming at each group of lattice point data, and respectively writing the at least one item of attribute information into the attribute field corresponding to the data record.
10. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the processing method of lattice data according to any one of claims 1 to 8.
CN202110305003.6A 2021-03-23 2021-03-23 Processing method of lattice point data and related equipment thereof Active CN112699102B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110305003.6A CN112699102B (en) 2021-03-23 2021-03-23 Processing method of lattice point data and related equipment thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110305003.6A CN112699102B (en) 2021-03-23 2021-03-23 Processing method of lattice point data and related equipment thereof

Publications (2)

Publication Number Publication Date
CN112699102A true CN112699102A (en) 2021-04-23
CN112699102B CN112699102B (en) 2021-06-29

Family

ID=75515325

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110305003.6A Active CN112699102B (en) 2021-03-23 2021-03-23 Processing method of lattice point data and related equipment thereof

Country Status (1)

Country Link
CN (1) CN112699102B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114911853A (en) * 2022-04-20 2022-08-16 中山大学 Python-based hydrological meteorological data area scale extraction and visualization method
CN116126973A (en) * 2022-12-29 2023-05-16 国家气象信息中心(中国气象局气象数据中心) Meteorological lattice data management method and device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102693293A (en) * 2012-05-15 2012-09-26 清华大学 Range query method and system for multivariable spatio-temporal data
CN105224714A (en) * 2015-08-31 2016-01-06 中国华能集团清洁能源技术研究院有限公司 The disposal route of weather data and device
CN105930483A (en) * 2016-04-29 2016-09-07 北京数码大方科技股份有限公司 Object format generation method, apparatus and system
JP6501561B2 (en) * 2015-03-06 2019-04-17 キヤノン株式会社 DATA PROCESSING APPARATUS, CONTROL METHOD THEREOF, AND PROGRAM
CN112214472A (en) * 2020-09-02 2021-01-12 国家气象信息中心 Meteorological grid point data storage and query method, device and storage medium
CN112347118A (en) * 2021-01-08 2021-02-09 阿里云计算有限公司 Data storage, query and generation method, database engine and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102693293A (en) * 2012-05-15 2012-09-26 清华大学 Range query method and system for multivariable spatio-temporal data
JP6501561B2 (en) * 2015-03-06 2019-04-17 キヤノン株式会社 DATA PROCESSING APPARATUS, CONTROL METHOD THEREOF, AND PROGRAM
CN105224714A (en) * 2015-08-31 2016-01-06 中国华能集团清洁能源技术研究院有限公司 The disposal route of weather data and device
CN105930483A (en) * 2016-04-29 2016-09-07 北京数码大方科技股份有限公司 Object format generation method, apparatus and system
CN112214472A (en) * 2020-09-02 2021-01-12 国家气象信息中心 Meteorological grid point data storage and query method, device and storage medium
CN112347118A (en) * 2021-01-08 2021-02-09 阿里云计算有限公司 Data storage, query and generation method, database engine and storage medium

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114911853A (en) * 2022-04-20 2022-08-16 中山大学 Python-based hydrological meteorological data area scale extraction and visualization method
CN114911853B (en) * 2022-04-20 2022-11-29 中山大学 Python-based hydrological meteorological data regional scale extraction and visualization method
CN116126973A (en) * 2022-12-29 2023-05-16 国家气象信息中心(中国气象局气象数据中心) Meteorological lattice data management method and device

Also Published As

Publication number Publication date
CN112699102B (en) 2021-06-29

Similar Documents

Publication Publication Date Title
CN112699102B (en) Processing method of lattice point data and related equipment thereof
US10387457B2 (en) Grid-based analysis of geospatial trajectories
US8370303B1 (en) Generating snapshots of data tables
Tsendbazar et al. Integrating global land cover datasets for deriving user-specific maps
CN112214472B (en) Meteorological lattice data storage and query method, device and storage medium
CN111447292B (en) IPv6 geographical position positioning method, device, equipment and storage medium
US20180095959A1 (en) Large scale processor for satellite data
Lewis et al. Lidar data management pipeline; from spatial database population to web-application visualization
Liang et al. Sentinel-1 EW mode dataset for Antarctica from 2014–2020 produced by the CASEarth Cloud Service Platform
Haynes et al. Terra Populus’ architecture for integrated big geospatial services
Sharma et al. Satellite data planning for flood mapping activities based on high rainfall events generated using TRMM, GEFS and disaster news
CN112700547B (en) Map making method and related equipment
CN117557681B (en) High-precision topographic map generation method and device based on multi-source mapping data
CN113220728B (en) Data query method, device, equipment and storage medium
Gyori et al. GOP-TropDB Developments for Tropospheric Product Evaluation and Monitoring: Design, Functionality and Initial Results
Cao et al. A large scale training sample database system for intelligent interpretation of remote sensing imagery
CN107292232B (en) Method and system for detecting precision of remote sensing classified products
CN112506948A (en) Index query method of service information and related equipment
CN104881425A (en) WebGIS (geographic information system) map optimization based gradation loading method for tens of thousands of cameras
CN115357675B (en) Method and system for establishing image control point database through standardized processing of image control points
Krdžalić et al. A precise geoid model of Bosnia and Herzegovina by the KTH method and its validation
CN112560413B (en) Report form expansion method, device and equipment based on configuration mode
US9436715B2 (en) Data management apparatus and data management method
CN110110016B (en) Remote sensing image analysis method and device
Yengoh et al. Main global NDVI datasets, databases, and software

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant