WO2024021476A1 - 一种数据处理方法、装置、电子设备和存储介质 - Google Patents

一种数据处理方法、装置、电子设备和存储介质 Download PDF

Info

Publication number
WO2024021476A1
WO2024021476A1 PCT/CN2022/141519 CN2022141519W WO2024021476A1 WO 2024021476 A1 WO2024021476 A1 WO 2024021476A1 CN 2022141519 W CN2022141519 W CN 2022141519W WO 2024021476 A1 WO2024021476 A1 WO 2024021476A1
Authority
WO
WIPO (PCT)
Prior art keywords
message
message data
data
piece
identification information
Prior art date
Application number
PCT/CN2022/141519
Other languages
English (en)
French (fr)
Inventor
朱志鹏
郑云
黎海育
杨傲
Original Assignee
天翼云科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 天翼云科技有限公司 filed Critical 天翼云科技有限公司
Publication of WO2024021476A1 publication Critical patent/WO2024021476A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04—Protocols for data compression, e.g. ROHC
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04B—TRANSMISSION
    • H04B7/00—Radio transmission systems, i.e. using radiation field
    • H04B7/14—Relay systems
    • H04B7/15—Active relay systems
    • H04B7/185—Space-based or airborne stations; Stations for satellite systems
    • H04B7/18502—Airborne stations
    • H04B7/18506—Communications with or from aircraft, i.e. aeronautical mobile service
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]

Definitions

  • the present application relates to the field of telecommunications technology, and in particular, to a data processing method, device, electronic equipment and storage medium.
  • Hierarchical Data Format Format is a set of file formats designed to store and organize large amounts of data. Since the 1990s, the HDF format has become a standard format for storing and publishing Earth Observation System (EOS) data, and the current version is HDF5. For this historical reason, the domestic and foreign aviation industry generally adopts HDF5 as the storage format for aviation messages. Since the cloud-native multi-mode database can provide a variety of data models such as wide tables, files, and searches, support millisecond-level online data processing, and low-cost storage and analysis of massive data, currently, HDF5 aviation messages are mainly processed through the cloud-native multi-mode database. Data is stored.
  • EOS Earth Observation System
  • the HDF5 aviation message data is converted into binary data through the writing module of the cloud-native multi-mode database, written into a queue, and then submitted in batches.
  • the storage module saves and manages the received data.
  • the performance of the storage module is affected by machine performance and can easily become a bottleneck in the data import process. Data import based on the above method is slow. low efficiency.
  • Embodiments of the present application provide a data processing method, device, electronic device, and storage medium to improve data processing efficiency.
  • the format of the message data is hierarchical data format HDF5, and the identification information is used to identify the sender and sending time of the corresponding message data;
  • each piece of said message data is divided into multiple message sets.
  • the sender of the message data contained in each message set is the same and the sending time belongs to the same time. interval;
  • the message data in the multiple message sets are mapped respectively, the message records of each message set are obtained, and each of the message records is stored in the cloud native multi-mode database. middle.
  • the message data is divided into multiple message sets according to the sender and sending time in advance.
  • Preset mapping rules compress the message set into one message record, which can greatly reduce the storage space required for the message.
  • the message record can be directly stored in the cloud-native multi-mode database, which can effectively improve data processing efficiency.
  • the message data in the multiple message sets are respectively compressed based on the preset mapping rules to obtain respective message records of each message set, including:
  • the merged messages are mapped based on preset mapping rules to obtain message records corresponding to the message set.
  • the method before dividing each piece of message data into multiple message sets based on the respective identification information of each piece of message data, the method further includes:
  • a plurality of pieces of the message data are analyzed respectively to obtain message data in a standard format for each piece of message data.
  • the standard format is a compressed format that complies with the hypertext transfer protocol http.
  • mapping the merged messages based on preset mapping rules, and obtaining message records corresponding to the message set includes:
  • the bytes contained in the merged message are mapped to obtain the message record in binary format corresponding to the message set.
  • mapping merged messages based on preset mapping rules can further reduce the storage space required for merged messages.
  • the mapped message records can be directly stored in the cloud native multi-mode database, which improves the efficiency of the merged messages. Data processing efficiency.
  • each piece of the message data is divided into multiple message sets, including:
  • each piece of the message data Based on the identification information of each piece of the message data, determine the sender of each piece of the message data, and based on the sender of each piece of the message data, divide each piece of the message data into a plurality of intermediate Set, the sender of message data in the same intermediate set is the same;
  • For any intermediate set determine the sending time of each piece of message data based on the identification information of each piece of message data contained in any one of the intermediate sets, and based on the sending time of each piece of said message data, Each piece of message data is divided into multiple message sets, and the sending time of the message data in the same message set belongs to the same preset time interval.
  • the message data includes information used to characterize the current operating status of the sender of the message data.
  • An acquisition unit is used to acquire multiple pieces of message data and identification information of each piece of the message data.
  • the format of the message data is hierarchical data format HDF5, and the identification information is used to identify the sending of the corresponding message data.
  • a dividing unit configured to divide each piece of said message data into multiple message sets based on the identification information of each piece of said message data.
  • the sender of the message data contained in each message set is the same, and The sending time belongs to the same time interval;
  • a mapping unit configured to map the message data in the plurality of message sets respectively based on preset mapping rules, obtain respective message records of each message set, and store each of the message records in the cloud In native multi-mode database.
  • mapping unit is specifically used for:
  • the merged messages are mapped based on preset mapping rules to obtain message records corresponding to the message set.
  • the device also includes an analysis unit, used for;
  • a plurality of pieces of the message data are analyzed respectively to obtain message data in a standard format for each piece of message data.
  • the standard format is a compressed format that complies with the hypertext transfer protocol http.
  • mapping unit is specifically used for:
  • the bytes contained in the merged message are mapped to obtain the message record in binary format corresponding to the message set.
  • the dividing unit is specifically used for:
  • any intermediate set determine the sending time of the message data based on the identification information of the message data contained in any intermediate set, and based on the sending time, combine the messages belonging to the same time interval
  • the data is divided into the same message set to obtain multiple message sets.
  • the message data includes information used to characterize the current operating status of the sender of the message data.
  • An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program.
  • the processor is caused to execute any of the above. Steps of the data processing method.
  • Embodiments of the present application provide a computer-readable storage medium, which includes a computer program.
  • the computer program When the computer program is run on an electronic device, the computer program is used to cause the electronic device to execute any one of the above data processing methods. step.
  • Embodiments of the present application provide a computer program product.
  • the computer program product includes a computer program.
  • the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, When executing a program, the processor executes the computer program, causing the electronic device to execute the steps of any one of the above data processing methods.
  • Figure 1 is an optional schematic diagram of an application scenario in the embodiment of the present application.
  • Figure 2 is an implementation flow chart of a data processing method in the embodiment of the present application.
  • Figure 3 is a schematic flow chart of a data partitioning method in an embodiment of the present application.
  • Figure 4 is a logical schematic diagram of a data processing method in an embodiment of the present application.
  • Figure 5 is a schematic structural diagram of a data processing device in an embodiment of the present application.
  • Figure 6 is a schematic structural diagram of a hardware structure of an electronic device applying an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a hardware structure of another electronic device applying an embodiment of the present application.
  • HDF5 uses group + data set organization file format, which is naturally highly compressible. After calculation, 50GB of message data can be compressed into 2GB, achieving a compression rate of 25 times. Data in HDF5 format must be converted into a compression format (standard format) that complies with http before it can be written into a cloud-native multi-mode database. This means that data in HDF5 format must be decompressed into single or batch standard format data before it can be submitted to the cloud-native multi-mode database. model database.
  • Cloud-native multi-mode database Provides multiple data models such as wide tables, files, and searches, and supports millisecond-level online data processing, low-cost storage and analysis of massive data. Provides unified SQL capabilities for real-time query, retrieval and analysis, and built-in analysis and calculation engine to meet computing needs. Meet the needs of multiple business scenarios such as aviation, Internet of Things, Internet of Vehicles, and Industrial Internet.
  • the cloud native multi-mode database can be divided into writing instances and storage bases.
  • the writing instance is responsible for converting imported data into binary data and writing it into the queue. Then submit them to the storage base in batches.
  • the storage base is responsible for the actual data storage and management. Among them, write instances can be infinitely expanded horizontally, and storage bases are related to machine performance. Therefore, the bottleneck in importing HDF5 data into cloud-native multi-mode databases lies in the performance of the storage base.
  • the application scenario diagram includes two terminal devices 110 and a server 120.
  • the terminal device 110 includes but is not limited to mobile phones, tablet computers, notebook computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; the terminal device may be installed with data Process related clients, which can be software (such as browsers, data import software, etc.), web pages, small programs, etc.
  • the server 120 is a background server corresponding to the software, web pages, small programs, etc. , or a server specifically used for data processing, which is not specifically limited in this application.
  • Server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, Middleware services, domain name services, security services, content distribution network (Content Delivery Network (CDN), as well as cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.
  • cloud databases cloud computing, cloud functions, cloud storage, network services, cloud communications, Middleware services, domain name services, security services, content distribution network (Content Delivery Network (CDN), as well as cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.
  • the data processing method in the embodiment of the present application can be executed by an electronic device, and the electronic device can be the server 120 or the terminal device 110. That is, the method can be executed by the server 120 or the terminal device 110 alone, or by The server 120 and the terminal device 110 execute together.
  • the terminal device 110 obtains multiple pieces of message data and the identification information of each piece of message data, and sends the message data and the corresponding identification information to the server 120; the server 120 based on each piece of message data.
  • the identification information of the message data divides each message data into multiple message sets; based on the preset mapping rules, the message data in the multiple message sets are mapped respectively, and the respective message sets of each message set are obtained.
  • Message records and storage store each message record in a cloud-native multi-mode database.
  • the terminal device 110 and the server 120 can communicate through a communication network.
  • the communication network is a wired network or a wireless network.
  • multiple servers can be composed into a blockchain, and the server is a node on the blockchain; as in the data processing method disclosed in the embodiment of this application, where The aviation messages involved can be saved on the blockchain.
  • embodiments of the present application can be applied to various scenarios, including not only data processing scenarios, but also including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.
  • FIG. 2 is an implementation flow chart of a data processing method provided by an embodiment of the present application.
  • the specific implementation process of the method includes the following steps S21-S23:
  • the format of the message data is hierarchical data format HDF5, and the identification information is used to identify the sender and sending time of the corresponding message data.
  • the message data can be an aviation message, and the sender of the message data can be various equipment included in the aircraft, such as aircraft switches, engines, and other equipment.
  • the message data can also be vehicle-mounted messages, and the sender of the message data can be
  • the sender can be various equipment included in the car, such as the car transmission system, engine and other equipment, which is not specifically limited in this application.
  • the message data is mainly an aviation message.
  • the following is a detailed description of 100 aviation messages related to aircraft switches in the HDF5 format:
  • gear means the switch
  • flight_no:AC0073 means the aircraft number is AC0073
  • gear_no:no29 means the switch number is 29, 1645050457000-1645050457099 means the sending time of the message, 1, 0, 1, 1...1 means the message contains value.
  • the identification information is: gear flight_no:AC0073, gear_no:no29 1645050457000, among which, gear flight_no:AC0073, gear_no:no29 are used to identify the sender of the aviation message, and 1645050457000 is used to identify the sending time of the aviation message.
  • the message data includes information used to characterize the current operating status of the sender of the message data.
  • the aircraft contains a large number of devices. Each device records its own operating status every 1 second or every 2 seconds and sends it out in the form of aviation messages. Take the message 1 listed above as an example. "1" is the message. The value contained in the text is the information contained in the aviation message used to characterize the current operating status of the aircraft switch.
  • the sender of the message data contained in each message set is the same, and the sending time belongs to the same time interval.
  • the messages 1 to 100 listed above are messages with the same sender.
  • the messages with the same sender can be further divided according to the sending time. For example, the sending time can be divided into hours.
  • the message data between 0 o'clock and 1 o'clock is divided into a message set, and the message data sent between 1 o'clock and 2 o'clock is divided into a message set. We will not list them one by one here.
  • the message data in a message collection is mapped into a message record and stored directly in the cloud-native multi-mode database.
  • the message data is divided into multiple message sets according to the sender and sending time in advance.
  • the message set is compressed into a message record, which can greatly reduce the storage space required by the message.
  • the message record can be directly stored in the cloud-native multi-mode database, which can effectively improve the data processing efficiency.
  • step S211 may also be implemented
  • the standard format is a compressed format that conforms to the Hypertext Transfer Protocol http. Since data in HDF5 format must be converted into a standard format before it can be written to the database, the message data needs to be parsed and converted into message data that conforms to the http compression format.
  • step S22 can be implemented as the following steps S31-S34:
  • S34 Based on the sending time of each message data, divide each message data into multiple message sets. The sending times of message data in the same message set belong to the same preset time interval.
  • the message data of the same sender is divided into the same intermediate set, and then for each intermediate set, based on the identification information of the message data, the messages whose sending time belongs to the same time interval are divided into Divide the message data into the same message set to obtain multiple message sets. For example, divide the message sets with the same sender into the same intermediate set to obtain intermediate set 1 and intermediate set 2, then divide the message data whose sending time belongs to the same time interval into the same message set, and divide the intermediate set 1 For message set 1 and message set 2, the intermediate set 2 is divided into message set 3 and message set 4, and finally four message sets are obtained.
  • step S23 can be implemented as the following steps 1-2:
  • Step 1 For any message set, merge each message based on the same field of the identification information of each message data in any message set to obtain a merged message;
  • Step 2 Map the merged messages based on the preset mapping rules to obtain the message records corresponding to the message set.
  • the senders of the message data in the same message set are the same, and the same fields can be fields used to identify the senders of the message data.
  • the same fields are: gear flight_no:AC0073, gear_no:no29
  • the merged message obtained after merging is: gear flight_no:AC0073, gear_no:no29 1645050457000, 1645050457001...1645050457099 1, 0, 1, 1...1.
  • mapping is performed according to the preset mapping rules to obtain the message records corresponding to the message set.
  • step 2 can be implemented as:
  • the bytes contained in the merged message are mapped to obtain the message record in binary format corresponding to the message set.
  • the merged packets can be mapped through byte mapping.
  • the message records obtained by merging the messages for mapping are: 001 002003004005 00600061...00699 1, 0, 1, 1...1.
  • Such message records in binary format can be directly written into the cloud native multi-mode database.
  • mapping the merged messages based on the preset mapping rules can further reduce the storage space required for the merged messages.
  • the mapped message records can be directly stored in the cloud-native multi-mode database. , improving data processing efficiency.
  • the cloud native multi-mode database can be divided into writing instances and storage bases.
  • the writing instance is responsible for converting imported data into binary data and then writing it to the queue. Then submit them to the storage base in batches.
  • the storage base is responsible for actual data storage and management. For better compression rate, the storage base can compress one hour of data into one record, and then overwrite and write it to the storage.
  • Write instances can be scaled horizontally and infinitely, and the storage base and machine performance are related. Therefore, the bottleneck of HDF5 file import lies in the performance of the storage base.
  • the data processing method in this application can be applied to import HDF5 aviation messages into a cloud-native multi-mode database.
  • Figure 4 is a logical schematic diagram of a data processing method in an embodiment of this application, in which the aviation HDF5 file refers to File containing HDF5 aeronautical messages.
  • SDK cloud-native multi-mode data HDF5 dedicated software development kit
  • the writing instance receives a one-hour message collection, it directly compresses a message collection into a binary record (message record).
  • the storage base receives this binary record, it directly saves it without any compression operation. It can effectively improve data import efficiency.
  • the clear structure of aviation HDF5 files is used, and the HDF5 file data is read in batches through a customized SDK, and is divided and submitted by the hour.
  • the data is creatively submitted in batches per hour.
  • one hour of data is converted into a record through a cloud-native multi-mode database writing instance, and written to the storage base. Resolved bottlenecks and compression issues during data import.
  • the cloud-native multi-mode database can provide millions of writes per second under the 3-host cluster specification.
  • a universal interface is used to read a 2GB HDF5 file containing approximately 960 million pieces of aerospace engineering value data. It takes 17 hours to fully import the cloud-native multi-mode database. If 100TB of HDF5 files are generated in a year, it will take at least 45 days to import the data. This import speed is too slow.
  • the time it takes to import a 2GB HDF5 file into a cloud-native multi-mode database is directly reduced from 17 hours to 0.5 hours, which increases the import speed by 34 times. The time for importing 100T data Reduced from 45 days to 1.5 days.
  • FIG. 5 it is a schematic structural diagram of a data processing device 500, which may include:
  • the acquisition unit 501 is used to acquire multiple pieces of message data and identification information of each piece of message data.
  • the format of the message data is the hierarchical data format HDF5, and the identification information is used to identify the sender and sending time of the corresponding message data;
  • the dividing unit 502 is configured to divide each piece of message data into multiple message sets based on the identification information of each piece of message data.
  • the sender of the message data contained in each message set is the same, and the sending time belongs to the same time interval;
  • the mapping unit 503 is used to map message data in multiple message sets based on preset mapping rules, obtain message records of each message set, and store each message record in the cloud native multi-mode in the database.
  • mapping unit 503 is specifically used for:
  • the merged messages are mapped based on the preset mapping rules to obtain message records corresponding to the message set.
  • the device also includes a parsing unit 504, used for;
  • the standard format is a compressed format that conforms to the Hypertext Transfer Protocol http.
  • mapping unit 503 is specifically used for:
  • the bytes contained in the merged message are mapped to obtain the message record in binary format corresponding to the message set.
  • the dividing unit 502 is specifically used for:
  • the sender of each piece of message data is determined, and based on the sender of each piece of message data, each piece of message data with the same sender is divided into the same intermediate set to obtain multiple an intermediate set;
  • any intermediate set determine the sending time of the message data based on the identification information of the message data contained in any intermediate set, and classify the message data belonging to the same time interval into the same message set based on the sending time. , obtain multiple message sets.
  • the message data includes information used to characterize the current operating status of the sender of the message data.
  • each of the above parts is divided into modules (or units) according to their functions and described separately.
  • the functions of each module (or unit) can be implemented in the same or multiple software or hardware.
  • the embodiment of the present application also provides an electronic device.
  • the electronic device may be a server, such as server 120 as shown in FIG. 1 .
  • the structure of the electronic device may be as shown in FIG. 6 , including a memory 601, a communication module 603, and one or more processors 602.
  • Memory 601 is used to store computer programs executed by processor 602.
  • the memory 601 may mainly include a program storage area and a data storage area.
  • the program storage area may store the operating system and programs required to run instant messaging functions.
  • the storage data area may store various instant messaging information and operating instruction sets.
  • the memory 601 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 601 may also be a non-volatile memory (non-volatile memory). memory), such as read-only memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or the memory 601 is capable of carrying or storing instructions or data. Any other medium in the structural form of a desired computer program and capable of being accessed by a computer, without limitation.
  • the memory 601 may be a combination of the above memories.
  • Processor 602 may include one or more central processing units (central processing units). processing unit (CPU) or digital processing unit, etc.
  • CPU central processing unit
  • the processor 602 is used to implement the above data processing method when calling the computer program stored in the memory 601.
  • the communication module 603 is used to communicate with terminal devices and other servers.
  • connection medium between the above-mentioned memory 601, communication module 603 and processor 602 is not limited in the embodiment of the present application.
  • the memory 601 and the processor 602 are connected through a bus 604.
  • the bus 604 is depicted as a thick line in Figure 6.
  • the connection between other components is only a schematic explanation and does not To be limited.
  • the bus 604 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, only one thick line is used in Figure 6, but it does not describe only one bus or one type of bus.
  • a computer storage medium is stored in the memory 601, and computer executable instructions are stored in the computer storage medium.
  • the computer executable instructions are used to implement the data processing method of the embodiment of the present application.
  • the processor 602 is used to execute the above data processing method, as shown in Figure 2.
  • the electronic device may also be other electronic devices, such as the terminal device 110 shown in FIG. 1 .
  • the structure of the electronic device can be shown in Figure 7, including: communication component 710, memory 720, display unit 730, camera 740, sensor 750, audio circuit 760, Bluetooth module 770, processor 780 and other components.
  • Communication component 710 is used to communicate with the server.
  • a circuit wireless fidelity (Wireless Fidelity, WiFi) module may be included.
  • the WiFi module belongs to short-distance wireless transmission technology, and the electronic device can help users send and receive information through the WiFi module.
  • Memory 720 may be used to store software programs and data.
  • the processor 780 executes software programs or data stored in the memory 720 to perform various functions and data processing of the terminal device 110 .
  • Memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.
  • the memory 720 stores an operating system that enables the terminal device 110 to run. In this application, the memory 720 can store the operating system and various application programs, and can also store computer programs that execute the data processing methods in the embodiments of this application.
  • the display unit 730 may also be used to display information input by the user or information provided to the user and a graphical user interface (graphical user interface) of various menus of the terminal device 110 user interface, GUI).
  • the display unit 730 may include a display screen 732 provided on the front of the terminal device 110 .
  • the display screen 732 can be configured in the form of a liquid crystal display, a light-emitting diode, etc.
  • the display unit 730 may be used to display the data processing user interface, etc. in the embodiment of the present application.
  • the display unit 730 can also be used to receive input numeric or character information and generate signal input related to user settings and function control of the terminal device 110.
  • the display unit 730 can include a touch screen 731 disposed on the front of the terminal device 110, which can collect The user's touch operations on or near it, such as clicking a button, dragging a scroll box, etc.
  • the touch screen 731 can cover the display screen 732, or the touch screen 731 and the display screen 732 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen.
  • the display unit 730 can display application programs and corresponding operation steps.
  • the camera 740 can be used to capture still images, and the user can post comments on the images captured by the camera 740 through the application. There may be one camera 740 or multiple cameras.
  • the object passes through the lens to produce an optical image that is projected onto the photosensitive element.
  • the photosensitive element can be a charge coupled device (charge coupled device) device, CCD) or complementary metal oxide semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor.
  • CCD charge coupled device
  • CMOS complementary metal oxide semiconductor
  • the terminal device may also include at least one sensor 750, such as an acceleration sensor 751, a distance sensor 752, a fingerprint sensor 753, and a temperature sensor 754.
  • the terminal device can also be equipped with other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, motion sensor, etc.
  • the audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the terminal device 110.
  • the audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, and the speaker 761 converts the electrical signal into a sound signal and outputs it.
  • the terminal device 110 may also be configured with a volume button for adjusting the volume of the sound signal.
  • the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, and then outputs the audio data to the communication component 710 to be sent to, for example, another terminal device 110, or the audio data is Output to memory 720 for further processing.
  • the Bluetooth module 770 is used to interact with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol.
  • the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 770 to perform data exchange.
  • a wearable electronic device such as a smart watch
  • the processor 780 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes the functions of the terminal device by running or executing software programs stored in the memory 720 and calling data stored in the memory 720. Various functions and processing data.
  • the processor 780 may include one or more processing units; the processor 780 may also integrate an application processor and a baseband processor, where the application processor mainly processes the operating system, user interface, application programs, etc., and the baseband processor The processor primarily handles wireless communications. It can be understood that the above-mentioned baseband processor may not be integrated into the processor 780 .
  • the processor 780 in this application can run an operating system, application programs, user interface display and touch response, as well as the data processing method in the embodiment of this application.
  • the processor 780 is coupled with the display unit 730.
  • various aspects of the data processing method provided by this application can also be implemented in the form of a program product, which includes a computer program.
  • the program product is run on an electronic device, the computer program is used to make the electronic device
  • the device performs the steps in the data processing method according to various exemplary embodiments of the present application described above in this specification.
  • the electronic device may perform the steps as shown in Figure 2.
  • the Program Product may take the form of one or more readable media in any combination.
  • the readable medium may be a readable signal medium or a readable storage medium.
  • the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • the program product of embodiments of the present application may take the form of a portable compact disk read-only memory (CD-ROM) and include a computer program, and may be run on an electronic device.
  • CD-ROM portable compact disk read-only memory
  • the program product of the present application is not limited thereto.
  • a readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with a command execution system, apparatus or device.
  • the readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying a readable computer program therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a readable signal medium may also be any readable medium other than a readable storage medium that can send, propagate, or transport a program for use by or in connection with a command execution system, apparatus, or device.
  • Computer program embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.
  • Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., as well as conventional procedural programming. Language—such as "C” or a similar programming language.
  • the computer program may execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server execute on.
  • the remote electronic devices may be connected to the user electronic device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., using an Internet service provider). connected via the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider e.g., using an Internet service provider
  • embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) embodying a computer-usable computer program therein.
  • computer-usable storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc.
  • These computer program commands may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a particular manner, such that the commands stored in the computer-readable memory produce an article of manufacture including command means, the commands
  • the device implements the functions specified in a process or processes in the flowchart and/or in a block or blocks in the block diagram.
  • These computer program commands may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device.
  • a command provides steps for implementing a function specified in a process or processes in a flowchart and/or in a block or blocks in a block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Physics & Mathematics (AREA)
  • Astronomy & Astrophysics (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请涉及数据库技术领域，尤其涉及一种数据处理方法、装置、电子设备和存储介质，用以提高数据处理效率。其中，方法包括：获取多条报文数据以及各报文数据的标识信息，报文数据的格式为层级数据格式HDF5；基于各报文数据的标识信息，将各报文数据划分为多个报文集合；基于预设映射规则，分别对多个报文集合中的报文数据进行映射，获得各报文集合的报文记录，并将各报文记录存储在云原生多模数据库中。由于本申请通过预先将HDF5报文数据按照发送方和发送时间划分为多个报文集合，并基于预设映射规则，将报文集合压缩为一条报文记录，直接存储在云原生多模数据库即可，能够有效提高数据处理效率。

Description

一种数据处理方法、装置、电子设备和存储介质 技术领域
本申请涉及电信技术领域，尤其涉及一种数据处理方法、装置、电子设备和存储介质。
 背景技术
层级数据格式（Hierarchical Data Format，HDF）是设计用来存储和组织大量数据的一组文件格式。从二十世纪九十年代开始，HDF格式成为存储和发布地球观测系统（Earth Observation System，EOS）数据的标准格式，当前版本是HDF5。基于这个历史原因，国内外航空业普遍采用HDF5作为航空报文的存储格式。由于云原生多模数据库能够提供宽表、文件、搜索等多种数据模型，支持毫秒级在线数据处理、海量数据低成本存储和分析，目前，主要是通过云原生多模数据库对HDF5航空报文数据进行存储。
 技术问题
相关技术中，在将HDF5航空报文数据导入云原生多模数据库的过程中，通过云原生多模数据库的写入模块将HDF5航空报文数据转化为二进制数据，并写入队列，然后批量提交给存储模块，存储模块对接收的数据进行保存和管理。但是，由于HDF5航空报文数据量大，存储模块的性能受机器性能影响，容易成为数据导入过程的瓶颈，基于上述方式进行数据导入，导入速度缓慢。效率低。
 技术解决方案
本申请实施例提供一种数据处理方法、装置、电子设备和存储介质，用以提高数据处理效率。
本申请实施例提供的一种数据处理方法，包括：
获取多条报文数据以及各条所述报文数据的标识信息，所述报文数据的格式为层级数据格式HDF5，所述标识信息用于标识对应的报文数据的发送方和发送时间；
基于各条所述报文数据的标识信息，将各条所述报文数据划分为多个报文集合，每个报文集合中包含的报文数据的发送方相同，且发送时间属于同一时间区间；
基于预设映射规则，分别对所述多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并存储将各个所述报文记录 存储在云原生多模数据库中。
在本申请中，基于HDF5报文数据结构清晰明了的特性，预先将报文数据按照发送方和发送时间划分为多个报文集合，在将报文数据导入云原生多模数据库过程中，基于预设映射规则，将报文集合压缩为一条报文记录，能够大幅减少报文所需占用的存储空间，最后直接将报文记录存储在云原生多模数据库即可，能够有效提高数据处理效率。
在一种可选的实施方式中，所述基于预设映射规则，分别对所述多个报文集合中的报文数据进行压缩，获得各个报文集合各自的报文记录，包括：
针对任意一个报文集合，基于所述任意一个报文集合中的各条报文数据的标识信息的相同字段，对所述各条报文进行合并得到合并报文；
基于预设映射规则对所述合并报文进行映射，获得所述报文集合对应的报文记录。
基于上述方式，通过按照报文数据的相同字段合并报文数据，能够减少报文数据中的重复数据，减少报文数据所需占用的存储空间，进而在将报文数据导入云原生多模数据库过程中，提高数据处理效率。
在一种可选的实施方式中，在所述基于各条报文数据各自的标识信息，将所述各条报文数据划分为多个报文集合之前，还包括；
分别对多条所述报文数据进行解析，获得各条标准格式的报文数据，所述标准格式为符合超文本传输协议http的压缩格式。
在一种可选的实施方式中，所述基于预设映射规则对所述合并报文进行映射，获得所述报文集合对应的报文记录包括：
基于预设映射规则，对所述合并报文包含的字节进行映射，获得所述报文集合对应的二进制格式的所述报文记录。
基于上述方式，基于预设映射规则，对合并报文进行映射，能够进一步减少合并报文所需占用的存储空间，同时映射得到的报文记录可以直接存储在云原生多模数据库中，提高了数据处理效率。
在一种可选的实施方式中，所述基于各条所述报文数据的标识信息，将各条所述报文数据划分为多个报文集合，包括：
基于各条所述报文数据的标识信息，确定各条所述报文数据的发送方，并基于各条所述报文数据的发送方，将各条所述报文数据划分为多个中间集合，同一中间集合中的报文数据的发送方相同；
针对任意一个中间集合，基于所述任意一个中间集合中包含的各条报文数据的标识信息，确定各条所述报文数据的发送时间，并基于各条所述报文数据的发送时间，将各条所述报文数据划分为多个报文集合，同一报文集合中的报文数据的发送时间属于同一预设时间区间。
基于上述方式，通过按照报文数据的标识信息将报文数据划分为多个报文集合，在云原生多模数据库存储报文数据时，无需进行分组操作，节省云原生多模数据库的计算资源，提高数据处理效率。
在一种可选的实施方式中，所述报文数据中包含用于表征所述报文数据的发送方当前的运行状态的信息。
本申请实施例提供的一种数据处理装置，包括：
获取单元，用于获取多条报文数据以及各条所述报文数据的标识信息，所述报文数据的格式为层级数据格式HDF5，所述标识信息用于标识对应的报文数据的发送方和发送时间；
划分单元，用于基于各条所述报文数据的标识信息，将各条所述报文数据划分为多个报文集合，每个报文集合中包含的报文数据的发送方相同，且发送时间属于同一时间区间；
映射单元，用于基于预设映射规则，分别对所述多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并将各个所述报文记录存储在云原生多模数据库中。
可选的，所述映射单元具体用于：
针对任意一个报文集合，基于所述任意一个报文集合中的各条报文数据的标识信息的相同字段，对所述各条报文进行合并得到合并报文；
基于预设映射规则对所述合并报文进行映射，获得所述报文集合对应的报文记录。
可选的，所述装置还包括解析单元，用于；
分别对多条所述报文数据进行解析，获得各条标准格式的报文数据，所述标准格式为符合超文本传输协议http的压缩格式。
可选的，所述映射单元具体用于：
基于预设映射规则，对所述合并报文包含的字节进行映射，获得所述报文集合对应的二进制格式的所述报文记录。
可选的，所述划分单元具体用于：
基于各条所述报文数据的标识信息，确定各条所述报文数据的发送方，并基于各条所述报文数据的发送方，将发送方相同的各条所述报文数据划分至同一中间集合中，获得多个中间集合；
针对任意一个中间集合，基于所述任意一个中间集合中包含的报文数据的标识信息，确定所述报文数据的发送时间，并基于所述发送时间，将属于同一时间区间的所述报文数据划分至同一报文集合中，获得多个所述报文集合。
可选的，所述报文数据中包含用于表征所述报文数据的发送方当前的运行状态的信息。
本申请实施例提供的一种电子设备，包括处理器和存储器，其中，所述存储器存储有计算机程序，当所述计算机程序被所述处理器执行时，使得所述处理器执行上述任意一种数据处理方法的步骤。
本申请实施例提供一种计算机可读存储介质，其包括计算机程序，当所述计算机程序在电子设备上运行时，所述计算机程序用于使所述电子设备执行上述任意一种数据处理方法的步骤。
本申请实施例提供一种计算机程序产品，所述计算机程序产品包括计算机程序，所述计算机程序存储在计算机可读存储介质中；当电子设备的处理器从计算机可读存储介质读取所述计算机程序时，所述处理器执行所述计算机程序，使得所述电子设备执行上述任意一种数据处理方法的步骤。
本申请的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的目的和其他优点可通过在所写的说明书、权利要求书、以及附图中所特别指出的结构来实现和获得。
 附图说明
此处所说明的附图用来提供对本申请的进一步理解，构成本申请的一部分，本申请的示意性实施例及其说明用于解释本申请，并不构成对本申请的不当限定。在附图中：
图1为本申请实施例中的一种应用场景的一个可选的示意图；
图2为本申请实施例中的一种数据处理方法的实施流程图；
图3为本申请实施例中的一种数据划分方法的流程示意图；
图4为本申请实施例中的一种数据处理方法的逻辑示意图；
图5为本申请实施例中的一种数据处理装置的结构示意图；
图6为应用本申请实施例的一种电子设备的一个硬件组成结构示意图；
图7为应用本申请实施例的另一种电子设备的一个硬件组成结构示意图。
 本发明的实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请技术方案的一部分实施例，而不是全部的实施例。基于本申请文件中记载的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请技术方案保护的范围。
下面对本申请实施例中涉及的部分概念进行介绍。
航空报文：经历了二进制到结构化（HDF5）的转化，目前正转向知识化和智能化趋势。所谓知识化和智能化趋势即是航空报文大量数据上云，通过云上对航空报文进行统一处理、分析以及训练，将结果反馈至机务人员，机务人员根据结果提升飞机故障检查能力以及服务水平，形成正向循环。
HDF5：采用群组+数据集组织文件格式，天然具有高压缩性。经过测算，50GB的报文数据可以压缩为2GB，能够达到25倍的压缩率。HDF5格式的数据必须经过转换成符合http的压缩格式（标准格式）才能写入云原生多模数据库，这里就导致HDF5格式的数据必须解压成单笔或是批量标准格式数据才能提交至云原生多模数据库。
云原生多模数据库：提供宽表、文件、搜索等多种数据模型，支持毫秒级在线数据处理、海量数据低成本存储和分析。提供统一SQL能力，进行实时查询、检索和分析，内置分析计算引擎满足计算需求。满足航空、物联网、车联网、工业互联网等多业务场景需求。云原生多模数据库内部可分为写入实例和存储底座，写入实例负责将导入数据转化为二进制数据，并写入队列。然后批量提交给存储底座。存储底座负责实际的数据保存和管理。其中，写入实例可以横向无限扩展，存储底座和机器性能相关。因此HDF5数据导入云原生多模数据库的瓶颈在于存储底座的性能。
以下结合说明书附图对本申请的优选实施例进行说明，应当理解，此处所描述的优选实施例仅用于说明和解释本申请，并不用于限定本申请，并且在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
如图1所示，其为本申请实施例的应用场景示意图。该应用场景图中包括两个终端设备110和一个服务器120。
在本申请实施例中，终端设备110包括但不限于手机、平板电脑、笔记本电脑、台式电脑、电子书阅读器、智能语音交互设备、智能家电、车载终端等设备；终端设备上可以安装有数据处理相关的客户端，该客户端可以是软件（例如浏览器、数据导入软件等），也可以是网页、小程序等，服务器120则是与软件或是网页、小程序等相对应的后台服务器，或者是专门用于进行数据处理的服务器，本申请不做具体限定。服务器120可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络（Content Delivery Network，CDN）、以及大数据和人工智能平台等基础云计算服务的云服务器。
需要说明的是，本申请实施例中的数据处理方法可以由电子设备执行，该电子设备可以为服务器120或者终端设备110，即，该方法可以由服务器120或者终端设备110单独执行，也可以由服务器120和终端设备110共同执行。比如由服务器120和终端设备110共同执行时，终端设备110获取多条报文数据以及各条报文数据的标识信息，并将报文数据和对应的标识信息发送给服务器120；服务器120基于各条报文数据的标识信息，将各条报文数据划分为多个报文集合；基于预设映射规则，分别对多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并存储将各个报文记录存储在云原生多模数据库中。
在一种可选的实施方式中，终端设备110与服务器120之间可以通过通信网络进行通信。
在一种可选的实施方式中，通信网络是有线网络或无线网络。
需要说明的是，图1所示只是举例说明，实际上终端设备和服务器的数量不受限制，在本申请实施例中不做具体限定。
本申请实施例中，当服务器的数量为多个时，多个服务器可组成为一区块链，而服务器为区块链上的节点；如本申请实施例所公开的数据处理方法，其中所涉及的航空报文可保存于区块链上。
此外，本申请实施例可应用于各种场景，不仅包括数据处理场景，还包括但不限于云技术、人工智能、智慧交通、辅助驾驶等场景。
下面结合上述描述的应用场景，参考附图来描述本申请示例性实施方式提供的数据处理方法，需要注意的是，上述应用场景仅是为了便于理解本申请的精神和原理而示出，本申请的实施方式在此方面不受任何限制。
参阅图2所示，为本申请实施例提供的一种数据处理方法的实施流程图，该方法的具体实施流程包括如下步骤S21-S23：
S21：获取多条报文数据以及各条报文数据的标识信息；
其中，报文数据的格式为层级数据格式HDF5，标识信息用于标识对应的报文数据的发送方和发送时间。报文数据可以是航空报文，则报文数据的发送方可以是飞机包含的各种设备，例如，飞机电门、发动机等设备，报文数据还可以是车载报文，则报文数据的发送方可以是汽车包含的各种设备，例如，汽车变速系统、发动机等设备，本申请在此不做具体限定。
在本申请实施例中，主要是以报文数据为航空报文为例进行说明的，下面以100条飞机电门相关的航空报文在HDF5格式下进行具体说明：
报文1：gear  flight_no:AC0073， gear_no:no29  1645050457000  1；
报文2：gear  flight_no:AC0073， gear_no:no29  1645050457001  0；
报文3：gear  flight_no:AC0073， gear_no:no29  1645050457002  1；
报文4：gear  flight_no:AC0073， gear_no:no29  1645050457003  1；
...
报文100： gear  flight_no:AC0073， gear_no:no29  1645050457099  1。
其中，gear表示电门，flight_no:AC0073表示飞机编号为AC0073，gear_no:no29表示电门编号为29，1645050457000-1645050457099表示报文的发送时间，1、0、1、1…1表示报文中包含的值。以报文1为例，标识信息为：gear  flight_no:AC0073， gear_no:no29  1645050457000，其中，gear  flight_no:AC0073， gear_no:no29用于标识航空报文的发送方，1645050457000用于标识航空报文的发送时间。
在一种可选的实施方式中，报文数据中包含用于表征报文数据的发送方当前的运行状态的信息。
具体地，飞机上包含大量设备，每个设备每1秒或每2秒都会记录自身的运行状态，并以航空报文的形式发送出去，以上述列举的报文1为例“1”为报文中包含的值，即为航空报文中包含的用于表征飞机电门当前的运行状态的信息。
S22：基于各条报文数据的标识信息，将各条报文数据划分为多个报文集合；
其中，每个报文集合中包含的报文数据的发送方相同，且发送时间属于同一时间区间。上述所列举的报文1-报文100为发送方相同的报文，可以根据报文数据的标识信息，将发送方相同的报文进一步按照发送时间划分，例如，按照小时分割，将发送时间在0点-1点的报文数据划分为一个报文集合，将发送时间在1点-2点的报文划分为一个报文集合，在此不进行一一列举。
S23：基于预设映射规则，分别对多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并存储将各个报文记录 存储在云原生多模数据库中。
具体地，将报文数据按照发送方和发送时间划分后，由于在云原生多模数据库中是将一个小时的数据压缩为一条记录，然后覆盖写入存储的，因此可以基于预设映射规则，将一个报文集合中的报文数据映射为一条报文记录，直接存储在云原生多模数据库中。
在本申请实施例中，基于HDF5报文数据结构清晰明了的特性，预先将报文数据按照发送方和发送时间划分为多个报文集合，在将报文数据导入云原生多模数据库过程中，基于预设映射规则，将报文集合压缩为一条报文记录，能够大幅减少报文所需占用的存储空间，最后直接将报文记录存储在云原生多模数据库即可，能够有效提高数据处理效率。
在一种可选的实施方式中，在步骤S22之前，还可实施步骤S211；
分别对多条报文数据进行解析，获得各条标准格式的报文数据。
其中，标准格式为符合超文本传输协议http的压缩格式。由于HDF5格式的数据必须转换为标准格式后才能写入数据库，因此需要对报文数据进行解析，转换为符合http压缩格式的报文数据。
在一种可选的实施方式中，如图3所示，步骤S22可以实施为以下步骤S31-S34：
S31：基于各条报文数据的标识信息，确定各条报文数据的发送方；
S32：基于各条报文数据的发送方，将各条报文数据划分为多个中间集合，同一中间集合中的报文数据的发送方相同；
S33：针对任意一个中间集合，基于任意一个中间集合中包含的各条报文数据的标识信息，确定各条报文数据的发送时间，
S34：基于各条报文数据的发送时间，将各条报文数据划分为多个报文集合，同一报文集合中的报文数据的发送时间属于同一预设时间区间。
具体地，首先，基于报文数据的标识信息将发送方相同的报文数据划分至同一中间集合中，然后对于每个中间集合，基于报文数据的标识信息将发送时间属于同一时间区间的报文数据划分至同一报文集合中，获得多个报文集合。例如，将发送方相同的报文集合划分至同一中间集合，获得中间集合1和中间集合2，然后将发送时间属于同一时间区间的报文数据划分至同一报文集合中，将中间集合1划分为报文集合1和报文集合2，将中间集合2划分为报文集合3和报文集合4，最终获得4个报文集合。
在本申请实施例中，通过按照报文数据的标识信息将报文数据划分为多个报文集合，在云原生多模数据库存储报文数据时，无需进行分组操作，节省云原生多模数据库的计算资源，提高数据处理效率。
在一种可选的实施方式中，步骤S23可以实施为以下步骤1-2：
步骤1：针对任意一个报文集合，基于任意一个报文集合中的各条报文数据的标识信息的相同字段，对各条报文进行合并得到合并报文；
步骤2：基于预设映射规则对合并报文进行映射，获得报文集合对应的报文记录。
具体地，同一报文集合中的报文数据的发送方相同，相同字段可以是用于标识报文数据的发送方的字段，仍以上述列举的报文1-100为例，相同字段为：gear flight_no:AC0073，gear_no:no29，合并后得到的合并报文为：gear flight_no:AC0073，gear_no:no29 1645050457000，1645050457001…1645050457099 1，0，1，1…1。最后按照预设映射规则进行映射，获得报文集合对应的报文记录。
在本申请实施例中，通过按照报文数据的相同字段合并报文数据，能够减少报文数据中的重复数据，减少报文数据所需占用的存储空间，进而在将报文数据导入云原生多模数据库过程中，提高数据处理效率。
在一种可选的实施方式中，步骤2可以实施为：
基于预设映射规则，对合并报文包含的字节进行映射，获得报文集合对应的二进制格式的报文记录。
具体地，可以通过字节映射的方式对合并报文进行映射，例如，预设映射规则为：gear=001；flight_no=002；AC0073=003；gear_no=004；no29=005；1645050457000=006，对合并报文进行映射获得的报文记录为：001 002003004005 00600061…00699 1，0，1，1…1，这样的二进制格式的报文记录，可以直接写入云原生多模数据库。
在本申请实施例中，基于预设映射规则，对合并报文进行映射，能够进一步减少合并报文所需占用的存储空间，同时映射得到的报文记录可以直接存储在云原生多模数据库中，提高了数据处理效率。
云原生多模数据库内部可分为写入实例和存储底座，写入实例负责将导入数据转化为二进制数据，然后写入队列。然后批量提交给存储底座。存储底座负责实际的数据保存和管理，存储底座为了更好的压缩率，可以将一个小时的数据压缩成一条记录，然后覆盖写入存储。写入实例可以横向无限扩展，存储底座和机器性能相关。因此HDF5文件导入的瓶颈在于存储底座的性能。
本申请中的数据处理方法可以应用于将HDF5航空报文导入云原生多模数据库中，参阅图4，其为本申请实施例中的一种数据处理方法的逻辑示意图，其中，航空HDF5文件指包含HDF5航空报文的文件。首先，云原生多模数据HDF5专用软件开发工具包（以下简称SDK），负责读取航空HDF5文件，将航空HDF5文件解析成标准格式，然后将标准格式的航空报文按照发送方划分，并按照小时分割，获得报文集合，将多个报文集合批量提交至云原生多模数据库写入实例（以下简称写入实例）。写入实例接收到一个小时的报文集合后，直接将一个报文集合压缩为一笔二进制记录（报文记录），存储底座接收到这一笔二进制记录后，不用经过压缩操作，直接保存，能够有效提高数据导入效率。
在本申请实施例中，利用航空HDF5文件结构清晰明了的特性，通过定制化SDK，批量读取HDF5文件数据，并按小时切分提交，创造性按照一小时批量提交数据。然后通过云原生多模数据库写入实例将一小时数据转化为一笔记录，写入存储底座。解决了数据导入过程中的瓶颈和压缩问题。
另外，在进行数据导入之前，首先下载SDK，开通或安装云原生多模数据库，然后，设置SDK相关配置，保证能够访问云原生多模数据库，最后，设置SDK读取HDF5文件路径，启动SDK程序即可。
常规测试场景下，云原生多模数据库在3主机集群规格下，可提供每秒百万写入能力。利用通用接口读取2GB的HDF5文件，约9.6亿条航空工程值数据，需17小时才能完全导入云原生多模数据库。假如一年产生100TB的HDF5文件，至少需要45天以上才能导入这些数据。这种导入速度过于缓慢，而采用本申请中的数据处理方法，2GB的HDF5文件导入云原生多模数据库从原来17个小时直接缩减为0.5小时，提升了34倍的导入速度，100T数据导入时间从45天缩短至1.5天。
基于相同的发明构思，本申请实施例还提供一种数据处理装置。如图5所示，其为数据处理装置500的结构示意图，可以包括：
获取单元501，用于获取多条报文数据以及各条报文数据的标识信息，报文数据的格式为层级数据格式HDF5，标识信息用于标识对应的报文数据的发送方和发送时间；
划分单元502，用于基于各条报文数据的标识信息，将各条报文数据划分为多个报文集合，每个报文集合中包含的报文数据的发送方相同，且发送时间属于同一时间区间；
映射单元503，用于基于预设映射规则，分别对多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并将各个报文记录存储在云原生多模数据库中。
可选的，映射单元503具体用于：
针对任意一个报文集合，基于任意一个报文集合中的各条报文数据的标识信息的相同字段，对各条报文进行合并得到合并报文；
基于预设映射规则对合并报文进行映射，获得报文集合对应的报文记录。
可选的，装置还包括解析单元504，用于；
分别对多条报文数据进行解析，获得各条标准格式的报文数据，标准格式为符合超文本传输协议http的压缩格式。
可选的，映射单元503具体用于：
基于预设映射规则，对合并报文包含的字节进行映射，获得报文集合对应的二进制格式的报文记录。
可选的，划分单元502具体用于：
基于各条报文数据的标识信息，确定各条报文数据的发送方，并基于各条报文数据的发送方，将发送方相同的各条报文数据划分至同一中间集合中，获得多个中间集合；
针对任意一个中间集合，基于任意一个中间集合中包含的报文数据的标识信息，确定报文数据的发送时间，并基于发送时间，将属于同一时间区间的报文数据划分至同一报文集合中，获得多个报文集合。
可选的，报文数据中包含用于表征报文数据的发送方当前的运行状态的信息。
为了描述的方便，以上各部分按照功能划分为各模块（或单元）分别描述。当然，在实施本申请时可以把各模块（或单元）的功能在同一个或多个软件或硬件中实现。
所属技术领域的技术人员能够理解，本申请的各个方面可以实现为系统、方法或程序产品。因此，本申请的各个方面可以具体实现为以下形式，即：完全的硬件实施方式、完全的软件实施方式（包括固件、微代码等），或硬件和软件方面结合的实施方式，这里可以统称为“电路”、“模块”或“系统”。
与上述方法实施例基于同一发明构思，本申请实施例中还提供了一种电子设备。在一种实施例中，该电子设备可以是服务器，如图1所示的服务器120。在该实施例中，电子设备的结构可以如图6所示，包括存储器601，通讯模块603以及一个或多个处理器602。
存储器601，用于存储处理器602执行的计算机程序。存储器601可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统，以及运行即时通讯功能所需的程序等；存储数据区可存储各种即时通讯信息和操作指令集等。
存储器601可以是易失性存储器（volatile memory），例如随机存取存储器（random-access memory，RAM）；存储器601也可以是非易失性存储器（non-volatile memory），例如只读存储器，快闪存储器（flash memory），硬盘（hard disk drive，HDD）或固态硬盘（solid-state drive，SSD）；或者存储器601是能够用于携带或存储具有指令或数据结构形式的期望的计算机程序并能够由计算机存取的任何其他介质，但不限于此。存储器601可以是上述存储器的组合。
处理器602，可以包括一个或多个中央处理单元（central processing unit， CPU）或者为数字处理单元等等。处理器602，用于调用存储器601中存储的计算机程序时实现上述数据处理方法。
通讯模块603用于与终端设备和其他服务器进行通信。
本申请实施例中不限定上述存储器601、通讯模块603和处理器602之间的具体连接介质。本申请实施例在图6中以存储器601和处理器602之间通过总线604连接，总线604在图6中以粗线描述，其它部件之间的连接方式，仅是进行示意性说明，并不引以为限。总线604可以分为地址总线、数据总线、控制总线等。为便于描述，图6中仅用一条粗线描述，但并不描述仅有一根总线或一种类型的总线。
存储器601中存储有计算机存储介质，计算机存储介质中存储有计算机可执行指令，计算机可执行指令用于实现本申请实施例的数据处理方法。处理器602用于执行上述的数据处理方法，如图2所示。
在另一种实施例中，电子设备也可以是其他电子设备，如图1所示的终端设备110。在该实施例中，电子设备的结构可以如图7所示，包括：通信组件710、存储器720、显示单元730、摄像头740、传感器750、音频电路760、蓝牙模块770、处理器780等部件。
通信组件710用于与服务器进行通信。在一些实施例中，可以包括电路无线保真（Wireless Fidelity，WiFi）模块，WiFi模块属于短距离无线传输技术，电子设备通过WiFi模块可以帮助用户收发信息。
存储器720可用于存储软件程序及数据。处理器780通过运行存储在存储器720的软件程序或数据，从而执行终端设备110的各种功能以及数据处理。存储器720可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。存储器720存储有使得终端设备110能运行的操作系统。本申请中存储器720可以存储操作系统及各种应用程序，还可以存储执行本申请实施例数据处理方法的计算机程序。
显示单元730还可用于显示由用户输入的信息或提供给用户的信息以及终端设备110的各种菜单的图形用户界面（graphical user interface，GUI）。具体地，显示单元730可以包括设置在终端设备110正面的显示屏732。其中，显示屏732可以采用液晶显示器、发光二极管等形式来配置。显示单元730可以用于显示本申请实施例中的数据处理用户界面等。
显示单元730还可用于接收输入的数字或字符信息，产生与终端设备110的用户设置以及功能控制有关的信号输入，具体地，显示单元730可以包括设置在终端设备110正面的触摸屏731，可收集用户在其上或附近的触摸操作，例如点击按钮，拖动滚动框等。
其中，触摸屏731可以覆盖在显示屏732之上，也可以将触摸屏731与显示屏732集成而实现终端设备110的输入和输出功能，集成后可以简称触摸显示屏。本申请中显示单元730可以显示应用程序以及对应的操作步骤。
摄像头740可用于捕获静态图像，用户可以将摄像头740拍摄的图像通过应用发布评论。摄像头740可以是一个，也可以是多个。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件（charge coupled device，CCD）或互补金属氧化物半导体（complementary metal-oxide-semiconductor，CMOS）光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给处理器780转换成数字图像信号。
终端设备还可以包括至少一种传感器750，比如加速度传感器751、距离传感器752、指纹传感器753、温度传感器754。终端设备还可配置有陀螺仪、气压计、湿度计、温度计、红外线传感器、光传感器、运动传感器等其他传感器。
音频电路760、扬声器761、传声器762可提供用户与终端设备110之间的音频接口。音频电路760可将接收到的音频数据转换后的电信号，传输到扬声器761，由扬声器761转换为声音信号输出。终端设备110还可配置音量按钮，用于调节声音信号的音量。另一方面，传声器762将收集的声音信号转换为电信号，由音频电路760接收后转换为音频数据，再将音频数据输出至通信组件710以发送给比如另一终端设备110，或者将音频数据输出至存储器720以便进一步处理。
蓝牙模块770用于通过蓝牙协议来与其他具有蓝牙模块的蓝牙设备进行信息交互。例如，终端设备可以通过蓝牙模块770与同样具备蓝牙模块的可穿戴电子设备（例如智能手表）建立蓝牙连接，从而进行数据交互。
处理器780是终端设备的控制中心，利用各种接口和线路连接整个终端的各个部分，通过运行或执行存储在存储器720内的软件程序，以及调用存储在存储器720内的数据，执行终端设备的各种功能和处理数据。在一些实施例中，处理器780可包括一个或多个处理单元；处理器780还可以集成应用处理器和基带处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，基带处理器主要处理无线通信。可以理解的是，上述基带处理器也可以不集成到处理器780中。本申请中处理器780可以运行操作系统、应用程序、用户界面显示及触控响应，以及本申请实施例的数据处理方法。另外，处理器780与显示单元730耦接。
在一些可能的实施方式中，本申请提供的数据处理方法的各个方面还可以实现为一种程序产品的形式，其包括计算机程序，当程序产品在电子设备上运行时，计算机程序用于使电子设备执行本说明书上述描述的根据本申请各种示例性实施方式的数据处理方法中的步骤，例如，电子设备可以执行如图2中所示的步骤。
程序产品可以采用一个或多个可读介质的任意组合。可读介质可以是可读信号介质或者可读存储介质。可读存储介质例如可以是但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。可读存储介质的更具体的例子（非穷举的列表）包括：具有一个或多个导线的电连接、便携式盘、硬盘、随机存取存储器（RAM）、只读存储器（ROM）、可擦式可编程只读存储器（EPROM或闪存）、光纤、便携式紧凑盘只读存储器（CD-ROM）、光存储器件、磁存储器件、或者上述的任意合适的组合。
本申请的实施方式的程序产品可以采用便携式紧凑盘只读存储器（CD-ROM）并包括计算机程序，并可以在电子设备上运行。然而，本申请的程序产品不限于此，在本文件中，可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被命令执行系统、装置或者器件使用或者与其结合使用。
可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了可读计算机程序。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。可读信号介质还可以是可读存储介质以外的任何可读介质，该可读介质可以发送、传播或者传输用于由命令执行系统、装置或者器件使用或者与其结合使用的程序。
可读介质上包含的计算机程序可以用任何适当的介质传输，包括但不限于无线、有线、光缆、RF等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言的任意组合来编写用于执行本申请操作的计算机程序，程序设计语言包括面向对象的程序设计语言—诸如Java、C++等，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。计算机程序可以完全地在用户电子设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户电子设备上部分在远程电子设备上执行、或者完全在远程电子设备或服务器上执行。在涉及远程电子设备的情形中，远程电子设备可以通过任意种类的网络包括局域网（LAN）或广域网（WAN）连接到用户电子设备，或者，可以连接到外部电子设备（例如利用因特网服务提供商来通过因特网连接）。
应当注意，尽管在上文详细描述中提及了装置的若干单元或子单元，但是这种划分仅仅是示例性的并非强制性的。实际上，根据本申请的实施方式，上文描述的两个或更多单元的特征和功能可以在一个单元中具体化。反之，上文描述的一个单元的特征和功能可以进一步划分为由多个单元来具体化。
此外，尽管在附图中以特定顺序描述了本申请方法的操作，但是，这并非要求或者暗示必须按照该特定顺序来执行这些操作，或是必须执行全部所示的操作才能实现期望的结果。附加地或备选地，可以省略某些步骤，将多个步骤合并为一个步骤执行，和/或将一个步骤分解为多个步骤执行。
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用计算机程序的计算机可用存储介质（包括但不限于磁盘存储器、CD-ROM、光学存储器等）上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备（系统）、和计算机程序产品的流程图和／或方框图来描述的。应理解可由计算机程序命令实现流程图和／或方框图中的每一流程和／或方框、以及流程图和／或方框图中的流程和／或方框的结合。可提供这些计算机程序命令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的命令产生用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序命令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的命令产生包括命令装置的制造品，该命令装置实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能。
这些计算机程序命令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的命令提供用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的步骤。
尽管已描述了本申请的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例做出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本申请范围的所有变更和修改。
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。

Claims (10)

  1. 一种数据处理方法，其特征在于，该方法包括：
    获取多条报文数据以及各条所述报文数据的标识信息，所述报文数据的格式为层级数据格式HDF5，所述标识信息用于标识对应的报文数据的发送方和发送时间；
    基于各条所述报文数据的标识信息，将各条所述报文数据划分为多个报文集合，每个报文集合中包含的报文数据的发送方相同，且发送时间属于同一时间区间；
    基于预设映射规则，分别对所述多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并将各个所述报文记录存储在云原生多模数据库中。
  2. 如权利要求1所述的方法，其特征在于，所述基于预设映射规则，分别对所述多个报文集合中的报文数据进行压缩，获得各个报文集合各自的报文记录，包括：
    针对任意一个报文集合，基于所述任意一个报文集合中的各条报文数据的标识信息的相同字段，对所述各条报文进行合并得到合并报文；
    基于预设映射规则对所述合并报文进行映射，获得所述报文集合对应的报文记录。
  3. 如权利要求1所述的方法，其特征在于，在所述基于各条报文数据各自的标识信息，将所述各条报文数据划分为多个报文集合之前，还包括；
    分别对多条所述报文数据进行解析，获得各条标准格式的报文数据，所述标准格式为符合超文本传输协议http的压缩格式。
  4. 如权利要求2所述的方法，其特征在于，所述基于预设映射规则对所述合并报文进行映射，获得所述报文集合对应的报文记录包括：
    基于预设映射规则，对所述合并报文包含的字节进行映射，获得所述报文集合对应的二进制格式的所述报文记录。
  5. 如权利要求1所述的方法，其特征在于，所述基于各条所述报文数据的标识信息，将各条所述报文数据划分为多个报文集合，包括：
    基于各条所述报文数据的标识信息，确定各条所述报文数据的发送方，并基于各条所述报文数据的发送方，将各条所述报文数据划分为多个中间集合，同一中间集合中的报文数据的发送方相同；
    针对任意一个中间集合，基于所述任意一个中间集合中包含的各条报文数据的标识信息，确定各条所述报文数据的发送时间，并基于各条所述报文数据的发送时间，将各条所述报文数据划分为多个报文集合，同一报文集合中的报文数据的发送时间属于同一预设时间区间。
  6. 如权利要求1所述的方法，其特征在于，所述报文数据中包含用于表征所述报文数据的发送方当前的运行状态的信息。
  7. 一种数据处理装置，其特征在于，包括：
    获取单元，用于获取多条报文数据以及各条所述报文数据的标识信息，所述报文数据的格式为层级数据格式HDF5，所述标识信息用于标识对应的报文数据的发送方和发送时间；
    划分单元，用于基于各条所述报文数据的标识信息，将各条所述报文数据划分为多个报文集合，每个报文集合中包含的报文数据的发送方相同，且发送时间属于同一时间区间；
    映射单元，用于基于预设映射规则，分别对所述多个报文集合中的报文数据进行映射，获得各个报文集合各自的报文记录，并将各个所述报文记录存储在云原生多模数据库中。
  8. 一种电子设备，其特征在于，其包括处理器和存储器，其中，所述存储器存储有计算机程序，当所述计算机程序被所述处理器执行时，使得所述处理器执行权利要求1~6中任一所述方法的步骤。
  9. 一种计算机可读存储介质，其特征在于，其包括计算机程序，当所述计算机程序在电子设备上运行时，所述计算机程序用于使所述电子设备执行权利要求1~6中任一所述方法的步骤。
  10. 一种计算机程序产品，其特征在于，包括计算机程序，所述计算机程序存储在计算机可读存储介质中；当电子设备的处理器从所述计算机可读存储介质读取所述计算机程序时，所述处理器执行所述计算机程序，使得所述电子设备执行权利要求1~6中任一所述方法的步骤。
PCT/CN2022/141519 2022-07-29 2022-12-23 一种数据处理方法、装置、电子设备和存储介质 WO2024021476A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210902315.X 2022-07-29
CN202210902315.XA CN115297183B (zh) 2022-07-29 2022-07-29 一种数据处理方法、装置、电子设备和存储介质

Publications (1)

Publication Number Publication Date
WO2024021476A1 true WO2024021476A1 (zh) 2024-02-01

Family

ID=83823501

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/141519 WO2024021476A1 (zh) 2022-07-29 2022-12-23 一种数据处理方法、装置、电子设备和存储介质

Country Status (2)

Country Link
CN (1) CN115297183B (zh)
WO (1) WO2024021476A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115297183B (zh) * 2022-07-29 2023-11-03 天翼云科技有限公司 一种数据处理方法、装置、电子设备和存储介质
CN116521063B (zh) * 2023-03-31 2024-03-26 北京瑞风协同科技股份有限公司 一种hdf5的试验数据高效读写方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050222964A1 (en) * 2002-07-29 2005-10-06 Marco Winter Database model for hierarchical data formats
US20180089250A1 (en) * 2016-09-26 2018-03-29 Amazon Technologies, Inc. Atomic application of multiple updates to a hierarchical data structure
CN109597575A (zh) * 2018-11-29 2019-04-09 中国科学院合肥物质科学研究院 一种基于hdf5的分块式数据存储及读取方法
CN113890902A (zh) * 2021-09-15 2022-01-04 奇安信科技集团股份有限公司 特征识别库的构建方法及装置、流量识别方法
CN115297183A (zh) * 2022-07-29 2022-11-04 天翼云科技有限公司 一种数据处理方法、装置、电子设备和存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100586109C (zh) * 2007-12-20 2010-01-27 北京大学 基于自定义模板的通用业务数据通讯方法与系统
EP2700081B1 (en) * 2011-04-22 2022-11-02 ASML Netherlands B.V. Network architecture for lithography machine cluster
CN105049357B (zh) * 2015-06-18 2018-04-24 华北科技学院 基于北斗报文的物联网实时监测远程传输有效性优化方法
CN107818118B (zh) * 2016-09-14 2019-04-30 北京百度网讯科技有限公司 数据存储方法和装置
CN110928681A (zh) * 2019-11-11 2020-03-27 北京明略软件系统有限公司 数据的处理方法和装置、存储介质及电子装置
CN111897831A (zh) * 2020-07-31 2020-11-06 平安普惠企业管理有限公司 业务报文的生成方法、装置、电子设备及存储介质
CN112468370B (zh) * 2020-11-30 2024-02-20 北京锐驰信安技术有限公司 一种支持自定义规则的高速网络报文监测分析方法及系统
CN112925749A (zh) * 2021-02-20 2021-06-08 北京同邦卓益科技有限公司 一种数据处理方法、装置、电子设备及存储介质
CN113329099A (zh) * 2021-06-29 2021-08-31 中国农业银行股份有限公司 报文解析方法、装置、介质、设备及程序产品
CN114006831B (zh) * 2021-10-30 2023-07-21 杭州迪普信息技术有限公司 报文数据处理方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050222964A1 (en) * 2002-07-29 2005-10-06 Marco Winter Database model for hierarchical data formats
US20180089250A1 (en) * 2016-09-26 2018-03-29 Amazon Technologies, Inc. Atomic application of multiple updates to a hierarchical data structure
CN109597575A (zh) * 2018-11-29 2019-04-09 中国科学院合肥物质科学研究院 一种基于hdf5的分块式数据存储及读取方法
CN113890902A (zh) * 2021-09-15 2022-01-04 奇安信科技集团股份有限公司 特征识别库的构建方法及装置、流量识别方法
CN115297183A (zh) * 2022-07-29 2022-11-04 天翼云科技有限公司 一种数据处理方法、装置、电子设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XUEFENG Lü; CHENGQI CHENG; JIANYA GONG; LI GUAN: "Review of data storage and management technologies for massive remote sensing data", SCIENCE IN CHINA SERIES E ; TECHNOLOGICAL SCIENCES, SCIENCE IN CHINA PRESS, BE, vol. 54, no. 12, 25 August 2011 (2011-08-25), Be , pages 3220 - 3232, XP019983057, ISSN: 1862-281X, DOI: 10.1007/s11431-011-4549-z *

Also Published As

Publication number Publication date
CN115297183B (zh) 2023-11-03
CN115297183A (zh) 2022-11-04

Similar Documents

Publication Publication Date Title
US10122788B2 (en) Managed function execution for processing data streams in real time
WO2024021476A1 (zh) 一种数据处理方法、装置、电子设备和存储介质
US9984408B1 (en) Method, medium, and system for live video cooperative shopping
JP2021010156A (ja) 情報を生成する方法および装置
CN111949850B (zh) 多源数据的采集方法、装置、设备及存储介质
JP2021103506A (ja) 情報を生成するための方法及び装置
CN111352800A (zh) 大数据集群监控方法及相关设备
US11494437B1 (en) System and method for performing object-modifying commands in an unstructured storage service
CN113282611B (zh) 一种流数据同步的方法、装置、计算机设备及存储介质
WO2018177032A1 (zh) 处理响应数据的方法、设备、客户端设备和电子设备
CN114035748A (zh) 一种数据文件的存取方法与系统
CN112181678A (zh) 业务数据的处理方法、装置和系统、存储介质、电子装置
CN112699111B (zh) 报表生成方法、装置、电子设备和计算机可读介质
CN113724398A (zh) 增强现实方法、装置、设备以及存储介质
WO2020098347A1 (zh) 信息生成方法、装置和系统
CN116069462A (zh) 一种大数据dag任务流调度方法、系统及存储介质
WO2021258972A1 (zh) 视频检索方法、装置、电子设备和计算机可读介质
CN110956349B (zh) 服务质量分析方法、系统、装置、服务器及电子设备
CN113779018A (zh) 一种数据处理方法和装置
KR20100067290A (ko) 데이터의 구조화 및 접근을 위한 시스템 및 방법
CN114448976B (zh) 网络报文的组装方法、装置、设备、介质和程序产品
US10516767B2 (en) Unifying realtime and static data for presenting over a web service
US11819762B1 (en) Registration and control of a game entity as a device
CN113342837B (zh) 数据发送方法、装置、电子设备和计算机可读介质
CN113986850B (zh) 电子卷宗的存储方法、装置、设备和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22952907

Country of ref document: EP

Kind code of ref document: A1