CN110502563B - Data processing method and device of multiple data sources and storage medium - Google Patents
Data processing method and device of multiple data sources and storage medium Download PDFInfo
- Publication number
- CN110502563B CN110502563B CN201910792843.2A CN201910792843A CN110502563B CN 110502563 B CN110502563 B CN 110502563B CN 201910792843 A CN201910792843 A CN 201910792843A CN 110502563 B CN110502563 B CN 110502563B
- Authority
- CN
- China
- Prior art keywords
- data
- field
- input data
- target
- difference
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method and device for multiple data sources, electronic equipment and a storage medium, wherein the data processing method for the multiple data sources comprises the following steps: receiving source input data from a plurality of data sources, converting the source input data into input data conforming to a standard format; according to the identification information of the input data, searching target historical data corresponding to the identification information in a database; when target historical data corresponding to the identification information exists in the database, comparing the input data with the target historical data to obtain a difference data set; updating the target historical data according to the difference data set; when the target history data does not exist in the database, the input data is stored in the database. The time consumption of data updating can be shortened, and the storage space occupied by the data is reduced.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a method and an apparatus for processing data of multiple data sources, and a storage medium.
Background
When the data is received by the data sources, the data amount of source input data provided by each data source is large, the data are in different formats, when the data in the data base are updated, all the received data are written into the data base by the related data processing method, the data processing method consumes long time, the occupied storage space is large, when the intelligent sound box is used for processing music data, for example, different music data are required to be acquired from different music providers, the data amount of the music data is large and the data have different formats, when the related intelligent sound box is used for updating the data, all the received music data are written into the data base, and the processing method of the music data consumes long time and occupies large storage space.
Disclosure of Invention
The embodiment of the invention provides a data processing method and device for multiple data sources, electronic equipment and a storage medium, which can shorten time consumption and reduce storage space occupied by data.
The technical scheme of the embodiment of the invention is realized as follows:
the embodiment of the invention provides a data processing method of multiple data sources, which comprises the following steps:
receiving source input data from a plurality of data sources, converting the source input data into input data conforming to a standard format;
according to the identification information of the input data, searching target historical data corresponding to the identification information in a database;
when target historical data corresponding to the identification information exists in the database, comparing the input data with the target historical data to obtain a difference data set;
updating the target historical data according to the difference data set;
and when the target historical data does not exist in the database, storing the input data into the database.
The embodiment of the invention also provides a data processing device with multiple data sources, which comprises:
the acquisition module is used for receiving source input data from a plurality of data sources and converting the source input data into input data conforming to a standard format;
The retrieval module is used for retrieving target historical data corresponding to the identification information in a database according to the identification information of the input data;
the comparison module is used for comparing the input data with the target historical data to obtain a difference data set when the target historical data corresponding to the identification information exists in the database;
the updating module is used for updating the target historical data according to the difference data set; and when the target historical data does not exist in the database, storing the input data into the database.
In the above scheme, the obtaining module is further configured to obtain the standard format according to a format of the historical data in the database; the format of the input data is converted into the standard format.
In the above scheme, the search module is further configured to confirm a field in the input data as an identity, where the identity is a field capable of uniquely identifying the input data; and retrieving the historical data with the same data content as the field in the database, and confirming the retrieved historical data as the target historical data.
In the above scheme, the comparison module is further configured to compare a field in the input data with a target field in the target history data, and form a difference data set based on a field with a difference;
wherein, the field name of the target field is the same as the field name of the field in the input data.
In the above aspect, the retrieving module is further configured to retrieve the target field in the target history data based on a field name of a field in the input data;
the comparison module is further configured to confirm a field in the input data as a newly added field when the target field does not exist, and store the newly added field in the difference data set; and when the target field exists, comparing the field in the input data with the target field, and when the field in the input data is different from the target field, confirming the field in the input data as a replacement field and storing the replacement field into the difference data set.
In the above solution, the updating module is further configured to replace a field in the target history data corresponding to a replacement field with the replacement field when the field in the difference data set is the replacement field; and when the field in the difference data set is a new field, writing the new field into the target historical data.
In the above scheme, the acquiring module is further configured to receive operation data;
the comparison module is further configured to correct the difference data set based on the operation data.
The embodiment of the invention provides a data processing device of multiple data sources, which comprises:
a memory for storing executable instructions;
and the processor is used for realizing the method provided by the embodiment of the invention when executing the executable instructions stored in the memory.
The embodiment of the invention provides a storage medium which stores executable instructions for realizing the data processing method of multiple data sources provided by the embodiment of the invention when being executed by a processor.
The embodiment of the invention has the following beneficial effects:
when the data in the database is updated, the input data is compared with the historical data in the database, and only the part, which is different from the historical data, in the input data is stored in the database, so that the time for storing the same part, which is identical to the historical data, in the input data in the database is saved, the part, which is identical to the historical data, in the input data is prevented from occupying the storage space of the database, the time consumption of data updating can be shortened, and the storage space occupied by the data is reduced.
Drawings
FIG. 1 is a schematic diagram of an alternative architecture of a database update architecture according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an alternative architecture of a server according to an embodiment of the present invention;
FIG. 3 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 4 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 5 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 6 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 7 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 8 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 9 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 10 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention;
FIG. 11 is a flowchart comparing input music data, historical music data and manual operation data in a multi-data source data processing method according to an embodiment of the present invention;
FIG. 12 is a flow chart of comparing input music data with historical music data to obtain a difference data set in the data processing method of multiple data sources according to the embodiment of the invention;
fig. 13 is a schematic diagram of a frame of a multi-data source data processing apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be further described in detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present invention more apparent, and the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present invention.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.
The embodiment of the invention provides a data processing method, a device, equipment and a storage medium for multiple data sources, and an exemplary application of the terminal equipment provided by the embodiment of the invention is firstly described below.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an alternative database update architecture according to an embodiment of the present invention, and the process of updating data in a database of a multimedia playing platform is described below with reference to fig. 1.
The server 300 of the multimedia playing platform periodically transmits a data acquisition request to the server 100-1 of company a, the server 100-2 of company B, and the server 100-3 of company C, wherein company a, company B, and company C are providers of multimedia data. After the server 100-1 of company a, the server 100-2 of company B, and the server 100-3 of company C receive the data acquisition request, based on the content of the multimedia service purchased by the playback platform, the corresponding multimedia data is transmitted to the server 300 of the multimedia playback platform through the network 200, for example, the multimedia playback platform purchases the copyrights of songs of the singer X from company a, and the server 100-1 of company a transmits the music data of all the singers X stored in the server 100-1 of company a to the server 300 of the multimedia playback platform through the network after receiving the data acquisition request of the server 300 of the multimedia playback platform. The data processing device 355 of the multimedia playing platform 300 processes the multimedia data after receiving the multimedia data, and updates the multimedia data stored in the database 3 based on the processed data.
The multimedia playing device 400 is installed with a multimedia playing client 410, the multimedia playing device 400 transmits a user instruction triggered by a user in the multimedia playing client 410 to the server 300 of the multimedia playing platform, and the server 300 of the multimedia playing platform retrieves multimedia data corresponding to the user instruction in the database 356 based on the instruction and transmits the multimedia data to the multimedia playing device 400. The multimedia playing device 400 receives the multimedia data and plays the multimedia data through the multimedia playing client.
In some embodiments, the multimedia playing device 400 may be any device that can play multimedia data, such as a smart phone, smart television, or personal computer.
In some embodiments, the server 300 of the multimedia playing platform, when transmitting multimedia data to the multimedia playing device 400, compression-encodes the multimedia data, for example, a combination of compression algorithms employing the h.264 video compression algorithm and the advanced audio coding (AAC, advanced Audio Coding) algorithm. Taking compression encoding of video data in multimedia data by using an h.264 video compression algorithm as an example, the encoding of the video data at the video encoding layer is realized through a video encoding layer (VCL, video Coding Layer) and a network extraction layer (N AL, network Abstraction Layer) of the h.264, the encoded multimedia data including motion estimation, entropy encoding and other contents are encapsulated to form a multimedia data stream, and the encoded multimedia data is transmitted to a multimedia playing device 400 held by a user in a streaming manner such as a real-time streaming manner (Real time streaming) and a sequential streaming manner (progressive streami ng) for the multimedia playing client 410 of the multimedia playing device 400 to call decoding of an operating system for decoding and presenting in a playing interface of the multimedia playing device 400.
In some embodiments, the server 300 of the multimedia playing platform divides the multimedia data into a plurality of parts when transmitting the multimedia data to the multimedia playing device 400. After receiving a portion of multimedia data, the multimedia playing device 400 invokes decoding of the operating system by the multimedia playing client 410 to decode and present the portion of multimedia data in the playing interface of the multimedia playing device 400, and receives and decodes the rest of the multimedia data while playing the portion of multimedia data.
Referring to fig. 2, fig. 2 is a schematic diagram of an alternative structure of a server 300 (for example, the server 300 may be the server 300 of the multimedia playing platform shown in fig. 1) according to an embodiment of the present invention, where the server shown in fig. 2 includes: at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in server 300 are coupled together by bus system 340. It is understood that the bus system 340 is used to enable connected communications between these components. The bus system 340 includes a power bus, a control bus, and a status signal bus in addition to the data bus. But for clarity of illustration the various buses are labeled in fig. 2 as bus system 340.
The processor 310 may be an integrated circuit chip with signal processing capabilities such as a general purpose processor, which may be a microprocessor or any conventional processor, or the like, a digital signal processor (DSP, digital Signal Processor), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.
The user interface 330 includes one or more output devices 331, including server status indicator lights, that enable presentation of server status. The user interface 330 also includes one or more input devices 332 that include components that facilitate controlling the operating state of the server, such as a switch of the server or a restart button of the server.
Memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical drives, and the like. Memory 350 optionally includes one or more storage devices physically located remote from processor 310.
Memory 350 includes volatile memory or nonvolatile memory, and may also include both volatile and nonvolatile memory. The non-volatile memory may be read only memory (ROM, read Only Me mory) and the volatile memory may be random access memory (RAM, random Access Memor y). The memory 350 described in embodiments of the present invention is intended to comprise any suitable type of memory.
In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules and data structures, or subsets or supersets thereof, as exemplified below.
The operating system 351, which includes system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., is used to implement various basic services and process hardware-based tasks.
Network communication module 352 for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interface 320 includes: bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB, universal Serial Bus), etc.
A presentation unit 353 for enabling presentation of information (e.g., whether the server is overheated) via one or more output devices 331 (e.g., server status indicators) associated with the user interface 330.
An input processing module 354 for detecting one or more user inputs or interactions from one of the one or more input devices 332 and translating the detected inputs or interactions.
Database 356 stores data from network communication module 352.
In some embodiments, the multi-data-source data processing apparatus provided in the embodiments of the present invention may be implemented in software, and fig. 2 shows a multi-data-source data processing apparatus 355 stored in a memory 350, including four modules, i.e., an acquisition module 3551, a retrieval module 3552, a comparison module 3453, and an update module 3454, which may be software in the form of programs and plug-ins, and may be embedded in various servers, e.g., an online video server, and a data statistics server for mobile service communications. The acquisition module 3551 is configured to receive source input data from a plurality of data sources, and convert the source input data into input data that conforms to a standard format. And a retrieving module 3552 for retrieving the target history data corresponding to the identification information from the database 356 according to the identification information of the input data. And a comparison module 3553, configured to compare the input data with the target history data to obtain a difference data set when the target history data corresponding to the identification information exists in the database 356. An updating module 3554, configured to update the target historical data according to the difference data set; when there is no target history data in the database 356, the input data is stored in the database 356.
In other embodiments, the multi-data-source data processing apparatus provided by the embodiments of the present invention may be implemented in hardware, and may be, for example, a processor in the form of a hardware decoding processor programmed to perform the multi-data-source data processing method provided by the embodiments of the present invention, for example, a processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASIC, application Specific Integrated Circuit), DSPs, programmable logic devices (PLD, programmable Log ic Device), complex programmable logic devices (CPLD, complex Programmable Logic Dev ice), field programmable gate arrays (FPGA, field-Programmable Gate Array), or other electronic components.
The data processing method of multiple data sources provided by the embodiment of the invention can be executed by a server, and the process of implementing the data processing method of multiple data sources in the server through the embedded data processing device of multiple data sources in the server is described below in combination with the exemplary application and structure of the server, and the types of the server can be diversified, for example, a server special for a database, a server of a video playing platform, a server of a social network platform and the like; or a network cloud server constructed by a virtualization technology. In one embodiment, the client may virtualize the plurality of physical computing nodes into a network cloud server through a virtualization technology, abstract computing resources, network resources and storage resources of the plurality of physical computing nodes into a computing resource pool, a network resource pool and a storage resource pool, and implement a data processing method of multiple data sources by calling the computing resource pool, the network resource pool and the storage resource pool.
Referring to fig. 3, fig. 3 is a flow chart of a data processing method of multiple data sources according to an embodiment of the invention, and as shown in fig. 3, the flow chart of the data processing method includes:
step S201, the server receives source input data from a plurality of data sources and converts the source input data into input data conforming to a standard format.
In some embodiments, the source input data received from the different data sources is in a different format, and the server converts the source input data in the different format into input data conforming to a standard format to convert input data in the different format into input data in a uniform format.
In some embodiments, converting the source input data into input data conforming to a standard format includes: the field naming of the input data is converted into the field naming conforming to the standard, the field arrangement sequence of the input data is converted into the standard field arrangement sequence, and the data type of the input data is converted into the standard data type.
For example, the server acquires mobile communication data from three mobile service operators, respectively, the server acquires first source mobile communication data from a first mobile service operator, acquires second source mobile communication data from a second mobile service operator, and acquires third source mobile communication data from a third mobile service operator. The first source mobile communication data, the second source mobile communication data, and the third source mobile communication data are shown in table 1.
Table 1 source mobile communication data acquired by mobile service provider
The first source mobile communication data includes three fields, namely a phone number 11111111, a call duration of 14000 minutes, and an uplink traffic of 20GB (gigabits, the meanings of GB are the same herein). The second source mobile communication data comprises three fields of a user number, a month call duration and an uploading flow, wherein the user number is 22222222222, the month call duration is 100 hours, and the uploading flow is 10GB. The third source mobile communication data comprises three fields of telephone number, total monthly call duration and uploading traffic, wherein the telephone number is 33333333333, the total monthly call duration is 25 hours, and the uploading traffic is 30GB.
Converting the first source mobile communication data, the second source mobile communication data and the third source mobile communication data into mobile communication data conforming to a standard format includes converting field names of the source mobile communication data into mobile communication data conforming to the standard format, sorting fields conforming to the standard field names according to a standard sequence, converting data of the fields into the same type, for example, converting data of the same field in different source data into data with the same units and the same precision. The converted mobile communication data conforming to the standard format is shown in table 2.
Table 2 mobile communication data conforming to standard format
The mobile communication data conforming to the standard format comprises three fields of telephone numbers, total monthly call duration and uploading flow, wherein the three fields are arranged in the order of the telephone numbers, the total monthly call duration and the uploading flow in the mobile communication data, the data in the total monthly call duration fields are in units of hours, the last two digits of a decimal point are in precision, the data in the uploading flow fields are in units of GB, and the zero position after the decimal point is in precision.
By converting the input data into input data conforming to the standard format, the formats of the input data received by different data sources are unified, so that the data stored in the database can be ensured to have a consistent format, and the difficulty of searching the data in the database is reduced.
Step S202, the server searches the target historical data corresponding to the identification information in the database according to the identification information of the input data.
In some embodiments, the identification information refers to a field capable of uniquely identifying the input data. In some embodiments, the identification information refers to a field that can uniquely identify the input data. For example, the input data are mobile communication data, each mobile communication data has a different phone number field, the mobile communication data can be uniquely identified through the phone number field, and the phone number field is an identity of the mobile communication data. In other embodiments, the identification information refers to a plurality of fields capable of uniquely identifying the input data. For example, the input data is music data, one music data can be distinguished from other music data by a song name field, a singer name field and a version number field, and the music data can be uniquely identified, and the song name field, the singer name field and the version number field are the identity of the music data.
In some embodiments, the identity of the target history data is the same as the identity of the input data.
In step S203, when target history data corresponding to the identification information exists in the database, the server compares the input data with the target history data to obtain a difference data set.
In some embodiments, the fields of the input data are compared with corresponding fields of the target history data, and when the fields of the input data are different from the fields of the target history data, the comparison of the remaining fields is stopped, and all the fields of the input data are stored in the difference data set. In other embodiments, the fields of the input data are compared to corresponding fields of the target history data, and a difference data set is formed based on the fields of the input data that differ from the target history data.
For example, the input data is music data including four fields of song name, singer name, version number, album of which song name is X, singer name is Y, version number is 2015 years recording studio edition, albums of which W and Z are belonged, and the target history data in the database includes four fields of song name, singer name, version number, album of which song name is X, singer name is Y, version number is 2015 years recording studio edition, album of which W is belonged. And comparing the input data with the target historical data, and storing the album field into a difference data set.
And step S204, the server updates the target historical data according to the difference data set.
In some embodiments, the difference data set includes an identity of the target historical data and a corresponding difference field, the corresponding target historical data is retrieved based on the identity of the target historical data, a corresponding field in the target historical data is retrieved based on a field name of the difference field, and the corresponding field is rewritten into the difference field, so that the update of the target historical data is realized.
By comparing the input data with the target historical data to form a differential data set and updating the target historical data according to the differential data set, only the part of the input data, which is different from the historical data, is stored in the database, so that the time for storing the same part of the input data as the historical data in the database is saved, the part of the input data, which is the same as the historical data, is prevented from occupying the storage space of the database, the time consumption for updating the data can be shortened, and the storage space occupied by the data is reduced.
Step S205, when the target historical data does not exist in the database, the server stores the input data into the database.
In some embodiments, when the target historical data does not exist in the database, the identity of the historical data in the database is marked as different from the input data, the input data is the data which does not exist in the database, all fields of the input data are stored in the database, and the data in the database are updated.
In some embodiments, when there is no target history data in the database, the input data is stored in the difference data set, and the history data in the database is updated based on the difference data set. When updating the historical data in the database based on the difference data, the target historical data corresponding to the difference data in the difference data set is searched in the database, the difference data which can search the target historical data is updated based on the difference data, and the difference data which can not search the target historical data is stored in the database.
In some embodiments, as shown in fig. 4, fig. 4 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention, based on fig. 3, step S201 includes:
and step S2011, the server obtains a standard format according to the format of the historical data in the database.
In some embodiments, field names, field ordering, and data types of the fields in the historical data in the database are determined as standard formats to conform the format of the input data to the format of the historical data in the database.
In step S2012, the server converts the format of the input data into a standard format.
In some embodiments, the field names, field ordering, and data types of the fields of the input data in the standard format are the same as the field names, field ordering, and data types of the fields in the database.
By making the format of the input data identical to the format of the history data in the database, it is possible to ensure that the target history data cannot be retrieved due to the difference of field names when the target history data is retrieved in the database. In the exemplary description of the input data as mobile communication data, the identity of the input data is a telephone number field, the identity of the history data in the database is a telephone number field, and when the target history data is searched, even if the history data in the database, which has the same telephone number field as the output data, exists, the history data cannot be searched due to different field names.
By making the format of the input data consistent with the format of the historical data in the database, the input data can be ensured not to be stored in the differential data set due to different data types when the input data is compared with the target historical data, the input data is taken as mobile communication data for carrying out the exemplary explanation, the month total call duration field in the input data is 1200 minutes, the month total call duration field in the target historical data is 20 hours, and the month total call duration field in the input data is converted into data in units of hours by converting the unit of the month total call duration field in the input data into data in units of hours, so that the month total call duration in the input data is prevented from being stored in the differential data set due to different units adopted by the fields.
In some embodiments, referring to fig. 5, fig. 5 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention, based on fig. 3, step S202 includes:
in step S2021, the server confirms the field in the input data as an identity, and the identity is a field capable of uniquely identifying the input data.
In some embodiments, the identity of each input data is different, and one input data can be distinguished from the other input data by the identity.
In some embodiments, the fields which are included in all the input data and can uniquely identify the input data are confirmed as the identity, so that the identity deletion of the input data caused by taking the fields which are not included in part of the input data as the identity is avoided.
Step S2022, the server retrieves the history data having the same data content as the field of the identity of the input data in the database, and confirms the retrieved history data as the target history data.
For example, the input data is music data, the fields of the identity of the input data include three fields of song name, singer name and version number, the song name is X, the singer name is Y, the version number is 2015 recording studio edition, the history data of the song name is X, the singer name is Y, the version number is 2015 recording studio edition are searched in the database, and the searched history data is confirmed as target history data.
By retrieving the historical data with the same data content as the field of the identity mark of the input data and confirming the retrieved historical data as the target historical data, when the input data is compared with the historical data, the input data is only compared with the corresponding target historical data, and the input data is not required to be compared with all the historical data in the database one by one, so that the time for comparing the input data with the historical data is shortened, and the time consumed by data updating is shortened.
In some embodiments, referring to fig. 6, fig. 6 is a flowchart of a data processing method of multiple data sources according to an embodiment of the present invention, based on fig. 3, step S203 includes:
in step S2031, the server compares the fields in the input data with the target fields in the target history data, and forms a difference data set based on the fields having differences.
In some embodiments, the field name of the target field is the same as the field name of the field in the input data. The process of comparing input data with target data is exemplarily described below with the input data as music data.
The input data includes seven fields of song name, singer name, version number, composer name, album to which the singer belongs and company to which the singer belongs, the song name is X, the singer name is Y, the version number is 2015 recording studio version, the composer name is XX, the composer name is XY, the album to which the singer belongs is Z and W, and the company to which the singer belongs is DD. The target history data includes seven fields of song name, singer name, version number, composer name, album to which the singer belongs and company to which the singer belongs, the song name is X, the singer name is Y, the version number is 2015 recording studio version, the composer name is XX, the composer name is YY, the album to which the singer belongs is W, and the company to which the singer belongs is DD.
The method comprises the steps of comparing a song name field in input data with a song name field in target historical data, comparing a singer name field in the input data with a singer name field in the target historical data, comparing a version number field in the input data with a version number field in the target historical data, comparing a composer name in the input data with a composer field in the target historical data, comparing an album field in the input data with an album field in the target historical data, and comparing a company field of the singer in the input data with a company field of the singer in the target historical data. The difference data set is formed based on the fields where the differences exist, i.e., the difference data set is formed based on the learner field and the album field in the input data.
By comparing the fields in the input data with the target fields in the target historical data, the fields in the input data are not required to be compared with all the fields in the target historical data one by one, so that the time for comparing the input data with the target historical data is shortened, and the time consumed by data updating is further shortened.
In some embodiments, fields in the input data other than the fields corresponding to the identity are compared with fields in the target history data. Because the target historical data is the same field as the data content of the field corresponding to the identity of the input data, the data content of the field corresponding to the identity of the input data is necessarily consistent with the data content of the corresponding target field, so that the field corresponding to the identity in the input data is not compared with the target field, and the time for comparing the field corresponding to the identity with the target field can be saved.
In some embodiments, referring to fig. 7, fig. 7 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention, based on fig. 6, step S2031 includes:
in step S20311, the server searches the target history data for the target field based on the field name of the field in the input data.
In some embodiments, the target field is the same field as the field name of the field in the input data.
In step S20312, when the target field does not exist, the server confirms the field in the input data as a newly added field, and stores the newly added field in the difference data set.
In some embodiments, there is no target history field in the target history data, indicating that the field in the input data is a newly defined field that needs to be stored in the difference data set to update the data structure of the target history data.
In some embodiments, the identity of the input data to which the newly added field belongs is stored in the difference data set, and a correspondence between the newly added field and the identity is established.
In step S20313, when the target field exists, the server compares the field in the input data with the target field, and when the field in the input data is different from the target field, confirms the field in the standard data as a replacement field, and stores the replacement field in the difference data set.
In some embodiments, when the target field exists in the target history data, the field in the input data is a defined field, but the field in the input data is different from the data content of the target field, and the data content of the target field of the input data needs to be updated to the content of the corresponding field of the input data.
In some embodiments, the identity of the input data to which the replacement field belongs is stored in the difference data set, and a correspondence between the replacement data and the identity is established.
In some embodiments, referring to fig. 8, fig. 8 is a flowchart of a data processing method of multiple data sources according to an embodiment of the present invention, based on fig. 3, step S204 includes:
in step S2041, when a field in the differential data set is a replacement field, the server replaces a field corresponding to the replacement field in the target history data with the replacement field.
In some embodiments, the replacement field is a field in the input data that is different from the data content of the target field, and the content of the target field is replaced with the content of the replacement field. For example, the replacement field is an album field to which the data content of the replacement field is Z, W, the target field is an album field to which the data content of the target field in the target history data is W, and the data content W of the target field in the target history data is replaced with the data content Z, W of the replacement field.
In some embodiments, the target history data in the database is retrieved based on the identity in the set of difference data corresponding to the replacement field, and the field in the target history data corresponding to the replacement field is replaced with the replacement field.
Step S2042, when the field in the differential data set is the newly added field, the server writes the newly added field into the target history data.
In some embodiments, the newly added field is a newly defined field in the input data, and the field with the same field name as the field is not present in the target history data. And writing the newly added field into the target historical data to update the data structure of the target historical data.
In some embodiments, the target history data in the database is retrieved based on the identity in the set of difference data corresponding to the replacement field and the newly added field is written into the target field.
In some embodiments, the newly added field is written into a corresponding field of the target history data according to the relationship of the newly added field to parent and child levels of the remaining fields in the input data. For example, the input field is song data including two fields of song information and audio data, the song information field includes four subfields of song name, singer name, version number and album to which the song information field belongs, and by comparison with the target history data, when the album field is written into the target history data, the album field is written into the target history data as a sub-field of song information in the target history data.
As shown in fig. 9, fig. 9 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention, and based on fig. 3, the data processing method of multiple data sources further includes:
Step S206, the server receives the operation data.
In some embodiments, the input data received from the multiple data sources includes blank information or error information, and the intervention is performed by manual operation, so that the error data in the error input data is modified, and missing data in the input data is supplemented. Preventing erroneous data from being written into the database when updating the history data in the database.
For example, music data acquired by one music service provider includes four fields of song name, singer name, version number, and album to which the song belongs, wherein the album to which the song belongs is Z, W, F, but the song does not actually belong to F album, and an error in the music data is caused by an operation error of a worker of the music service provider when recording song information of the song, and in order to correct the error, the server is prevented from writing the error information into a database, and it is necessary to transmit operation data to the server.
Step S207, the server corrects the difference data set based on the operation data.
In some embodiments, the server corrects the differential data set by comparing the operational data with the differential data set and based on the comparison.
For example, the operation data includes a correction field and an identity corresponding to the correction field, the difference data set includes a difference field and an identity corresponding to the difference field, where the difference field includes a replacement field and a new field. And searching a target difference field in the difference data set based on the identity identifier corresponding to the correction field, wherein the identity identifier corresponding to the target difference field is the same as the identity identifier of the correction field, and the field name of the target difference field is the same as the field name of the correction field. And when the data content of the correction field is different from the data content of the target difference field, replacing the data content of the difference field with the data content of the correction field. Storing the corrected replacement field or the corrected new field into the difference data set, and deleting the target replacement field or the target new field in the difference data set. When the target difference field does not exist, the correction field and the confirmation are the newly added fields, and the correction field is stored in the difference data set.
For example, the operation data includes a correction field and an identification of the input data described in the correction field, where the field name of the correction field is an album, and the album is Z, W. And searching a target replacement field in the difference data set based on the identity mark in the operation data, wherein the identity mark of the input data to which the target replacement field belongs is the same as the identity mark of the input data to which the correction field belongs, the field name of the target replacement field is also an album to which the correction field belongs, the album to which the correction field belongs is Z, W, F, the correction field is compared with the target replacement field, and when the data content of the correction field is different from the data content of the target replacement field, the data content of the target replacement field is replaced with the data content of the correction field so as to correct the difference data set.
In some embodiments, the server screens the data in the differential data set based on the operational data, thereby correcting the differential data set.
For example, the operation data includes a filtering condition, and data satisfying the filtering condition in the differential data set is deleted to correct the differential data set. Of course, depending on the screening conditions, data in the differential data set that does not satisfy the screening conditions may also be deleted.
For example, the music service provider finds that the input data sent at this time has information errors of the album to which the input data belongs, so that the operation data carrying screening conditions is sent to the server, the screening conditions are fields of the album to which the field names belong, after the server receives the operation data, the server deletes the fields of all the fields names of the album to which the field names belong in the difference data set, and the field of the album to which the field names belong is not updated in this update.
By receiving the operational data and correcting the fields in the differential data set based on the operational data, the server can be prevented from writing erroneous input data into the database.
In order to more clearly explain the data processing method of multiple data sources provided by the embodiment of the present invention, the following server of the music playing platform exemplarily describes a process of acquiring music data from a plurality of music service providers and processing the music data acquired by the plurality of music service providers.
The intelligent sound box is connected with a server of the music playing platform through a network, the server comprises a music library, and the music library refers to a database in which music data are stored in the server. The intelligent sound box sends the search conditions input by the user to the server of the music playing platform, so that the server can search music data meeting the search conditions in the music library. And the server sends the retrieved music data meeting the search conditions to the intelligent sound box through the network. The intelligent sound box displays information of the received music data on the display device, and plays the music data corresponding to the selection operation in response to the user triggering the selection operation.
The music playing platform needs to acquire music data from different music service providers, and also needs to periodically acquire music data from different music service providers and update the music data in the music library based on the acquired music data. A data processing device with multiple data sources is embedded in a server of the music playing platform, and the music data in the music library is updated through the data processing device.
Fig. 10 is a flow chart of a data processing method of multiple data sources according to an embodiment of the present invention, where the data processing method includes:
In step S101, the acquisition module 3551 of the multi-data-source data processing apparatus 355 receives the input music data from the first music service provider server 100-1, the second music service provider server 100-2, and the third music service provider server 100-3, respectively.
In step S102, the acquiring module 3551 maps the received music data onto a predefined data structure, and sends the mapped music data to the retrieving module 3552 of the multi-data-source data processing apparatus 355, that is, the acquiring module 2551 converts the format of each music data into music data conforming to the standard format, and sends the format-converted music data to the retrieving module 3552.
In step S103, the retrieving module 2552 retrieves the target history music data from the music library, and sends the retrieved target history music data to the comparing module 3553 of the multi-data-source data processing apparatus 355.
In some embodiments, the target historical music data is retrieved in the music library based on the identity of the input music data. The identification is a field capable of uniquely identifying the input music data, and the target historical music data and the field have the same data content.
Step S104, the comparison module 3553 compares the input music data with the target historical music data to obtain a difference data set.
Step S105, comparison module 3553 receives manual operation data from first music service provider server 100-1, second music service provider server 100-2 and third music service provider server 100-3.
In step S106, the comparing module 3553 compares the manual operation data with the difference data set to obtain a corrected difference data set, and sends the corrected difference data set to the updating module 3554 of the data processing device 355 of the multi-data source.
Step S107, the updating module 3554 updates the music data in the music library based on the corrected difference data set.
In some embodiments, as shown in fig. 11, fig. 11 is a flowchart comparing input music data, historical music data and manual operation data in the data processing method of multiple data sources according to the embodiment of the present invention, where the flowchart includes:
step S11, when there is no history music data in the music library 356, the input music data is stored in the difference data set.
In step S12, when there is history music data in the music library 356, the input music data is compared with the history music data, and difference data is stored in the difference data set, wherein the difference data is different from the data of the history music data in the input music data.
And S13, when the manual operation data does not exist, confirming the difference data set as a corrected difference data set.
S14, when the manual operation data exist, comparing the difference data set with the manual operation data to obtain a corrected difference data set.
In some embodiments, as shown in fig. 12, fig. 12 is a schematic flow chart of comparing input music data with historical music data to obtain a difference data set in the data processing method of multiple data sources according to the embodiment of the present invention, where the flow chart includes:
s121, searching target historical music data in the music library 356 based on the identity of the input music data.
S122, when a target field exists in the target historical music data, comparing the data content in the field in the input music data with the data content in the target field, wherein the field name of the target field is the same as the field name of the field in the input music data.
S123, storing fields which are different from the data content in the target fields in the input music data into a difference data set.
S124, when the target field does not exist in the target historical music data, the field of the input music data is stored in the difference data set.
In some embodiments, as shown in fig. 13, fig. 13 is a schematic frame diagram of a multi-data source data processing apparatus according to an embodiment of the present invention, where the data processing apparatus includes:
a data processing configuration layer 3556, a data access layer 3557, a data variance calculation layer 3558, and a data storage layer 3559.
The data processing configuration layer 3556 is configured to transmit preset setting data to the data access layer 3556, for example, obtain a preset data structure.
The data access layer 3557 is configured to receive currently input music data, read historical music data from the music library 356, compare the currently input music data with the historical music data to obtain a difference data set, and transmit the difference data set to the data difference calculation layer 3558.
The data difference calculation layer 3558 is configured to receive the manual operation data, compare the manual operation data with the difference data set, obtain a corrected difference data set, and transmit the corrected difference data set to the data storage layer 3559.
The data storage layer 3559 is configured to update the historical music data in the music library 356 based on the corrected difference data set, and transmit the updated historical music data to the data access layer 3557 when the data access layer 3557 receives the input music data next time.
Continuing with the description below of an exemplary architecture of the multi-data-source data processing apparatus 355 implemented as software modules provided by embodiments of the present invention, in some embodiments, as shown in fig. 2, the software modules stored in the multi-data-source data processing apparatus 355 of the memory 340 may include: an acquisition module 3551, a retrieval module 3552, a comparison module 3553, and an update module 3554.
The acquisition module 3551 is configured to receive source input data from a plurality of data sources, and convert the source input data into input data that conforms to a standard format.
And a retrieving module 3552 for retrieving, based on the identification information of the input data, the target history data corresponding to the identification information in the database.
And the comparison module 3553 is configured to compare the input data with the target history data to obtain a difference data set when the target history data corresponding to the identification information exists in the database.
In some embodiments, the obtaining module 3551 is further configured to obtain a standard format according to a format of the historical data in the database; the format of the input data is converted into a standard format.
In some embodiments, the retrieving module 3552 is further configured to identify a field of the input data as an identity, the identity being a field capable of uniquely identifying the input data; and searching the historical data with the same data content in the fields corresponding to the identification marks in the database, and confirming the searched historical data as target historical data.
In some embodiments, the comparing module 3553 is further configured to compare the field in the input data with the target field in the target history data, and form a difference data set based on the fields having differences. Wherein the field name of the target field is the same as the field name of the field in the input data.
In some embodiments, the retrieving module 3552 is further configured to retrieve a target field in the target history data based on a field name of the field in the input data.
The comparison module 3553 is further configured to, when the target field does not exist, confirm the field in the input data as a newly added field, and store the newly added field in the difference data set; when the target field exists, comparing the field in the input data with the target field, and when the field in the input data is different from the target field, confirming the field in the input data as a replacement field and storing the replacement field in the difference data set.
In some embodiments, the updating module 3554 is configured to replace a field in the target history data that corresponds to a replacement field with the replacement field when the field in the difference data set is the replacement field; when the field in the differential data set is a new field, writing the new field into the target historical data.
In some embodiments, the receiving module 3551 is further configured to receive operational data.
The comparison module 3553 is further configured to correct the differential data set based on the operation data.
Embodiments of the present invention provide a storage medium having stored therein executable instructions which, when executed by a processor, cause the processor to perform a method provided by embodiments of the present invention, for example, a multi-data source data processing method as illustrated in any of fig. 3 to 9.
In some embodiments, the storage medium may be FRAM, ROM, PROM, EPROM, EE PROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; but may be a variety of devices including one or any combination of the above memories.
In some embodiments, the executable instructions may be in the form of programs, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and they may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.
As an example, the executable instructions may, but need not, correspond to files in a file system, may be stored as part of a file that holds other programs or data, such as in one or more scripts in a hypertext markup language (html, hyper Text Markup Language) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
As an example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or, alternatively, distributed across multiple sites and interconnected by a communication network.
In summary, the embodiment of the invention has the following beneficial effects:
1) When the data in the database is updated, the input data is compared with the historical data in the database, and only the part of the input data which is different from the historical data is stored in the database, so that the time for storing the same part of the input data as the historical data in the database is saved, the part of the input data which is the same as the historical data is prevented from occupying the storage space of the database, the time consumption of data updating can be shortened, and the storage space occupied by the data is reduced.
2) When the input data is compared with the historical data, the input data is compared with the target historical data only, and the input data is not required to be compared with all the historical data, so that the time consumed by comparing the input data with the historical data is shortened, and the time consumption for updating the data is shortened.
3) When the fields in the input data are compared with the fields in the target historical data, only the fields in the input data are compared with the target fields, but not all the fields in the input data are compared with all the fields in the target historical data one by one, so that the time consumption for comparing the input data with the target historical data is shortened, and the time consumption for updating the data is further shortened.
4) By introducing operation data and correcting the difference data set based on the operation data, erroneous data in the difference data set can be removed, and the erroneous data is prevented from being written into the database.
The above is merely an example of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and scope of the present invention are included in the protection scope of the present invention.
Claims (9)
1. A method of data processing for multiple data sources, the method comprising:
receiving source input data from a plurality of data sources, converting the source input data into input data conforming to a standard format, wherein the converting the source input data into input data conforming to the standard format comprises: converting the field naming of the source input data into field naming conforming to the standard, converting the field arrangement sequence of the source input data into standard field ordering, and converting the data type of the source input data into standard data type;
searching target historical data corresponding to the identification information in a database according to the identification information of the input data, wherein the identification information refers to a plurality of fields capable of uniquely identifying the input data;
when target historical data corresponding to the identification information exists in the database, comparing the fields of the input data with the corresponding fields of the target historical data to obtain a difference data set, wherein the difference data set comprises the identification of the target historical data and the corresponding difference fields;
receiving operation data, and correcting the difference data set based on the operation data;
Updating the target historical data according to the difference data set;
and when the target historical data does not exist in the database, storing the input data into the database.
2. The method of claim 1, wherein converting the source input data into input data having a standard format comprises:
obtaining the standard format according to the format of the historical data in the database;
the format of the source input data is converted to the standard format.
3. The method according to claim 1, wherein the retrieving, in a database, the target history data corresponding to the identification information based on the identification information of the input data, comprises:
confirming a field in the input data as an identity, wherein the identity is a field capable of uniquely identifying the input data;
and retrieving the historical data with the same data content as the field in the database, and confirming the retrieved historical data as the target historical data.
4. The method of claim 1, wherein comparing the input data with the target history data results in a difference data set, comprising:
Comparing the fields in the input data with the target fields in the target historical data, and forming a difference data set based on the fields with differences;
wherein, the field name of the target field is the same as the field name of the field in the input data.
5. The method of claim 4, wherein comparing the fields in the input data with the target fields in the target history data forms a difference data set based on the fields that have differences, comprising:
retrieving the target field in the target history data based on a field name of a field in the input data;
and when the target field does not exist, confirming the field in the input data as a newly added field, and storing the newly added field into the difference data set.
6. The method of claim 5, wherein the method further comprises:
and when the target field exists, comparing the field in the input data with the target field, and when the field in the input data is different from the target field, confirming the field in the input data as a replacement field and storing the replacement field into the difference data set.
7. The method of claim 1, wherein updating the target history data from the set of difference data comprises:
when the field in the difference data set is a replacement field, replacing a field corresponding to the replacement field in the target historical data with the replacement field;
and when the field in the difference data set is a new field, writing the new field into the target historical data.
8. A data processing apparatus for multiple data sources, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for receiving source input data from a plurality of data sources and converting the source input data into input data conforming to a standard format, and the conversion of the source input data into the input data conforming to the standard format comprises the following steps: converting the field naming of the source input data into field naming conforming to the standard, converting the field arrangement sequence of the source input data into standard field ordering, and converting the data type of the source input data into standard data type;
the retrieval module is used for retrieving target historical data corresponding to the identification information in a database according to the identification information of the input data, wherein the identification information refers to a plurality of fields capable of uniquely identifying the input data;
The comparison module is used for comparing the fields of the input data with the corresponding fields of the target historical data when the target historical data corresponding to the identification information exists in the database to obtain a difference data set, wherein the difference data set comprises the identity identification of the target historical data and the corresponding difference fields;
the receiving module is used for receiving the operation data;
the comparison module is further used for correcting the difference data set based on the operation data;
the updating module is used for updating the target historical data according to the difference data set; and when the target historical data does not exist in the database, storing the input data into the database.
9. A storage medium having stored thereon executable instructions for causing a processor to perform the multi-data source data processing method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910792843.2A CN110502563B (en) | 2019-08-26 | 2019-08-26 | Data processing method and device of multiple data sources and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910792843.2A CN110502563B (en) | 2019-08-26 | 2019-08-26 | Data processing method and device of multiple data sources and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110502563A CN110502563A (en) | 2019-11-26 |
| CN110502563B true CN110502563B (en) | 2023-09-29 |
Family
ID=68589853
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910792843.2A Active CN110502563B (en) | 2019-08-26 | 2019-08-26 | Data processing method and device of multiple data sources and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110502563B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111090691B (en) * | 2019-12-19 | 2021-02-09 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113127495B (en) * | 2019-12-31 | 2024-08-23 | 深圳云天励飞技术有限公司 | Database updating method and device |
| CN113886404A (en) * | 2020-07-03 | 2022-01-04 | 北京达佳互联信息技术有限公司 | Method and device for updating data, electronic equipment and storage medium |
| CN112148953B (en) * | 2020-09-30 | 2024-06-07 | 北京金堤征信服务有限公司 | Access method and device for difference data |
| CN112784068B (en) * | 2021-01-28 | 2022-05-17 | 新华智云科技有限公司 | Method and system for managing customized catalogs of media assets |
| CN114398437A (en) * | 2022-01-04 | 2022-04-26 | 多点生活(成都)科技有限公司 | User information processing method and device, electronic equipment and computer readable medium |
| CN117591035B (en) * | 2024-01-18 | 2024-05-10 | 天津医康互联科技有限公司 | Data set processing method, device and computer readable storage medium |
| CN117591496B (en) * | 2024-01-18 | 2024-05-03 | 中核武汉核电运行技术股份有限公司 | High-reliability time sequence data transmission and storage system based on cloud edge cooperation |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101499069A (en) * | 2008-02-02 | 2009-08-05 | 中兴通讯股份有限公司 | Internal memory database file updating method and apparatus |
| CN101996361A (en) * | 2009-08-06 | 2011-03-30 | 埃森哲环球服务有限公司 | Data comparison system |
| CN106503091A (en) * | 2016-10-12 | 2017-03-15 | 济南浪潮高新科技投资发展有限公司 | A kind of implementation method of changeable data structure automatic synchronization coupling |
| CN106776635A (en) * | 2015-11-24 | 2017-05-31 | 北京国双科技有限公司 | The update method and device of knowledge database data |
| CN107665227A (en) * | 2017-04-10 | 2018-02-06 | 平安科技（深圳）有限公司 | A kind of data version control method and versions of data controller |
| CN107833637A (en) * | 2017-06-19 | 2018-03-23 | 平安医疗健康管理股份有限公司 | Medicine regular record update method, device, computer equipment and medium |
| CN107895018A (en) * | 2017-11-15 | 2018-04-10 | 广州视睿电子科技有限公司 | Database modification SQ L generation method, system, storage medium and computer device |
| CN108200220A (en) * | 2018-04-08 | 2018-06-22 | 武汉斗鱼网络科技有限公司 | A kind of method of data synchronization, server and storage medium |
| CN108241618A (en) * | 2016-12-23 | 2018-07-03 | 航天信息股份有限公司 | The method and device of database migration, the method and device of service routine migration |
| CN108829727A (en) * | 2018-05-09 | 2018-11-16 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109033186A (en) * | 2018-06-27 | 2018-12-18 | 东软集团股份有限公司 | Data consistency detection, device, storage medium and electronic equipment |
| CN109299183A (en) * | 2018-11-20 | 2019-02-01 | 北京锐安科技有限公司 | A kind of data processing method, device, terminal device and storage medium |
| CN109508355A (en) * | 2018-10-19 | 2019-03-22 | 平安科技（深圳）有限公司 | A kind of data pick-up method, system and terminal device |
| CN109690521A (en) * | 2017-12-28 | 2019-04-26 | 深圳配天智能技术研究院有限公司 | A kind of method and device of database combining |
| US10366069B1 (en) * | 2015-12-01 | 2019-07-30 | Sprint Communications Company L.P. | Systems and methods for database management and administration |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6357161B2 (en) * | 2012-10-22 | 2018-07-11 | アビニシオ テクノロジー エルエルシー | Method for characterizing a data source in a data storage system |
-
2019
- 2019-08-26 CN CN201910792843.2A patent/CN110502563B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101499069A (en) * | 2008-02-02 | 2009-08-05 | 中兴通讯股份有限公司 | Internal memory database file updating method and apparatus |
| CN101996361A (en) * | 2009-08-06 | 2011-03-30 | 埃森哲环球服务有限公司 | Data comparison system |
| CN106776635A (en) * | 2015-11-24 | 2017-05-31 | 北京国双科技有限公司 | The update method and device of knowledge database data |
| US10366069B1 (en) * | 2015-12-01 | 2019-07-30 | Sprint Communications Company L.P. | Systems and methods for database management and administration |
| CN106503091A (en) * | 2016-10-12 | 2017-03-15 | 济南浪潮高新科技投资发展有限公司 | A kind of implementation method of changeable data structure automatic synchronization coupling |
| CN108241618A (en) * | 2016-12-23 | 2018-07-03 | 航天信息股份有限公司 | The method and device of database migration, the method and device of service routine migration |
| CN107665227A (en) * | 2017-04-10 | 2018-02-06 | 平安科技（深圳）有限公司 | A kind of data version control method and versions of data controller |
| CN107833637A (en) * | 2017-06-19 | 2018-03-23 | 平安医疗健康管理股份有限公司 | Medicine regular record update method, device, computer equipment and medium |
| CN107895018A (en) * | 2017-11-15 | 2018-04-10 | 广州视睿电子科技有限公司 | Database modification SQ L generation method, system, storage medium and computer device |
| CN109690521A (en) * | 2017-12-28 | 2019-04-26 | 深圳配天智能技术研究院有限公司 | A kind of method and device of database combining |
| CN108200220A (en) * | 2018-04-08 | 2018-06-22 | 武汉斗鱼网络科技有限公司 | A kind of method of data synchronization, server and storage medium |
| CN108829727A (en) * | 2018-05-09 | 2018-11-16 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109033186A (en) * | 2018-06-27 | 2018-12-18 | 东软集团股份有限公司 | Data consistency detection, device, storage medium and electronic equipment |
| CN109508355A (en) * | 2018-10-19 | 2019-03-22 | 平安科技（深圳）有限公司 | A kind of data pick-up method, system and terminal device |
| CN109299183A (en) * | 2018-11-20 | 2019-02-01 | 北京锐安科技有限公司 | A kind of data processing method, device, terminal device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 水利普查成果与防洪工程数据库数据对接研究;杨海坤等;水利信息化;15-19 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110502563A (en) | 2019-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110502563B (en) | Data processing method and device of multiple data sources and storage medium | |
| CN107368593B (en) | Data import method and device and server | |
| US20160188318A1 (en) | Data processing for upgrading medical equipment | |
| CN111078701B (en) | Data extraction method and device based on relational database | |
| CN105740303A (en) | Improved object storage method and apparatus | |
| WO2015099961A1 (en) | Systems and methods for hosting an in-memory database | |
| CN103955361A (en) | Modular development and publishing system for automatic compiling and establishing of web front-end codes | |
| CN114064073A (en) | Software version upgrading method and device, computer equipment and storage medium | |
| CN103019795A (en) | Method and system for upgrading software basic data | |
| CN113204558B (en) | Automatic data table structure updating method and device | |
| CN108733758B (en) | Hotel static data pushing method and device, electronic equipment and readable storage medium | |
| CN102387121A (en) | Management server, image dissemination control system and image dissemination control method | |
| CN111897827B (en) | Data updating method and system for data warehouse and electronic equipment | |
| CN113946565A (en) | Method and system for supporting sub-base and sub-table migration of heterogeneous data | |
| US20050010595A1 (en) | System and method for automating an identification mechanism and type information configuration process for a real-time data feed to a database | |
| CN116701317A (en) | File storage and downloading method, device and equipment | |
| CN111147869B (en) | Video transcoding system and method based on distributed object storage | |
| CN1218246C (en) | Automatic upgrading method for business management access point in intelligent network | |
| CN116049142A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111352984B (en) | Data distribution method, device and storage medium | |
| CN116150236A (en) | Data synchronization method and device, electronic equipment and computer readable storage medium | |
| JP2010113631A (en) | Network system and management device | |
| CN110262752B (en) | Method, device and storage medium for storing streaming media data | |
| CN114741357A (en) | Data migration method, system, equipment and storage medium | |
| CN111654337B (en) | Information processing method, terminal and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |