CN117472947A

CN117472947A - Data processing method, apparatus, device, storage medium and computer program product

Info

Publication number: CN117472947A
Application number: CN202210856872.2A
Authority: CN
Inventors: 颜勇; 张秋明; 徐中杰; 杨金松; 陈君发; 余锋
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-07-20
Filing date: 2022-07-20
Publication date: 2024-01-30

Abstract

The application provides a data processing method, apparatus, device, storage medium and computer program product, the method comprising: responding to a data service request of a request object, and determining a data service type; determining a matching query module matched with the data service type; if the matching query module is a first type query module, acquiring matching medium characteristic data from a corresponding storage module and acquiring target data; returning target data to the request object; the data processing system comprises a plurality of inquiry modules, a data main memory module and a characteristic data updating module, wherein the inquiry modules are used for executing a plurality of types of data services, and the data main memory module is used for storing medium content data and medium characteristic data and generating characteristic data updating streams based on changing events of the medium characteristic data; the media characteristic data stored by the storage module is determined based on the characteristic data change stream synchronized by the query module from the data main storage module. The method and the device can support data query of multiple types of data services and can effectively improve the data query efficiency.

Description

Data processing method, apparatus, device, storage medium and computer program product

Technical Field

The present application relates to the field of computer technology, and in particular, to a data processing method, a data processing apparatus, a computer device, a computer readable storage medium, and a computer program product.

Background

With the continuous development of the internet, data of different services are rapidly increased, the service data are required to be stored, and the data writing and inquiring requirements under different service scenes are met. The requirement of a large amount of business data storage promotes the performance requirement of a content library continuously, high concurrent data query can bring huge pressure to a database system, unstable database system is caused, high time delay is easy to generate, and the data query efficiency is low.

Disclosure of Invention

The application provides a data processing method, a device, equipment, a storage medium and a computer program product, which can support data query of multiple types of data services and can effectively improve the efficiency of the data query.

In a first aspect, the present application provides a data processing method, including:

determining a data service type of the data service request in response to the data service request of the request object;

determining a matching query module matched with the data service type from a plurality of configured query modules;

If the matching query module is a query module of a first type, the matching query module is called to acquire matching medium characteristic data matched with the data service request from medium characteristic data stored in a storage module corresponding to the matching query module, and target data requested by the data service request is acquired based on the matching medium characteristic data;

returning the target data to the request object;

each inquiry module is used for executing one type of data service, and a plurality of inquiry modules correspond to a plurality of types of data services; the data processing system comprises a plurality of inquiry modules and a data storage module, wherein the inquiry modules are contained in the data processing system; the data main memory module is used for storing media content data and media characteristic data of the media and generating characteristic data change streams based on change events of the media characteristic data; the media characteristic data stored by the storage module is determined based on the characteristic data change stream synchronized by the query module from the data main storage module.

In a second aspect, the present application provides a data processing apparatus comprising:

a processing unit, configured to determine a data service type of the data service request in response to the data service request of the request object;

The processing unit is used for determining a matching query module matched with the data service type from a plurality of configured query modules; if the matching query module is a query module of a first type, the matching query module is called to acquire matching medium characteristic data matched with the data service request from medium characteristic data stored in a storage module corresponding to the matching query module, and target data requested by the data service request is acquired based on the matching medium characteristic data;

the receiving and transmitting unit is used for returning target data to the request object;

In a third aspect, the present application provides a data processing apparatus, including a processor, a communication interface, and a memory, where the processor, the communication interface, and the memory are connected to each other, and the memory stores executable program codes, and the processor is configured to invoke the executable program codes to implement a data processing method provided in the present application.

In a fifth aspect, the present application provides a computer readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to implement the data processing method provided herein.

In a sixth aspect, the present application provides a computer program product comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device implements the data processing method provided in the present application.

The data processing method is realized based on a data processing system comprising a data main memory module and a plurality of inquiry modules, wherein the data main memory module is used for storing medium data and medium characteristic data of each medium and generating characteristic data change streams based on change events of the medium characteristic data; each query module is configured to execute a type of data service, and the media characteristic data stored in the storage module corresponding to each query module is determined based on the characteristic data change stream synchronized by the query module from the data main storage module. Therefore, by constructing the data main memory module and configuring the plurality of query modules, the method adopts different storage schemes for the service data required by each data service to form a framework for separating command query responsibilities (Command Query Responsibility Segregation, CQRS), so that the data processing system can respond to the service requests of the plurality of data services, and has stable service performance under the condition of high concurrency; in addition, the query module synchronizes data through the data change stream of the data main memory module, so that the time delay of data synchronization can be reduced, and the data consistency between the data main memory module and the query module is ensured.

When the data processing method is specifically implemented, determining the data service type of the data service request of the request object, and determining a matching query module matched with the data service type from a plurality of query modules configured by a data processing system according to the data service type; if the matching query module is a first type query module, the matching query module can be called, matching media characteristic data matched with the service request is obtained from the corresponding storage module, target data requested by the data service request is obtained based on the matching media characteristic protector, and the target data is returned to the request object. Therefore, the data query method and the data query device allocate different query modules for data query aiming at different types of data service requests, so that not only can the data query of the multi-type data service be supported, but also the synchronous data query of the multi-type data service can be realized through the different query modules, and the efficiency of the data query is effectively improved.

Drawings

In order to more clearly illustrate the technical solutions of the present application or the prior art, the following description will briefly introduce the drawings that are required to be used in the embodiments or the prior art descriptions, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a scenario of a data processing system according to an embodiment of the present application;

FIG. 3a is a schematic flow chart of a data processing method according to an embodiment of the present application;

fig. 3b is a schematic diagram of a scenario of a data service request result provided in an embodiment of the present application;

FIG. 4a is a flowchart illustrating another data processing method according to an embodiment of the present disclosure;

FIG. 4b is a schematic diagram of a data writing process performed by the data main memory module according to the embodiment of the present application;

FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

For ease of understanding, the terms referred to in this application will first be described.

Cloud technology (Cloud technology) refers to a hosting technology for integrating hardware, software, network and other series resources in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. The cloud technology is a generic term of network technology, information technology, integration technology, management platform technology, application technology and the like based on cloud computing business model application, can form a resource pool, and is flexible and convenient as required. Cloud computing technology will become an important support. In this application, the background services of the data processing system require a significant amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.

The database (Date Base) can be regarded as an electronic file cabinet, which is a place for storing electronic files, and users can perform operations such as adding, inquiring, updating, deleting and the like on data in the files. A "database" is a collection of data stored together in a manner that can be shared with multiple users, with as little redundancy as possible, independent of the application. The database management system (Database Management System, DBMS) is a computer software system designed for managing databases, and generally has basic functions of storage, interception, security, backup, and the like. The database management system may classify according to the database model it supports, e.g., relational, extensible markup language (Extensible Markup Language, XML); or by the type of computer supported, e.g., server cluster, mobile phone; or by the query language used, such as structured query language (Structured Query Language, SQL); or by performance impact emphasis, such as maximum scale, highest operating speed; or other classification schemes. Regardless of the manner of classification used, some DBMSs are able to support multiple query languages across categories, for example, simultaneously.

High Concurrency (High Concurrency) is one of the factors to be considered in the design of the architecture of the internet distributed system, and generally refers to ensuring that the system can process many requests in parallel at the same time through the design. Some of the commonly used indicators of high concurrency are Response Time (Response Time), throughput (Throughput), query Per Second (QPS), number of concurrency users, etc. Wherein the response time refers to the time that the system responds to the request; throughput refers to the number of requests processed per unit time; QPS refers to the number of response requests per second; the number of concurrent users refers to the number of users that simultaneously carry normal use system functions, such as a news Application (APP), while the number of online users represents the number of concurrent users of the system to some extent.

The content library is a data support for constructing network media, and is also a key link and technology for constructing network media such as news. With the popularity of network media, content libraries as data supports are a technology focus of attack. Generally, a content library uses a physical storage medium to support data of different types or different media, so as to construct and implement a unified content service capability of full media application characteristics, and mainly focus on how to implement the construction and implementation of storage and retrieval capabilities of high-capacity, high-speed and multi-media fusion. For example, a content library constructed based on an Elastic Search (ES) server can be applied to query and Search of contents in the content library; as another example, conventional relational databases are mainly used, focusing on relational database designs.

The scene service logic facing the existing content library system is generally simpler, and a storage mode is adopted to cover the demands of network media application on writing and inquiring of data. However, in some high concurrency cases, the use of a single database may not guarantee a stable quality of service. For example, in a news APP, there may be a situation where the number of active users on a single day reaches tens of millions, in this case, the requirements on service performance of the news APP on a content library are higher, and besides the requirement of meeting the requirement of high concurrency of reading and writing of data, the requirement of efficient data retrieval, and the requirements of a large number of data analysis, data mining and the like, the requirement of scanning the data content of the whole content library is generally required. The data query efficiency, time delay and other performance requirements of the various data services on the content library are extremely high, and all requirements are difficult to realize by adopting a single storage system. For example, to realize the retrieval of multiple fields, multiple indexes are needed, which makes the data table structure too complex and difficult to achieve the higher read-write speed requirement; in addition, a large-scale database scanning is required at high frequency, so that a single storage system cannot simultaneously meet multiple complex targets such as multi-source access, complex retrieval, historical data backtracking, efficient access, off-line analysis and the like.

Different business scenes have different data query requirements, for example, a news content management background system often needs to search related content according to specified conditions, which requires a content database to have content retrieval capability; when managing a service, it is generally necessary to measure the time-consuming content provision condition of the data content stream, so as to measure the current situation of the service from a global perspective, which requires the content repository to support the data analysis capability; the recommendation index requires consumption of distributable data, which requires filtering of distributable content before data delivery, and supporting full delivery to support full construction, which requires the content master library to have streaming delivery and full construction capabilities.

The data query requirements of different business scenes are quite different, the data query requirements of the plurality of types are realized on one scene, the complexity of the system design is very high, and even if the data query requirements are realized, the high stability and the high performance are difficult to ensure.

In order to solve the above problems, the present application provides a data processing system for performing a data processing method and providing various data services, as shown in fig. 1, the data processing system includes a data hosting module 110, a query module 120 and a data production module 130, and the data hosting module 110 and the query modules 120 form a content library. The data processing system establishes a connection with the data requesting terminal 140 through a network and performs data interaction. In the data processing system, a plurality of query modules 120 may be included, and the data requesting end 140 may include one or more request objects. It should be noted that the number and configuration of the devices in the data processing system shown in fig. 1 are used for illustration, and are not limited to the embodiments of the present application.

The data production module 130 may perform content processing and feature calculation on various contents in the data processing system, such as pictures, videos, articles, etc., generate media contents and media feature data, and write the obtained media contents and media feature data into the data hosting module 110. The data production module 130 may include a feature scheduling pipeline for providing scheduling of content feature process flows so that media content, media feature data, can be written to the data hosting module in an orderly fashion. The data production module 130 may also access external data sources, and may enable multi-source access of data.

The data hosting module 110 may receive the media content data and the media characteristic data generated from the data production module 130 and may be configured to generate a characteristic data update stream based on the change event of the media characteristic data and to generate the characteristic data update stream based on the change event of the media characteristic data. The data hosting module 110 may send the generated feature data change stream to the various query modules. The configuration center in the data main memory module 110 may configure event identifiers (Identity Document, ID) for the generated data change streams, and is configured to sort out the time sequence of the occurrence of the data change streams according to the value of the event ID, so as to ensure the order of the subsequent data change streams when processing. Wherein, the data hosting module 110 may include a stream generator for generating a feature data update stream based on a change event of the media feature data.

The query module 120 may be a combination of multiple types of query modules, each for performing one type of data service, and multiple query modules corresponding to multiple types of data services. The query module may obtain the media characteristic data through the characteristic data change stream generated by the data main storage module 110, and store the obtained media characteristic data in the storage module corresponding to the query module.

The data request end 140 may send various types of service requests to the data processing system, where the service requests may carry data service types, or the data service types may be obtained by the data processing system through parsing the service requests. The data request end 140 may be an intelligent terminal, an application program such as a news APP is configured on the intelligent terminal, and can initiate a service request to a data processing system, for example, a user installs a news consultation APP in a smart phone, when the user starts the APP, the APP can initiate a service request such as top page news recommendation to the data processing system, after determining a request type of the service request, the data processing system can call a query module matched with the request type, so as to obtain target data from a storage module corresponding to the matched query module and send the target data to the news APP, and form a recommendation page on a top page of the APP for the user to review.

The data processing system provided by the embodiment of the application can be realized based on one or more of cloud technology, artificial intelligence technology and blockchain technology. For example, one or more of Cloud storage (Cloud storage), cloud Database (Cloud Database) in Cloud technology may be involved. For example, at least part of data (e.g., media content data and feature data of respective media, etc.) involved in performing the data processing method is stored in the cloud database. As another example, at least a portion of the data involved in performing the data processing method may be stored in blocks on a blockchain; in addition, the computer device performing the data processing method may be a node device in a blockchain network.

It can be seen that the data processing system shown in fig. 1 implements a framework for separating command query responsibilities (Command Query Responsibility Segregation, CQRS) by constructing various query modules for executing various data services, and the data hosting module is mainly used for storing data. CQRS is an architectural schema that enables the separation of commands that change the state of a model from queries of the model state. The Command (Command) end is responsible for data updating, and the data Query is mainly responsible for a separate Query (Query) end, and the updating and the Query are realized by different modules, so that the complexity of the system architecture implementation can be simplified.

Based on the structure of the data processing system shown in fig. 1, a scenario diagram of the data processing system is provided in an embodiment of the present application. Referring to FIG. 2, the data processing system may include, but is not limited to: one or more terminals 210, one or more servers 220. Communication connection is established between the terminal 210 and the server 220 through a wired network or a wireless network, and data interaction is performed. It should be noted that the number and the form of the apparatus shown in fig. 2 are used as examples, and are not limited to the embodiments of the present application.

In the embodiment of the present application, the terminal 210 may include, but is not limited to, smart devices such as smart phones, tablet computers, notebook computers, desktop computers, smart speakers, smart watches, vehicle terminals, smart home appliances, smart voice interaction devices, and aircrafts.

For the application, the terminal 210 may be configured to send a service request to the server 220 as a request object, and receive target data sent from the server 220. For example, the terminal 210 may be a computer for managing a news APP, through which a manager may send a service request to the server 220, e.g. statistics of the reading amount of news articles written by a news APP contractor in the first quarter, then the server 220 may perform data processing according to the service type by matching with a suitable query module, and return the matched target data to the terminal.

In this embodiment of the present application, the server 220 in the data processing system may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and basic cloud computing services such as big data and an artificial intelligence platform.

Applied to the present application, the server 220 stores media content data and media profile data of media, and generates a profile data update stream based on a change event of the media profile data. The server 220 may be further configured to receive and respond to a service request sent by a request object (may be the terminal 210), determine a data service type requested by the request object according to the service request, invoke a query module matched with the data service type, obtain matching media feature data matched with the data service request from a storage module corresponding to the matched query module, obtain target data based on the matching media feature data, and return the obtained target data to the request object.

Server 220 may be a data processing system configured as shown in fig. 1, and includes a data hosting module and a plurality of query modules. Alternatively, the server 220 may be configured with a plurality of query modules in the data processing system as shown in fig. 1, and may be configured to obtain media characteristic data based on a data change stream generated by another server configured with a data main storage module in the distributed system and store the media characteristic data in a storage module corresponding to the query module; when a query module matching the data service type is invoked, the matching query module configured in the server 220 may then obtain matching media feature data matching the data service request from the corresponding storage module, obtain target data based on the matching media feature data, and return the obtained target data to the request object.

The data processing method provided by the embodiment of the application is briefly described above, and a specific implementation manner of the data processing method is described in detail below.

The embodiments of the present application may be applied to various scenarios including, but not limited to, cloud technology, artificial intelligence, intelligent transportation, assisted driving, and the like.

It will be appreciated that in the specific embodiments of the present application, related data such as data service requests, media content, media feature data, etc., when the embodiments of the present application are applied to specific products or technologies, the related data needs to be licensed or agreed upon by the related objects, and the collection, use and processing of the related data needs to comply with the relevant laws and regulations and standards of the relevant countries and regions.

Based on the data processing system, the application provides a data processing method, wherein the application scene can be a news content library, a fused media content library and the like, and an execution subject of the method is data processing equipment. As shown in fig. 3a, taking an application scenario of a news content library as an example, the data processing method includes, but is not limited to, the following steps:

s301: in response to a data service request of a request object, a data service type of the data service request is determined.

In the application, the request object may be a terminal, such as a smart phone or a computer configured with an application program related to a data service, and the data service type of the data service request may be a news keyword search service of a news application program, etc.; the request object may also be a server, such as a background server of a news APP, and the data service type of the data service request may be analyzing the number of active users in the APP for a certain period of time, etc.

Optionally, the data service request may carry a data service type, and the data processing device may directly obtain the corresponding data service type from the data service request.

Alternatively, the data traffic types may include various types of data analysis, data retrieval, data recommendation, content review, historical data query, and so forth. Wherein, the data analysis can be to analyze which type of news articles are most popular to readers according to the data (such as reading quantity, praise quantity, etc.) of a certain latest period; the data retrieval can be to retrieve whether the illegal data exists or not under the corresponding category according to the classification (such as the category of politics, weather and the like) of the data so as to manage the illegal data; the data recommendation can be to construct a series of recommended contents according to the historical browsing record of the user; the content review may be to provide real-time content review in high concurrency to reduce latency of service responses; the historical data query may be a service that provides the user with data that refers to a node of a past time, such as viewing, in 2022, related data such as news articles published on a day in 2008, and comments at that time.

S302: a matching query module that matches the data traffic type is determined from the configured plurality of query modules.

In the application, a plurality of inquiry modules and a data main memory module are configured in the data processing system, and the data main memory module is used for storing media content data and media characteristic data of media and generating characteristic data change streams based on change events of the media characteristic data. The query module can synchronize data from the data change stream generated by the data main memory module, and different types of query modules can select a proper storage medium (such as an ES) to store the feature data obtained by synchronization in the storage module corresponding to the query module according to the functions and requirements of data query under different scenes. The query module may be a materialized read view constructed based on the data in the data hosting module, the materialized read view being a database object including a query result that is a local copy of the remote data, or a summary table for generating a summary table based on a summation of the data tables, and may be used to store the remote table based data. Each inquiry module is used for executing one type of data service, and a plurality of inquiry modules correspond to a plurality of types of data services.

In this application, media refers to data content such as pictures, articles, and videos related to news and having various presentation forms, and media content data refers to news content included in the pictures, articles, and videos, and media feature data refers to data features of the media content data, such as news release time, authors, reading amount, forwarding amount, and types of news categories.

The data change stream may be generated by the data main memory module in the form of an event log based on the media characteristic data before the change of the media and the acquired media characteristic data after the change. Each time a data change occurs to the media characteristic data stored in the data hosting module, for example, an author modifies text content in a published news article, a change Event Log (Event Log) is recorded. The change event log includes a time of recording a change, an object of which data change has occurred, feature data of which change has occurred, and a value of which feature data has been changed. The change event log is stored in an additional mode and cannot be covered, so that all change records of data can be obtained from the recorded change event log, the data processing system is enabled to have historical backtracking capability, and the recorded data of any time node can be backtracked based on all change event logs of the current record. By synchronizing the data changes in the data change stream, the synchronization delay of the data changes can be in the millisecond level and is not perceived on the actual service system, so that all the data changes can be considered as strong consistency between the data main memory module and each query module.

Optionally, to speed up the acquisition of the latest feature data of the article, we also store a snapshot, and each time the article event log is updated, the snapshot data of the updated article is also overwritten. The snapshot stores the latest feature data of the article.

In one implementation, the media feature data of the media may include static feature data and dynamic feature data, where the static feature data refers to feature data with low change frequency, such as text content of a news article, a map in the news content, and video; the dynamic feature data refers to feature data which is changed frequently, such as browsing amount of articles, playing amount of videos, number of received comments, and the like.

Because the feature data which is changed frequently and the data quantity of the change event log generated based on the feature data can cause huge storage pressure on the data main memory module, optionally, the data main memory module can comprise a main memory database and a dynamic feature unit, and the static feature data and the change event log generated based on the data change of the static feature data can be stored in the main memory database of the data main memory module; the dynamic feature unit is a storage module independent of the main memory, and the dynamic feature data and the change event log generated based on the dynamic feature data can be stored in the dynamic feature unit. Based on the method, the static characteristic data and the dynamic characteristic data are stored in an isolated mode, storage pressure is shared for the data main memory module, and expansion of main memory capacity is avoided.

In one implementation, the dynamic feature unit may be a memory unit contained in a master database. The method comprises the steps of uniformly storing characteristics in a main storage database, and constructing the self-adaptive compression capacity of the main storage database for change event logs: for the characteristic data which is changed frequently, the characteristics of a change log are recorded for each change, and the change event log is compressed through a separate background process; for feature data that is not changed frequently, compression is not performed. Based on the method, the storage performance of the main storage database can be expanded, and unified storage of the feature data is realized.

The main memory database can adopt key-value (KV) storage mode, adopts no-mode design, and supports data writing in various forms and formats. The mode logic is relatively simple, can provide high-performance data writing, is stable and reliable, solves the problem of fragmentation storage of a plurality of data, and reduces the customized development and maintenance cost caused by service isomerism.

In one implementation, when the dynamic feature unit stores the change of the dynamic feature data, the change of the dynamic feature data may be sampled according to a fixed time window, for example, the feature data is sampled every 1 hour, and the sampled changed dynamic feature data is stored in the main memory database, so that the data main memory module generates a data change stream based on the change event log. Based on this, the change of the frequently recorded feature data can be avoided, and the storage pressure of the main database can be reduced.

In one implementation, the data hosting module may generate three forms of data change streams, including incremental change streams, full-text streams, and change-contrast streams, based on pre-change media characteristic data and acquired post-change media characteristic data. Wherein the incremental change stream may include a changed media identifier (Identity Document, ID), a changed characteristic data ID, and a changed value of the characteristic data; the full text stream may include the media identifier of the media that was changed, the media content data of the media after the change, and the value of the media characteristic data; the change control stream may include a changed medium identifier, changed medium feature data, a value of the medium feature data before the change, and a value of the medium feature data after the change. Each generated data change stream has an event ID, the event ID has uniqueness, the value of the event ID increases with time, and the event ID can be generated by adopting a snowball rolling algorithm. If the event IDs of the data change streams are compared, the time sequence of the data change streams can be ordered according to the value of the event ID, so that the sequence of the data change streams in the subsequent processing process can be ensured.

In one implementation, the data consistency between the data main memory module and each query module can be ensured through the synchronization of the data change streams. Optionally, a partition (part) mechanism of the kafka message queue may be used to ensure the order of the data change stream sent by the data change stream, and after the data change stream is generated by the data hosting module, a consistent hash algorithm is used to calculate a partition identifier (part_id) based on a media identifier, where the data change stream of the same media (such as a news article) is placed into the same part, and based on an operation mechanism of the message queue, only one part is consumed by one query module. After the query module consumes the data change stream, the fixed mapping can be made on a plurality of concurrent cooperation programs based on the medium identification in the data change stream, so that the data change stream of one medium can be processed by only one cooperation program in one query module, the data change stream of the same medium can be processed in series, the processing sequence of the data change stream can be strictly ensured, the correctness of data update in the query module is ensured, and the data consistency is realized.

In the application, a plurality of query modules can be configured in the data processing system, and each query module is used for meeting the data service request under one application scene; the storage module corresponding to each query module stores the data adapting to the data service type, and can acquire the target data requested by the request object from the storage module. Each query module can be multiple and is used for executing multiple types of data services, so that the data services which can be provided by the data processing system are distributed to the multiple modules for processing, and the service capacity and service efficiency of the data processing system on various data services can be improved.

S303: and if the matching query module is a query module of a first type, calling the matching query module to acquire matching medium characteristic data matched with the data service request from the medium characteristic data stored in the storage module corresponding to the matching query module, and acquiring target data requested by the data service request based on the matching medium characteristic data.

Each inquiry module stores a plurality of media characteristic data sets in a corresponding storage module, wherein each characteristic data set comprises static characteristic data and dynamic characteristic data of a media, for example, static characteristic data such as release time, video duration and the like of a certain news video, and dynamic characteristic data such as video play quantity, video praise quantity and the like. In addition, the first type of query module refers to a query module that needs to be processed according to feature data to provide services, and may include a data analysis module, a data retrieval module, and a data recommendation module.

Alternatively, the data service type may be a data analysis service type, such as measuring the provision of news content when managing a certain service of the news APP, and measuring the current situation of the service from a global perspective; the data service request may carry a feature identifier to be analyzed (e.g., a playing amount of a news video) and a media identifier to be analyzed (e.g., a playing link of the news video). In this case, the matching query module that matches the data traffic type may be a data analysis module (or analysis view), where the static feature data stored in the storage module corresponding to the data analysis module includes a media identifier. In response to the data service request, a data analysis module can be called, and a first medium characteristic data set containing a medium identifier matched with the medium identifier to be analyzed is determined from a plurality of medium characteristic data sets stored in a storage module corresponding to the data analysis module; acquiring matching medium characteristic data matched with the characteristic identification to be analyzed from a first medium characteristic data set; and carrying out data analysis on the matched medium characteristic data to obtain an analysis result, and determining the analysis result to be the target data requested by the data service request. It should be noted that, the data stored in the storage module of the data analysis module is time-efficient, and may be set according to a specific application scenario, for example, it is set to store only data in half a year so far, and data in more than half a year may be cleared.

The storage module corresponding to the data analysis module can be an off-line data storage module such as Hive or Iceberg built based on technologies such as a large data stream computing framework (Flink), and can meet the requirements of instant analysis and off-line analysis, and meanwhile, the storage module outputs a content table to a data management team of a data processing system to build a news content data warehouse. Wherein Iceberg is an open table format for a mass data analysis scenario; hive is a data warehouse tool based on Hadoop (a distributed system infrastructure), and can store, query and analyze large-scale data stored in Hadoop, and can be used for data extraction, data conversion and data loading; the adoption of Hive technology has the advantages that: the construction cost is relatively low; the index is supported to be created, so that the data query can be optimized; different storage types can be supported, such as plain text files and files in HBase; storing the data in a relational database can greatly reduce the time to perform semantic checks during the query process.

Alternatively, the data service type may be a data retrieval service type, such as a news content management background system often needs to search for associated content according to specified conditions; the data service request may carry a search keyword, such as "2022 national basketball tournament". In this case, the matching query module that matches the data service type may be a data retrieval module (or a retrieval view), where the static feature data stored in the storage module corresponding to the data retrieval module includes a media identifier and a media tag, where the media tag may refer to a category of the media, such as sports news, entertainment news, and the like. In response to the data service request, invoking a data retrieval module, and determining a second media characteristic data set containing media labels matched with the retrieval keywords from a plurality of media characteristic data sets stored in a storage module corresponding to the data retrieval module; acquiring a target medium identifier contained in the second medium characteristic data set, wherein the target medium identifier is matched medium characteristic data matched with the data service request; and acquiring media content data corresponding to the target media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request. It should be noted that, the data stored in the storage module of the data retrieval module is time-efficient, and may be set according to a specific application scenario, for example, it is set to store only data in half a year so far, and data in more than half a year may be cleared.

The storage module corresponding to the data retrieval module can be a storage module established based on an ES database. Meanwhile, the ES is also a distributed document database, in which each field can be indexed, and the data of each field can be searched, which can be laterally expanded to hundreds of servers to store and process PB-level (ibb=1024 TB) data, and a large amount of data can be stored, searched and analyzed in a very short time. Typically as a core engine with complex search scenarios. By applying the method and the device, efficient data retrieval or data search can be realized.

Alternatively, the data service type may be a data recommendation service type, and the matching query module matched with the data service type may be a data recommendation module (or distribution view), and the static feature data stored in the storage module corresponding to the data recommendation module may include a media identifier, a media tag and a release time. In response to the data service request, a data recommendation module may be invoked to determine a media characteristic data set satisfying a filtering condition from among the plurality of media characteristic data sets based on media tags (sports, entertainment, etc.) in the plurality of media characteristic data sets stored by the storage module, and perform filtering processing on the media characteristic data set satisfying the filtering condition. The data recommendation module may also determine a recommended media identifier from the unfiltered media feature data set based on the media tag and the release time in the unfiltered media feature data set stored by the storage module, the recommended media identifier being matching media feature data (sporting event, sports star; entertainment awards, entertainment stars, etc.) that matches the data service request; and acquiring media content data corresponding to the recommended media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request. The filtering condition may refer to that the search keyword a relates to illegal content or unhealthy content. The filtering process may be to delete the media characteristic data group satisfying the filtering condition or to add a filtering flag to the media characteristic data group satisfying the filtering condition so that the media characteristic data group satisfying the filtering condition is excluded from the target data when the target data is determined.

In one implementation, the query module that matches the data traffic type may also be a query module of a second type, which may be a query module that does not require processing according to the feature data to provide service. Wherein the second type of query module may include a content review module and a historical data query module.

Optionally, the data service type may be a content reference service type, and the data service request carries the identifier of the medium to be referred to, and the query module matched with the data service request is a content reference module (or cache view). The content reference module can synchronize the media content data from the data main memory module and store the synchronized media content data into the buffer memory module corresponding to the content reference module. Responding to the data service request, calling a content consulting module, and inquiring media content data corresponding to the media identifier to be consulted from a cache module; and determining the media content data corresponding to the media identifier to be referred to, which is inquired from the buffer module, as target data requested by the data service request.

Because the caching module caches data with timeliness, a time window for caching the data can be set according to different application scenes, for example, only data in three months from now is set, and data exceeding three months from now can be cleared, so that the condition that the medium content data corresponding to the medium identifier to be referred to is not queried in the caching module exists, and at the moment, the medium content data corresponding to the medium identifier to be referred to can be obtained from the main database.

The cache module is different from the storage module, and can be a storage constructed based on Redis storage, so that high concurrent access of online content can be supported, average query delay is within 10 milliseconds, and a single node can support 10 ten thousand query QPS.

Optionally, the data service type may be a historical data query service type, and the data service request may carry a medium identifier to be queried, a feature identifier to be queried and a time to be queried, and the query module matched with the data service type is a historical data query module. Responding to the data service request, calling a historical data consulting module, and inquiring a change event log of medium characteristic data of a medium corresponding to a medium identifier to be inquired from a data main memory module; determining historical medium characteristic data matched with the characteristic identification to be queried and the time to be queried based on a change event log of the queried medium characteristic data; the historical media characteristic data is determined as target data requested by the data service request. Because the change event logs stored in the data main memory module are stored in an additional mode, the change event logs cannot be covered, and the recorded historical data of any time node can be obtained.

The historical data query module is used for carrying out on-line problem searching and has the data rollback capability. For example, the method can be used for quickly removing Dirty data (Dirty Read) written by a specified business party in a specified time window, so as to achieve quick rollback of the data and quick recovery of business service. Dirty data refers to data that is not within a given range or that is meaningless to the actual business, or that is in an illegal data format, or that has unnormalized coding and ambiguous business logic.

The query module is flexible and extensible. In one implementation manner, a new first type query module or a second type query module may also be constructed according to other service scenarios, which is not limited in this application. For example, a time sequence view may be constructed, where the time sequence view is a first type of query module, and the corresponding storage module may be configured to collect all media content data and media characteristic data of the media in a time sequence, and perform data analysis based on the data stored in the storage module, for example, perform trend analysis according to a change of a historical reading amount of a certain news article, and predict a future reading amount change trend of the news article.

S304: and returning the target data to the request object.

In the application, the request object has different query modes and format requirements on the target data, so the matching query module can adapt and convert the data format according to the data service type of the request object so as to adapt to the data format requirements of different request objects.

In one implementation, the match query module may return target data to the requesting object based on the message queue, thereby ensuring the ordering of the returned target data.

For example, if the request object is a news web page end in the computer device and the requested data service type is a data retrieval service type, after the data processing system responds to the request and determines the target data, the target data can be returned to the news web page end and displayed according to the matched data format. Referring to fig. 3b, a scenario diagram of a data service request result provided in this embodiment of the present application is shown, a user may log in a news webpage, and input a search keyword, such as "sports", in a search field of the news webpage shown as 310 in fig. 3b, and after the news webpage receives the search keyword input by the user, a corresponding data service request may be generated and sent to a data processing system, so as to finally obtain target data about the search keyword, for example, a sports related search result list shown in the drawing may include a news article list shown as 320 in fig. 3b and related videos shown as 320 in fig. 3 b. The user may click on a title of the search result of interest, such as a news title as shown at 340 in fig. 3b, to obtain specific contents of the news article; the user may also click on the play key of a news video, as shown at 350 in fig. 3b, to view the video. In addition, the user may click "view more" as shown at 360 in FIG. 3b to obtain more news content related to the search term.

Therefore, the data query method and the data query device allocate different query modules for data query aiming at different types of data service requests, so that not only can the data query of the multi-type data service be supported, but also the synchronous data query of the multi-type data service can be realized through the different query modules, and the efficiency of the data query is effectively improved.

Referring to fig. 4a, fig. 4a is a schematic flow chart of another data processing method according to an embodiment of the present application, where an execution subject of the method is a data processing apparatus. As shown in fig. 4a, taking an application scenario of a news content library as an example, the data processing method includes, but is not limited to, the following steps:

s401: the medium data content and the medium characteristic data of each medium generated by the access data production module.

The data processing module can be included in the data processing system, and can process and calculate the characteristics of various contents, such as pictures, videos, articles and the like, generate media contents and media characteristic data, and write the obtained media contents and media characteristic data into the data main memory module. The data production module can comprise a characteristic scheduling pipeline for providing scheduling of the content characteristic processing flow, so that the medium content and the medium characteristic data can be orderly written into the data main memory module.

Fig. 4b is a schematic diagram of a flow of writing data by a data hosting module according to an embodiment of the present application. The data production module may include a plurality of data writers (such as each author or news agency publishing news in the news APP), and the data hosting module may further control data admission, such as based on the security barrier logic shown in fig. 4b, and determine whether the data writers are legitimate and whether the written features are registered legitimate features according to the set security review rules (such as 470 in fig. 4 b). The data main memory module can provide synchronous and asynchronous data access modes for the data production module, and the asynchronous access is realized through a mode of kafka message queue, so that the method is suitable for mass data updating; the synchronous access is to synchronously return the writing result through the request interface. The data main memory module can subscribe to the data message from the data production module, generate an event through message analysis, inquire the corresponding snapshot (shown as 420 in fig. 4 b) from the existing snapshot table (shown as 410 in fig. 4 b) and compare the content, if the corresponding snapshot is not different from the existing data, the writing is considered as invalid update, and the writing is directly ended; if the difference exists, generating a snapshot, and recording the latest data of the current change; when writing data into the data main memory module, the data writer identification (such as the identity of news agency) needs to be carried. The data processing system configures the priority of each data writer, the data written by the data writers with low priority cannot cover the data written by the data writers with high priority, if manual intervention configuration is needed, the data writing can be performed through a manual interface as shown in the figure, and the writing priority is higher than the result of machine calculation. Therefore, when the data host module performs data writing, it determines the priority of the writer (as shown by 430 in fig. 4 b), and if the priority of the data to be written is smaller than the priority of the current data writer, it considers that the data cannot be covered and returns directly. After the priority comparison, the data hosting module updates the store based on the data and updates the event table (shown as 440 in FIG. 4 b), as well as the snapshot table (shown as 410 in FIG. 4 b). Wherein, the "offline service kafka" shown as 450 in fig. 4b means that the data hosting module can provide offline services for users according to the kafka message queue, for example, perform offline analysis by using data; the "online service kafka" shown in 460 of fig. 4b refers to a service that provides online services, such as online analysis of data, to a user using a kafka message queue, and can feed back the analysis result to the user in real time.

The main memory database is mainly responsible for data writing, is in a schema less (schema less), performs data writing in an additional mode, is used for storing and maintaining change event logs and snapshots, is only added and not deleted, has simple logic and low error probability, has multiple copies, and can ensure high reliability of data. Meanwhile, the main database can also generate a data change stream through data comparison; the query modules can synchronize the data change streams to the storage modules corresponding to the query modules for storage by subscribing the data change streams, so that the consistency of the data between the main database and the query modules is ensured. In the main database, a data snapshot of any past time point of an article can be searched through the recorded change event log; the change event records of all data in a specified time period of an article can be searched; the method can rollback the change event records of all data of a designated time period and a designated data writer, has the capability of backtracking historical data, is beneficial to efficient attribution of business, and establishes the recovery capability of quick rollback of data.

S402: in response to a data service request of a request object, a data service type of the data service request is determined.

The data service request types can comprise service types such as data analysis, data retrieval, data recommendation and the like, and can also comprise service types such as content reference, historical data query and the like.

S403: a matching query module that matches the data traffic type is determined from the configured plurality of query modules. If the matching query module is a first type query module, then S404 is executed; if the matching query module is a second type query module, S405 is executed.

The query module can synchronize data from the data change stream generated by the data main memory module, and different types of query modules can select a proper storage medium (such as an ES) to store the feature data obtained by synchronization in the storage module corresponding to the query module according to the function and the requirement of data query under different scenes. The query module may be a materialized read view constructed based on data in the data hosting module. The first type of query module refers to a query module which needs to be processed according to characteristic data so as to provide service, and can comprise a data analysis module, a data retrieval module and a data recommendation module; the second type of query module refers to a query module that does not need to be processed according to the feature data to provide a service, and may include a content review module and a history data query module. A plurality of media characteristic data sets are stored in the storage modules corresponding to the query modules of the first type, each characteristic data set comprises static characteristic data and dynamic characteristic data of one media, and the characteristic data change stream is determined based on the characteristic data change stream synchronized by the query modules from the data main storage module.

S404: and calling a first type of matching query module to acquire matching medium characteristic data matched with the data service request from the medium characteristic data stored in the storage module corresponding to the matching query module.

Optionally, if the data service type may be a data analysis service type, the matching query module that matches the data service type may be a data analysis module, where the static feature data stored in the storage module corresponding to the data analysis module includes a media identifier. In response to the data service request, a data analysis module can be called, and a first medium characteristic data set containing a medium identifier matched with the medium identifier to be analyzed is determined from a plurality of medium characteristic data sets stored in a storage module corresponding to the data analysis module; acquiring matching medium characteristic data matched with the characteristic identification to be analyzed from a first medium characteristic data set; and carrying out data analysis on the matched medium characteristic data to obtain an analysis result, and determining the analysis result to be the target data requested by the data service request.

Optionally, the data service type may be a data retrieval service type, the matching query module matched with the data service type may be a data retrieval module, and the static feature data stored in the storage module corresponding to the data retrieval module includes a media identifier and a media tag, where the media tag may refer to a category of the media, such as sports news, entertainment news, and the like. In response to the data service request, invoking a data retrieval module, and determining a second media characteristic data set containing media labels matched with the retrieval keywords from a plurality of media characteristic data sets stored in a storage module corresponding to the data retrieval module; acquiring a target medium identifier contained in the second medium characteristic data set, wherein the target medium identifier is matched medium characteristic data matched with the data service request; and acquiring media content data corresponding to the target media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

Optionally, the data service type may be a data recommendation service type, the matching query module matched with the data service type may be a data recommendation module, and the static feature data stored in the storage module corresponding to the data recommendation module may include a media identifier, a media tag and a release time. In response to the data service request, a data recommendation module may be invoked to determine a media characteristic data set satisfying a filtering condition from among the plurality of media characteristic data sets based on media tags (sports, entertainment, etc.) in the plurality of media characteristic data sets stored by the storage module, and perform filtering processing on the media characteristic data set satisfying the filtering condition. The data recommendation module may also determine a recommended media identifier from the unfiltered media feature data set based on the media tag and the release time in the unfiltered media feature data set stored by the storage module, the recommended media identifier being matching media feature data (sporting event, sports star; entertainment awards, entertainment stars, etc.) that matches the data service request; and acquiring media content data corresponding to the recommended media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request. The filtering condition may refer to that the search keyword a relates to illegal content or unhealthy content. The filtering process may be to delete the media characteristic data group satisfying the filtering condition or to add a filtering flag to the media characteristic data group satisfying the filtering condition so that the media characteristic data group satisfying the filtering condition is excluded from the target data when the target data is determined.

S405: and calling a second type of matching query module to acquire matching medium characteristic data matched with the data service request from the medium characteristic data stored in the storage module corresponding to the matching query module.

Optionally, the data service type may be a content reference service type, and the data service request carries a medium identifier to be referred, and the query module matched with the data service request is a content reference module. The content reference module can synchronize the media content data from the data main memory module and store the synchronized media content data into the buffer memory module corresponding to the content reference module. Responding to the data service request, calling a content consulting module, and inquiring media content data corresponding to the media identifier to be consulted from a cache module; and determining the media content data corresponding to the media identifier to be referred to, which is inquired from the buffer module, as target data requested by the data service request.

Because the caching module caches data with timeliness, a time window for caching the data can be set according to different application scenes, for example, only data in three months from now is set, and data exceeding three months from now can be cleared, so that the condition that the medium content data corresponding to the medium identifier to be referred to is not queried in the caching module exists, and at the moment, the medium content data corresponding to the medium identifier to be referred to can be obtained from the main database. The cache module is different from the storage module, and can be a storage constructed based on Redis storage, so that high concurrent access of online content can be supported, average query delay is within 10 milliseconds, and a single node can support 10 ten thousand query QPS.

The historical data query module is used for carrying out on-line problem searching and has the data rollback capability. For example, a data processing system often has abnormal service performance at a certain time point, and when the reason analysis is performed, the current site needs to be restored, because the actual service scene is frequently updated, if only the latest data is stored, the site is destroyed, and the current site cannot be restored, so that the requirement of backtracking cannot be met; meanwhile, because many data writers in the data production module are involved, under the background of rapid business iteration, the data writers are not prevented from writing dirty data into the data main memory module due to business program errors, and the data main memory is required to have data rollback capability.

In one implementation, the query module is flexibly expandable. A new first type of query module or a second type of query module may also be constructed according to other service scenarios, which is not limited in this application. For example, a time sequence view may be constructed, where the time sequence view is a first type of query module, and the corresponding storage module may be configured to collect all media content data and media characteristic data of the media in a time sequence, and perform data analysis based on the data stored in the storage module, for example, perform trend analysis according to a change of a historical reading amount of a certain news article, and predict a future reading amount change trend of the news article.

S406: and returning the target data to the request object.

For example, if the request object is a smart phone installed with a news APP, and the requested data service type is a history data query, the data processing system may respond to the request and determine target data, and then return the target data to the display interface of the news APP. For example, a user can search the content such as a certain news article of any time node through the news APP, such as the text content, the map configuration of the news article, the reading quantity of the time node, the reader comment and other characteristic data.

It should be noted that, based on the same inventive concept, the technical details and principles of the data processing method in the above manner may be referred to the technical details and principles in S301-S304, and are not described herein again for brevity.

The method and the device can determine the data service type of the data service request of the request object, and determine a matching query module matched with the data service type from a plurality of query modules configured by the data processing system according to the data service type; if the matching query module is a first type query module, the matching query module can be called, matching media characteristic data matched with the service request is obtained from the corresponding storage module, target data requested by the data service request is obtained based on the matching media characteristic protector, and the target data is returned to the request object. Therefore, the data query method and the data query device allocate different query modules for data query aiming at different types of data service requests, so that not only can the data query of the multi-type data service be supported, but also the synchronous data query of the multi-type data service can be realized through the different query modules, and the efficiency of the data query is effectively improved.

Referring to fig. 5, fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 5, the data processing apparatus includes a processing unit 510 and a transceiver unit 520, wherein:

a processing unit 510, configured to determine a data service type of the data service request in response to the data service request of the request object;

A processing unit 510, configured to determine a matching query module that matches the data service type from the configured plurality of query modules; if the matching query module is a query module of a first type, the matching query module is called to acquire matching medium characteristic data matched with the data service request from medium characteristic data stored in a storage module corresponding to the matching query module, and target data requested by the data service request is acquired based on the matching medium characteristic data;

a transceiver unit 520 for returning the target data to the request object;

In one implementation, a storage module stores a plurality of media characteristic data sets, each media characteristic data set including static characteristic data and dynamic characteristic data of one media; the static characteristic data comprises a medium identifier; the data service type is a data analysis service type, the data service request carries a feature identifier to be analyzed and a medium identifier to be analyzed, the matching query module is a data analysis module for executing data analysis, and the data analysis module is a query module of a first type; the processing unit 510 is further configured to invoke a data analysis module, and determine, from a plurality of media characteristic data sets stored in a storage module corresponding to the data analysis module, a first media characteristic data set in which a contained media identifier matches a media identifier to be analyzed; acquiring matching medium characteristic data matched with the characteristic identification to be analyzed from a first medium characteristic data set; and carrying out data analysis on the matched medium characteristic data to obtain an analysis result, and determining the analysis result to be the target data requested by the data service request.

In one implementation, a storage module stores a plurality of media characteristic data sets, each media characteristic data set including static characteristic data and dynamic characteristic data of one media; the static characteristic data comprises a medium identifier and a medium label; the data service type is a data retrieval service type, the data service request carries a retrieval keyword, the matching query module is a data retrieval module for executing data retrieval, and the data retrieval module is a query module of a first type; the processing unit 510 is further configured to invoke the data retrieval module, and determine, from the plurality of media feature data sets stored in the storage module corresponding to the data retrieval module, a second media feature data set in which the media tag is matched with the retrieval keyword; acquiring target medium identifiers contained in the second medium characteristic data set, wherein the target medium identifiers are matched medium characteristic data matched with the data service request; and acquiring media content data corresponding to the target media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

In one implementation, a storage module stores a plurality of media characteristic data sets, each media characteristic data set including static characteristic data and dynamic characteristic data of one media; the static characteristic data comprises a medium identifier, a medium label and release time; the data service type is a data recommendation service type, the matching query module is a data recommendation module for executing data recommendation, and the data recommendation module is a query module of a first type; the processing unit 510 is further configured to invoke a data recommendation module, determine a media characteristic data set that satisfies a filtering condition from the plurality of media characteristic data sets based on the media tags in the plurality of media characteristic data sets stored by the storage module, and perform filtering processing on the media characteristic data set that satisfies the filtering condition; and the processing unit 510 is further configured to invoke a data recommendation module, determine a recommended media identifier from the unfiltered media feature data set based on the media tag and the release time in the unfiltered media feature data set stored by the storage module, where the recommended media identifier is matching media feature data that matches the data service request; and acquiring media content data corresponding to the recommended media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

In one implementation, the data service type is a content reference service type, and the data service request carries a medium identifier to be referred; the matching query module is a content query module for executing content query, the content query module is a second type query module, and the content query module is used for synchronizing media content data from the data main memory module and storing the synchronized media content data into a cache module corresponding to the content query module; the processing unit 510 is further configured to invoke a content review module, and query media content data corresponding to the media identifier to be reviewed from the cache module; and determining the media content data corresponding to the media identifier to be referred to, which is inquired from the buffer module, as target data requested by the data service request.

In one implementation, the data main memory module is further configured to store a change event log of the media characteristic data of each media, where the change event log is stored in an additional manner; the data service type is a historical data query service type, the data service request carries a medium identifier to be queried, a feature identifier to be queried and time to be queried, the matching query module is a historical data query module for executing historical data query, and the historical data query module is a query module of a second type; the processing unit 510 is further configured to invoke a historical data review module to query a change event log of media characteristic data of the media corresponding to the media identifier to be queried from the data main memory module; determining historical medium characteristic data matched with the characteristic identification to be queried and the time to be queried based on a change event log of the queried medium characteristic data; the historical media characteristic data is determined as target data requested by the data service request.

In one implementation, the data main memory module is used for storing static characteristic data and dynamic characteristic data of the medium, and the static characteristic data of the medium is stored in a main memory database of the data main memory module; the data main memory module also comprises a dynamic feature unit, wherein the dynamic feature unit is used for sampling the change of the dynamic feature data of the medium and storing the changed dynamic feature data obtained by sampling into a main memory database; the feature data change stream is generated based on the pre-change media feature data of the media and the acquired post-change media feature data.

According to one embodiment of the present application, the steps involved in the data processing method shown in fig. 3a and 4a may be performed by respective modules in the data processing apparatus shown in fig. 5. For example, step S301 shown in fig. 3a, and step S402 shown in fig. 4a may be performed by the processing unit 510 as shown in fig. 5; steps S302, S303 shown in fig. 3a and steps S401, S403-S405 shown in fig. 4a may be performed by the processing unit 510 as shown in fig. 5; step S304 shown in fig. 3a and step S406 shown in fig. 4a may be performed by the transceiving unit 520 as shown in fig. 5.

According to an embodiment of the present application, each module in the data processing apparatus shown in fig. 5 may be separately or all combined into one or several units to form a structure, or some (some) of the units may be further split into multiple sub-units with smaller functions, so that the same operation may be implemented without affecting the implementation of the technical effects of the embodiments of the present application. The above modules are divided based on logic functions, and in practical applications, the functions of one module may be implemented by a plurality of units, or the functions of a plurality of modules may be implemented by one unit. In other embodiments of the present application, the data processing apparatus may also include other units, and in practical applications, these functions may also be implemented with assistance from other units, and may be implemented by cooperation of a plurality of units.

It may be understood that the functions of each functional unit of the data processing apparatus described in the embodiments of the present application may be specifically implemented according to the method in the foregoing method embodiments, and the specific implementation process may refer to the relevant description of the foregoing method embodiments, which is not repeated herein.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a computer device according to an embodiment of the present application. As shown in fig. 6, the computer device described in the embodiment of the present application is configured to perform the data processing method described above, and includes: processor 610, communication interface 620, and memory 630. The processor 610, the communication interface 620, and the memory 630 may be connected by a bus or other means, which is exemplified in the present embodiment.

Among them, the processor 610 (or CPU (Central Processing Unit, central processing unit)) is a computing core and a control core of a computer device, which can parse various instructions in the computer device and process various data of the computer device, for example: the CPU can be used for analyzing the switching-on and switching-off instruction sent to the computer equipment and controlling the computer equipment to perform switching-on and switching-off operation; and the following steps: the CPU may transmit various types of interaction data between internal structures of the computer device, and so on. Communication interface 620 may optionally include a standard wired interface, a wireless interface (e.g., wi-Fi, mobile communication interface, etc.), controlled by processor 610 for transceiving data. Memory 630 (Memory) is a Memory device in a computer device for storing programs and data. It is understood that the memory 630 herein may include built-in memory of the computer device or may include extended memory supported by the computer device. Memory 630 provides storage space that stores the operating system of the computer device, which may include, but is not limited to: android systems, iOS systems, windows Phone systems, etc., which are not limiting in this application.

In the present embodiment, the processor 610 performs the following operations by executing executable program code in the memory 630:

determining a matching query module matched with the data service type from a plurality of configured query modules; if the matching query module is a query module of a first type, the matching query module is called to acquire matching medium characteristic data matched with the data service request from medium characteristic data stored in a storage module corresponding to the matching query module, and target data requested by the data service request is acquired based on the matching medium characteristic data;

returning target data to the request object;

In one implementation, a storage module stores a plurality of media characteristic data sets, each media characteristic data set including static characteristic data and dynamic characteristic data of one media; the static characteristic data comprises a medium identifier; the data service type is a data analysis service type, the data service request carries a feature identifier to be analyzed and a medium identifier to be analyzed, the matching query module is a data analysis module for executing data analysis, and the data analysis module is a query module of a first type; the processor 610, by executing executable program code in the memory 630, may also perform the following operations:

invoking a data analysis module, and determining a first medium characteristic data set, which is matched with a medium identifier to be analyzed, from a plurality of medium characteristic data sets stored in a storage module corresponding to the data analysis module; acquiring matching medium characteristic data matched with the characteristic identification to be analyzed from a first medium characteristic data set; and carrying out data analysis on the matched medium characteristic data to obtain an analysis result, and determining the analysis result to be the target data requested by the data service request.

In one implementation, a storage module stores a plurality of media characteristic data sets, each media characteristic data set including static characteristic data and dynamic characteristic data of one media; the static characteristic data comprises a medium identifier and a medium label; the data service type is a data retrieval service type, the data service request carries a retrieval keyword, the matching query module is a data retrieval module for executing data retrieval, and the data retrieval module is a query module of a first type; the processor 610, by executing executable program code in the memory 630, may also perform the following operations:

Invoking a data retrieval module, and determining a second medium characteristic data set with the contained medium label matched with the retrieval keyword from a plurality of medium characteristic data sets stored in a storage module corresponding to the data retrieval module; acquiring target medium identifiers contained in the second medium characteristic data set, wherein the target medium identifiers are matched medium characteristic data matched with the data service request; and acquiring media content data corresponding to the target media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

In one implementation, a storage module stores a plurality of media characteristic data sets, each media characteristic data set including static characteristic data and dynamic characteristic data of one media; the static characteristic data comprises a medium identifier, a medium label and release time; the data service type is a data recommendation service type, the matching query module is a data recommendation module for executing data recommendation, and the data recommendation module is a query module of a first type; the processor 610, by executing executable program code in the memory 630, may also perform the following operations:

based on the media labels in the plurality of media characteristic data sets stored by the storage module, determining the media characteristic data sets meeting the filtering conditions in the plurality of media characteristic data sets, and filtering the media characteristic data sets meeting the filtering conditions; and calling a data recommendation module, determining a recommended media identifier from the unfiltered media characteristic data set based on the media tag and the release time in the unfiltered media characteristic data set stored by the storage module, wherein the recommended media identifier is matched media characteristic data matched with the data service request; and acquiring media content data corresponding to the recommended media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

In one implementation, the data service type is a content reference service type, and the data service request carries a medium identifier to be referred; the matching query module is a content query module for executing content query, the content query module is a second type query module, and the content query module is used for synchronizing media content data from the data main memory module and storing the synchronized media content data into a cache module corresponding to the content query module; the processor 610, by executing executable program code in the memory 630, may also perform the following operations:

calling a content consulting module, and inquiring media content data corresponding to the media identifier to be consulted from the cache module; and determining the media content data corresponding to the media identifier to be referred to, which is inquired from the buffer module, as target data requested by the data service request.

In one implementation, the data main memory module is further configured to store a change event log of the media characteristic data of each media, where the change event log is stored in an additional manner; the data service type is a historical data query service type, the data service request carries a medium identifier to be queried, a feature identifier to be queried and time to be queried, the matching query module is a historical data query module for executing historical data query, and the historical data query module is a query module of a second type; the processor 610, by executing the executable program code in the memory 630, may also perform the following operations:

Calling a historical data consulting module, and inquiring a change event log of medium characteristic data of a medium corresponding to the medium identifier to be inquired from a data main memory module; determining historical medium characteristic data matched with the characteristic identification to be queried and the time to be queried based on a change event log of the queried medium characteristic data; the historical media characteristic data is determined as target data requested by the data service request.

The present application further provides a computer readable storage medium, in which a computer program executed by the aforementioned data processing apparatus is stored, and the computer program includes program instructions, when executed by a processor, can perform the description of the data processing method in the embodiment corresponding to fig. 3a and fig. 4a, and therefore, a detailed description will not be given here. In addition, the description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiments of the computer storage medium related to the present application, please refer to the description of the method embodiments of the present application.

As an example, the above-described program instructions may be executed on one computer device or on a plurality of computer devices disposed at one site, or alternatively, on a plurality of computer devices distributed at a plurality of sites and interconnected by a communication network, which may constitute a blockchain network.

The computer readable storage medium may be the data processing apparatus provided in any one of the foregoing embodiments or an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (flash card) or the like, which are provided on the computer device. Further, the computer-readable storage medium may also include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

The present application provides a computer program product comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above description of the data processing method in the corresponding embodiment of fig. 3a and fig. 4a, and therefore, a detailed description will not be given here. In addition, the description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiments of the computer-readable storage medium according to the present application, please refer to the description of the method embodiments of the present application.

The terms first, second and the like in the description and in the claims and drawings of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the term "include" and any variations thereof is intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, article, or device that comprises a list of steps or elements is not limited to the list of steps or modules but may, in the alternative, include other steps or modules not listed or inherent to such process, method, apparatus, article, or device.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The methods and related devices provided in the embodiments of the present application are described with reference to the method flowcharts and/or structure diagrams provided in the embodiments of the present application, and each flowchart and/or block of the method flowcharts and/or structure diagrams may be implemented by computer program instructions, and combinations of flowcharts and/or blocks in the flowchart and/or block diagrams. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or structural diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or structures.

The foregoing disclosure is only illustrative of the preferred embodiments of the present application and is not intended to limit the scope of the claims herein, as the equivalent of the claims herein shall be construed to fall within the scope of the claims herein.

Claims

1. A method of data processing, the method comprising:

determining a data service type of a data service request in response to the data service request of a request object;

if the matching query module is a query module of a first type, calling the matching query module to acquire matching media characteristic data matched with the data service request from media characteristic data stored in a storage module corresponding to the matching query module, and acquiring target data requested by the data service request based on the matching media characteristic data;

returning the target data to the request object;

each inquiry module is used for executing one type of data service, and the plurality of inquiry modules correspond to a plurality of types of data services; the plurality of inquiry modules are contained in a data processing system, and the data processing system further comprises a data main memory module; the data main memory module is used for storing media content data and media characteristic data of media and generating characteristic data change streams based on change events of the media characteristic data; the media characteristic data stored by the storage module is determined based on the characteristic data change stream synchronized by the query module from the data hosting module.

2. The method of claim 1, wherein the storage module stores a plurality of media characteristic data sets, each media characteristic data set comprising static characteristic data and dynamic characteristic data of one media, the static characteristic data comprising a media identification; the data service type is a data analysis service type, the data service request carries a feature identifier to be analyzed and a medium identifier to be analyzed, the matching query module is a data analysis module for executing data analysis, and the data analysis module is a query module of the first type;

the step of calling the matching query module to acquire matching media characteristic data matched with the data service request from the media characteristic data stored in the storage module corresponding to the matching query module, and acquiring target data requested by the data service request based on the matching media characteristic data comprises the following steps:

invoking the data analysis module, and determining a first medium characteristic data set with the contained medium identifier matched with the medium identifier to be analyzed from a plurality of medium characteristic data sets stored in a storage module corresponding to the data analysis module;

Acquiring matching medium characteristic data matched with the characteristic identification to be analyzed from the first medium characteristic data set;

and carrying out data analysis on the matched medium characteristic data to obtain an analysis result, and determining the analysis result as target data requested by the data service request.

3. The method of claim 1, wherein the storage module stores a plurality of media characteristic data sets, each media characteristic data set comprising static characteristic data and dynamic characteristic data of one media, the static characteristic data comprising a media identifier and a media tag; the data service type is a data retrieval service type, the data service request carries a retrieval keyword, the matching query module is a data retrieval module for executing data retrieval, and the data retrieval module is a query module of the first type;

Invoking the data retrieval module, and determining a second medium characteristic data set with the contained medium label matched with the retrieval keyword from a plurality of medium characteristic data sets stored in a storage module corresponding to the data retrieval module;

acquiring a target medium identifier contained in the second medium characteristic data set, wherein the target medium identifier is matched medium characteristic data matched with the data service request;

and acquiring the media content data corresponding to the target media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

4. The method of claim 1, wherein the storage module stores a plurality of media characteristic data sets, each media characteristic data set comprising static characteristic data and dynamic characteristic data of one media, the static characteristic data comprising a media identifier, a media tag, and a publication time; the data service type is a data recommendation service type, the matching query module is a data recommendation module for executing data recommendation, and the data recommendation module is a query module of the first type; the method further comprises the steps of:

Invoking the data recommendation module to determine a media characteristic data set satisfying a filtering condition among the plurality of media characteristic data sets based on media tags among the plurality of media characteristic data sets stored by the storage module,

filtering the medium characteristic data set meeting the filtering condition;

invoking the data recommendation module, and determining a recommended media identifier from the unfiltered media characteristic data set based on the media tag and the release time in the unfiltered media characteristic data set stored by the storage module, wherein the recommended media identifier is matched media characteristic data matched with the data service request;

and acquiring media content data corresponding to the recommended media identifier from the data main memory module, and determining the acquired media content data as target data requested by the data service request.

5. The method of claim 1, wherein the data service type is a content review service type, and the data service request carries a media identifier to be reviewed; the matching query module is a content query module for executing content query, the content query module is a second type query module, and the content query module is used for synchronizing media content data from the data main memory module and storing the synchronized media content data into a cache module corresponding to the content query module; the method further comprises the steps of:

invoking the content consulting module, and inquiring media content data corresponding to the media identifier to be consulted from the cache module;

and determining the media content data corresponding to the media identifier to be referred to, which is inquired from the buffer module, as target data requested by the data service request.

6. The method of claim 1, wherein the data hosting module is further configured to store a change event log of media characteristic data for each media, the change event log being stored in an append manner; the data service type is a historical data query service type, the data service request carries a medium identifier to be queried, a feature identifier to be queried and time to be queried, the matching query module is a historical data query module for executing historical data query, and the historical data query module is a query module of a second type; the method further comprises the steps of:

Invoking the historical data consulting module, and inquiring a change event log of the medium characteristic data of the medium corresponding to the medium identifier to be inquired from the data main memory module;

determining historical medium characteristic data matched with the characteristic identification to be queried and the time to be queried based on a change event log of the queried medium characteristic data;

and determining the historical media characteristic data as target data requested by the data service request.

7. The method of any of claims 1-6, wherein the data hosting module is configured to store static feature data and dynamic feature data of a medium, the static feature data of the medium being stored in a hosting database of the data hosting module; the data main memory module also comprises a dynamic feature unit, wherein the dynamic feature unit is used for sampling the change of the dynamic feature data of the medium and storing the changed dynamic feature data obtained by sampling into the main memory database; the feature data change stream is generated based on the pre-change media feature data of the media and the acquired post-change media feature data.

8. A data processing apparatus, comprising:

A processing unit, configured to determine a data service type of a data service request in response to the data service request of a request object;

the processing unit is used for determining a matching query module matched with the data service type from a plurality of configured query modules; if the matching query module is a query module of a first type, calling the matching query module to acquire matching media characteristic data matched with the data service request from media characteristic data stored in a storage module corresponding to the matching query module, and acquiring target data requested by the data service request based on the matching media characteristic data;

the receiving and transmitting unit is used for returning the target data to the request object;

9. A computer device comprising a processor, a communication interface and a memory, the processor, the communication interface and the memory being interconnected, wherein the memory stores executable program code, the processor being adapted to invoke the executable program code to implement the data processing method of any of claims 1-7.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a computer program for implementing the data processing method according to any of claims 1-7 when being executed by a processor.

11. A computer program product, characterized in that the computer program product comprises a computer program stored in a computer storage medium, which computer program, when being executed by a processor, is adapted to carry out the data processing method according to any one of claims 1-7.