CN110673962A - Content stream processing method, device, equipment and medium - Google Patents

Content stream processing method, device, equipment and medium Download PDF

Info

Publication number
CN110673962A
CN110673962A CN201910794944.3A CN201910794944A CN110673962A CN 110673962 A CN110673962 A CN 110673962A CN 201910794944 A CN201910794944 A CN 201910794944A CN 110673962 A CN110673962 A CN 110673962A
Authority
CN
China
Prior art keywords
service
micro
content object
target content
topology
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910794944.3A
Other languages
Chinese (zh)
Other versions
CN110673962B (en
Inventor
袁易之
程楠
姚方杰
朱新宇
司超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201910794944.3A priority Critical patent/CN110673962B/en
Publication of CN110673962A publication Critical patent/CN110673962A/en
Application granted granted Critical
Publication of CN110673962B publication Critical patent/CN110673962B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/54Indexing scheme relating to G06F9/54
    • G06F2209/548Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a content stream processing method, a device, equipment and a medium, wherein the method comprises the steps of obtaining original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space; acquiring a target content object in the storage space; acquiring at least one service associated with the target content object, and acquiring a micro-service call topology corresponding to the service according to the service to obtain a micro-service call topology set corresponding to the target content object; concurrently calling related micro-services to perform stream processing on the target content object according to each micro-service calling topology in the micro-service calling topology set, and storing calling associated information generated in the stream processing process into the target content object; and obtaining the service output object associated with the target content object from the target content object. The invention can simultaneously meet the flow processing requirements of a plurality of services to obtain the flow processing result required by each service.

Description

Content stream processing method, device, equipment and medium
Technical Field
The present invention relates to the field of stream processing, and in particular, to a method, an apparatus, a device, and a medium for processing a content stream.
Background
The distributed content stream processing aims at continuously processing the borderless content set, and even if a single point fails, the whole content processing process is not influenced, so that continuous service is provided for the outside. In the existing technical scheme related to distributed content stream processing, the execution process of each processing link of stream processing has a strong coupling relation with the specific service pointed by the stream processing, so that the dynamic change of each link in the stream processing may cause the change of service logic, and vice versa, thereby causing the difficulty in updating and expanding the technical scheme related to distributed content stream processing, restricting the updating and expanding of the service logic, and reducing the stability of the service.
Disclosure of Invention
In order to solve the technical problem in the prior art that each processing link of stream processing has a strong coupling relationship with a specific service pointed by stream processing, embodiments of the present invention provide a content stream processing method, apparatus, device, and medium.
In one aspect, the present invention provides a content stream processing method, including:
acquiring original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space;
acquiring a target content object in the storage space;
acquiring at least one service associated with the target content object, and acquiring a micro-service call topology corresponding to the service according to the service to obtain a micro-service call topology set corresponding to the target content object;
concurrently calling related micro-services to perform stream processing on the target content object according to each micro-service calling topology in the micro-service calling topology set, and storing calling associated information generated in the stream processing process into the target content object;
and obtaining the service output object corresponding to each service associated with the target content object from the target content object.
In another aspect, the present invention provides a content stream processing apparatus, including:
the original content processing module is used for acquiring original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space;
the target content object module is used for acquiring a target content object in the storage space;
a micro-service call topology set acquisition module, configured to acquire at least one service associated with the target content object, and acquire a corresponding micro-service call topology according to the service, so as to obtain a micro-service call topology set corresponding to the target content object;
the micro-service calling module is used for calling related micro-services to perform stream processing on the target content object according to each micro-service calling topology in the micro-service calling topology set and storing calling associated information generated in the stream processing process into the target content object;
and the service output object acquisition module is used for acquiring the service output object corresponding to each service associated with the target content object from the target content object.
In another aspect, the present invention provides an apparatus comprising a processor and a memory, wherein the memory has stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement a content stream processing method.
In another aspect, the present invention provides a computer storage medium, wherein at least one instruction, at least one program, set of codes, or set of instructions is stored in the storage medium, and the at least one instruction, at least one program, set of codes, or set of instructions is loaded by a processor and executes a content stream processing method.
The invention provides a content stream processing method, a content stream processing device, content stream processing equipment and a content stream processing medium. The invention encapsulates each processing link involved in the stream processing process to obtain the micro-service executing the atomic processing function, thereby realizing the atomization of the stream processing. The micro-service calling topology is generated for the business related to the stream processing, and the micro-service is called according to the micro-service calling topology to realize the decoupling of the business and the stream processing, so that the stream processing method can simultaneously meet the stream processing requirements of a plurality of businesses to obtain the stream processing result required by each business.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of an implementation environment of a content stream processing method provided by the present invention;
FIG. 2 is a flow chart of a content stream processing method provided by the present invention;
FIG. 3 is a flow chart of generating a content object corresponding to the original content according to the present invention;
FIG. 4 is a flow chart of obtaining a target content object in the storage space according to the present invention;
FIG. 5 is another flow chart of the present invention for retrieving a target content object in the storage space;
FIG. 6 is a flowchart of the present invention for constructing a micro service invocation topology corresponding to each service;
FIG. 7 is a schematic diagram of the execution logic of a micro service invocation topology in a set of micro service invocation topologies provided by the present invention;
FIG. 8 is a flowchart for concurrently invoking related micro-services to stream process the target content object according to each micro-service invocation topology in the micro-service invocation topology set and storing invocation correlation information generated in the stream process into the target content object, according to the present invention;
FIG. 9 is a flowchart for each scheduling service to invoke micro-services in sequence according to its corresponding micro-service invocation topology provided by the present invention;
fig. 10 is a schematic diagram of a content stream processing method applied to a video processing scene according to the present invention;
FIG. 11 is a schematic diagram illustrating the implementation of a stream processing method according to the present invention;
FIG. 12 is a block diagram of a content stream processing apparatus according to the present invention;
fig. 13 is a hardware structural diagram of an apparatus for implementing the method provided by the embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In order to make the objects, technical solutions and advantages disclosed in the embodiments of the present invention more clearly apparent, the embodiments of the present invention are described in further detail below with reference to the accompanying drawings and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the invention and are not intended to limit the embodiments of the invention.
In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present embodiment, "a plurality" means two or more unless otherwise specified.
In order to clearly explain the technical solutions described in the embodiments of the present invention, the embodiments of the present invention first explain related terms, technical background, and technical environment:
stream processing: stream processing is an important big data processing means, and the main characteristic is that the processed data comes continuously and in real time. Distributed stream processing is a fine-grained processing mode oriented to dynamic data, and continuously generated dynamic data are processed based on a distributed memory. The method plays an increasingly important role in large data processing due to the characteristics of high speed, high efficiency, low delay and the like of the data processing.
And (3) message transmission guarantee: there are generally three types of message transmission guarantees: at most once, At least once and exact once. The message transmission mechanism of the At most once is that each message is transmitted for zero time or once, namely the message may be lost; at least one success means that the message transmission may be repeated but not lost; the message transport mechanism of exact once is one and only one per message, i.e., the message transport is neither lost nor repeated.
Fault tolerance: failures in the flow processing framework can occur at various levels, such as network segments, disk crashes or node crashes. The stream processing framework should have the ability to recover from all such failures and re-consume from the last successful state (no dirty data).
Performance: latency (Latency), Throughput (Throughput), and Scalability (Scalability) are important metrics in stream processing applications.
Redis: the system is an open source log-type and Key-Value database which is written by using ANSI C language, supports network, can be based on memory and can also be persistent, and provides an application program structure of multiple languages. It supports relatively more Value types to be stored, including string, list, set, zset, and hash. These data types all support push/pop, add/remove, and intersect union and difference, and richer operations, and these operations are all atomic. On this basis, Redis supports various different ways of ordering. In order to ensure efficiency, data is cached in a memory, updated data is periodically written into a disk or modification operation is written into an additional recording file, and master-slave synchronization is realized on the basis.
A DAG: in graph theory, if a Directed graph cannot go from a vertex back to the point through several edges, the graph is a Directed Acyclic Graph (DAG). A directed acyclic graph may not be convertible to a tree because a directed graph does not necessarily form a ring from one point to another point through two routes, but any directed tree is a directed acyclic graph.
In order to describe the technical problems, technical solutions and technical effects actually solved by the embodiments of the present invention in more detail, a summary analysis is performed on the existing technical solution of distributed content stream processing in the embodiments of the present invention, and the existing technical solution of distributed content stream processing mainly includes three architectures, which are respectively:
(1) synchronization architecture
Each processing unit participating in stream processing in the content stream processing system processes the content set in sequence, and the execution of the processing unit downstream is triggered after the execution of the processing unit upstream is finished.
(2) Architecture based on message queue schema
The architecture based on the message queue mode is similar to the synchronous architecture, except that the execution of each processing unit in the content stream processing system is triggered by the message queue, and if a certain processing unit fails to execute, the content in the message queue is cyclically consumed by a downstream module for retry, although the architecture does not need to design the reprocessing logic in the service logic any more, the blocking of the message queue is easy to occur when the content which cannot be processed is encountered.
(3) Scanning framework
All contents related to the stream processing contents are uniformly stored in a preset storage space, each processing unit scans the storage space at regular time, executes the contents meeting the execution conditions, and writes the execution results into the storage space for a downstream processing unit to use.
The scanning architecture is actually a batch processing structure, and each processing unit scans the content meeting its own processing condition from the storage space at regular time, but this results in that each processing unit waits for the scanning process for most of the time in the whole content processing process, which increases the load of the storage space and increases the processing delay of the content stream processing system.
The actions of the processing units in the content stream processing systems with the three architectures have strong correlation with the service logic, and if the service logic changes, the processing units in the stream processing system may need to be modified, so that the content stream processing system can only be used in cooperation with a single service logic to provide content stream processing service for the single service logic, but cannot simultaneously provide content stream processing service for multiple service logics in parallel.
Further, due to the strong correlation between the content stream processing system and the service logic, the expansion of the service logic may result in a large modification of the content stream processing system, and as the complexity of the service logic gradually increases, it is likely that the maintenance cost of each processing unit in the content stream processing system is too high, and even the content stream processing system is difficult to adapt to the service logic and loses the use value.
In view of the above, embodiments of the present invention provide a content stream processing method, so as to provide a stream processing scheme decoupled from specific service logic, which is different from the existing content stream processing framework. First, the basic embodiment of the present invention provides one possible implementation environment of a content stream processing method.
Referring to fig. 1, the implementation environment includes: client 01, content console 03 and service server 05.
The content to be processed generated by one or more services in the client 01 is transmitted to the content console 03.
In the embodiment of the invention, a middlebox concept is used for constructing an implementation environment for content stream processing, and the middlebox aims to flexibly realize the support of various uncertain business rules and processing modes by designing a scientific technical framework so as to improve the cooperation efficiency of related business links.
The content center 03 processes various contents from the client 01 in a streaming manner, and sends the processing result to one or more service servers 05. The content center 03 encapsulates each processing unit participating in stream processing into a micro service, generates a micro service calling topology according to the requirements of each service server, and calls the micro service to process the content to be processed based on the micro service calling topology, so that the micro service is decoupled from the specific service through the micro service calling topology, and the purpose of providing the content stream processing service for a plurality of services in parallel is achieved.
The Client 01 is communicatively connected to the content center 03, and the Client 01 may communicate with the content center 03 based on a Browser/Server mode (Browser/Server, B/S) or a Client/Server mode (Client/Server, C/S). The client 01 may include: the physical devices may also include software running in the physical devices, such as applications, and the like. For example, the client 01 may run instant messaging software, micro video software, social software, image processing software, news software, and the like.
The content center 03 and the service server 05 may comprise a server operating independently, or a distributed server, or a server cluster composed of a plurality of servers.
Referring to fig. 2, a flow chart of a content stream processing method is shown, which may be implemented in the implementation environment of fig. 1, and the method may include:
s101, obtaining original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space.
In a possible embodiment, the original content may come from multiple services, for example, the original content may come from various small video programs, small news programs, etc. in the client, and the actual source of the content to be processed is not limited in the embodiment of the present invention. Further, the specific form of the content to be processed is not limited, and the content to be processed may be one or more of pictures, texts, videos, animations and voices.
In one possible embodiment, to ensure that the entire original content is obtained, the generation logic of the content object may be triggered based on a signal trigger. When the content center station 03 acquires the warehousing signal, representing to start acquiring the original content; if the original content includes a plurality of sub-contents, the content center station 03 acquires a content processable signal when each sub-content is completely acquired, and the content center station 03 acquires a content object generation signal when all sub-contents in the original content are completely acquired. Different links of the original content are obtained to correspond to different signals, so that the obtaining process is clearer, and the probability of errors is reduced.
The content center station 03 may acquire a plurality of to-be-processed contents in parallel, generate a content object for each to-be-processed content, and store the content object.
In particular, the content object may include an object identification, original content, and a current processing state. The object identifier in the content object is the unique identifier of the content object, the current processing state represents the processing state of the content object, and if the content object is not processed by the content console 03, the current processing state of the content object is unprocessed; if the content object is being processed by the content broker 03, the current processing state of the content object is in process.
In a possible embodiment, the storage of the content object and the processing of the content object are concurrent, and in order to facilitate selection of the processing order of the content object in the content center station 03, in the embodiment of the present invention, the following design is performed on a content object generation logic, where the generating of the content object corresponding to the original content includes, as shown in fig. 3:
s1011, obtaining the grading correlation attribute of the original content.
Specifically, the score association attribute is one or a group of attributes that can characterize the importance or urgency of the original content. For example, the scored association attributes may include the source of the original content or the popularity of keywords in the original content. In other embodiments, the score association attribute may also include the acquisition time of the original content.
For example, if the original content is sourced from official media or a premium publisher account, it is determined that the original content has a higher value and, correspondingly, has a higher score. If the keyword popularity of the original content is higher, the original content is judged to have higher popularity and belong to hotspot information, and correspondingly, the original content also has higher score. Due to the design, the original content with different grading association attributes corresponds to different priorities, such as some news hot spot burst content or some authoritative content, and can be processed as soon as possible, so that the content can be rapidly distributed to a corresponding service server, and a target user can be reached more quickly.
And S1013, scoring the original content according to the scoring correlation attributes and a preset scoring algorithm.
And S1015, generating a content object according to the original content and the score.
S1017, generating an object identifier and a current processing state for the content object.
And S103, acquiring the target content object in the storage space.
Specifically, step S103 may be triggered to execute in response to a preset control instruction, or may be triggered to execute by a timer, which may be executed multiple times, and the embodiment of the present invention does not limit a specific trigger execution manner thereof.
In one possible implementation, the obtaining the target content object in the storage space, as shown in fig. 4, includes:
and S1031, traversing the storage space to obtain a content object set to be processed, wherein the current processing state of each content object in the content object set to be processed is unprocessed.
S1033, traversing the content object set to be processed to obtain a target content object set, wherein the score of the content object in the target content object set is larger than a preset score threshold value.
And S1035, determining each content object in the target content object set as a target content object.
In another possible implementation, the obtaining the target content object in the storage space, as shown in fig. 5, includes:
s1032, traversing the storage space to obtain a content object set to be processed, wherein the current processing state of each content object in the content object set to be processed is unprocessed.
S1034, arranging the content objects in the content object set to be processed according to the corresponding scores in a descending order to obtain a target content sequence.
And S1036, sequentially determining each content object in the target content sequence as a target content object.
The two embodiments can be used alone or in combination, which can ensure that the station always processes the more important content preferentially, and to achieve this technical effect, the score of the corresponding content object should dynamically reflect the current importance or urgency of the content object. The embodiment of the invention supports concurrent processing of the target content objects, and the processing flow of a single target object is taken as an example for explanation in the subsequent embodiment.
And S105, acquiring at least one service associated with the target content object, and acquiring the corresponding micro-service call topology according to the service to obtain a micro-service call topology set corresponding to the target content object.
In the embodiment of the invention, different target content objects may be associated with different services, and each service has a micro-service call topology corresponding to the service. Therefore, before obtaining the micro-service invocation topology corresponding to the service according to the service, the method further includes a step of constructing the micro-service invocation topology corresponding to each service, where the constructing the micro-service invocation topology corresponding to each service is performed for each service, as shown in fig. 6, and includes:
s1, packaging each processing link of content stream processing to obtain a micro-service set, wherein each micro-service in the micro-service set is independently called to realize an atomic processing function.
Different from a framework in which a service is coupled with each processing link in content stream processing in the prior art, in order to achieve the purpose of decoupling, in the embodiment of the present invention, each processing action in the content stream processing process is encapsulated to obtain a single function execution body capable of independently performing an atomic content processing function.
And S3, responding to the service subscription request, and acquiring the content stream processing logic corresponding to the service.
And S5, obtaining the micro-service call topology corresponding to the service according to the content flow processing logic corresponding to the service and the micro-service set.
Specifically, the micro-service invocation topology points to a micro-service subset required for implementing the content stream processing logic corresponding to the service, and an invocation dependency relationship between micro-services in the micro-service subset is recorded. In one possible implementation, the microservice invocation topology may be represented by a directed acyclic graph.
Referring to FIG. 7, a schematic diagram of the execution logic of a micro service invocation topology in a set of micro service invocation topologies is shown. The micro-service calling topology set comprises two micro-service calling topologies, namely a micro-service calling topology A and a micro-service calling topology B, and each micro-service in the micro-service calling topology A and the micro-service calling topology B is sequentially scheduled according to the arrow indication.
The micro service invocation topology a and the micro service invocation topology B may each include three types of micro services, as detailed below:
(1) micro-services that are scheduled by different scheduling services to obtain different invocation results.
For example, the service calling topology a and the micro-service calling topology B both include a watermark micro-service and a cover micro-service, and the watermark micro-service and the cover micro-service can be respectively called by the scheduling service of the micro-service calling topology a and the scheduling service of the micro-service calling topology B, so that two groups of watermarks and two groups of covers are obtained.
(2) The microservice of the call results may be shared.
For example, the service invocation topology a and the micro-service invocation topology B both include a tag micro-service, which can be invoked only once by the scheduling service of the micro-service invocation topology a or the scheduling service of the micro-service invocation topology B to obtain a tag, and the tag can be shared by other scheduling services to save the scheduling overhead.
(3) A microservice scheduled by a single scheduling service.
Such as svr2, svr4, and svr7 in service invocation topology a and svr3, svr6, and svr8 in service invocation topology B.
In the embodiment of the invention, each micro-service calling topology set can have a plurality of sets of micro-service calling topologies, each micro-service calling topology can provide content stream processing capability for corresponding service, and the micro-service calling topology set can be constructed to enable a content center station to have the capability of providing various stream processing services for original content so as to meet different service requirements, which is a technical effect brought by decoupling of micro-service and a technical effect which is difficult to achieve by a content stream processing system under the existing synchronous architecture, scanning architecture and message queue-based architecture.
S107, according to each micro-service calling topology in the micro-service calling topology set, relevant micro-services are called concurrently to perform stream processing on the target content object, and calling associated information generated in the stream processing process is stored in the target content object.
In a feasible embodiment, the concurrently invoking, according to each micro service invocation topology in the micro service invocation topology set, a related micro service to perform stream processing on the target content object, and storing invocation associated information generated in a stream processing process into the target content object, as shown in fig. 8, includes:
s1071, modifying the current processing state of the target content object into processing.
S1073, generating the scheduling service corresponding to each micro-service scheduling topology according to each micro-service scheduling topology in the micro-service scheduling topology set.
The content center station can perform unified management and unified monitoring on each scheduling service of each target content object.
S1075, executing all the scheduling services concurrently, calling the micro services in sequence by each scheduling service according to the micro service calling topology corresponding to the scheduling service, and storing the calling associated information of the micro services into the target content object.
The calling associated information comprises a calling result of the micro service scheduling success or calling information of the micro service scheduling failure.
Specifically, each scheduling service sequentially calls the micro services according to the micro service call topology corresponding to the scheduling service, as shown in fig. 9, including:
s10751, the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service.
When the current micro service is scheduled, the parameters necessary for execution of the current micro service can be obtained according to the target content object, and the calling result is transmitted back to the scheduling service after the execution is finished.
S10753, if the current micro service exists, calling the current micro service.
If not, the scheduling service for the target content object is ended.
S10755, if the current micro service is successfully called, storing a calling result to the target content object, and returning to the step S10751.
S10757, if the current micro service is failed to be called, the current micro service is called again according to preset configuration.
S10759, if the micro service is successfully called in the retrying process, storing the calling result to the target content object, and returning to the step S10751.
S107511, if the micro-services are all failed to be called in the retry process, the current processing state of the target content object is modified to be unprocessed, and the scheduling service is ended.
If all the micro-services are failed to be called in the retry process, the scheduling service for the target content object is ended, and the current processing state of the target content object is modified to be unprocessed, so that the target content object is included in the to-be-processed content object set again in step S1031, and the target content object can participate in the next stream processing. And the related information of the scheduling service is reserved in the target content object, so that the stream processing can be continued according to the related information of the scheduling service. Namely, the next stream processing can be performed At the last interrupted position, thereby realizing the message transmission guarantee of the At least one entity.
S109, obtaining the service output object corresponding to each service associated with the target content object from the target content object.
In the embodiment of the invention, the calling associated information generated in the process that the target content object is processed by each scheduling service is stored in the target content object, in other words, the target content object comprises all information required by the associated service, so that the service output object can be obtained according to the target content object.
Taking fig. 7 as an example, after the target content object is processed by the micro-service related to the micro-service calling topology a and the micro-service calling topology B, the target content object includes the watermark 1, the watermark 2, the cover 1, the cover 2, a tag, and information from svr2, svr3, svr4, svr6, svr7, svr8, and a service output object required by the service 1 can be obtained according to the information from the watermark 1, the cover 1, the tag, and the information from svr2, svr4, svr7, and a service output object required by the service 2 can be obtained according to the information from the watermark 2, the cover 2, the tag, the information from svr3, svr6, svr 8.
Taking the processing of video content as an example, after an original content is captured, it may be necessary to perform content security quality control on the content (whether violence is present, whether it is popular, etc.), perform feature extraction modeling on the content (content classification, content subject extraction, content tag generation, etc.), understand and summarize the content (generate abstract, structure content related information, clip the content, etc.), perform manual processing on the content (review, labeling, etc.), thereby obtaining attribute tags that can be identified by various content recommendation services, and finally distribute the attribute tags to service servers corresponding to the recommendation services, so as to reach target clients quickly. In different services, some processing links have dependency relationships, for example, the selection of a cover map depends on picture pornography identification, some processing links have no dependency relationship and can perform parallel processing, and the content stream processing method provided by the embodiment of the invention can adapt to the corresponding micro-service call topology for each service so as to provide stream processing services for each service in parallel. Referring to fig. 10, a schematic diagram of a content stream processing method applied to a video processing scene according to an embodiment of the present invention is shown. The original content in the figure can be sourced from various sources, such as a hotspot video, a relaxation video, a diary video and a kitten video, and at the moment, a live video. The content stream processing related services can also be various, such as a hotspot video, a relaxation video, a diary video and a kitten video, and at the moment, a live video. One piece of content from the hot video can be pushed to the hot video, the relaxing video, the diary video and the kitten video, and at the moment, the business service corresponding to any one business in the live video.
The content stream processing method is implemented in the content middle platform, uniformly places all original contents in a storage space to finish uniform content storage, and uniformly delivers the service output objects obtained through stream processing.
The method comprises the following steps that a great number of micro services are participated in a specific stream processing process, and the stream processing process is divided into a machine processing micro service set and a manual processing micro service set according to an automatic execution degree, wherein the machine processing micro service set comprises a video transcoding micro service, an information security micro service, a picture processing micro service, a cover processing micro service, an intelligent screenshot micro service, a video duplication removal micro service, a picture processing micro service and the like; the manual processing micro-service set comprises a video classification labeling micro-service, a yellow gambling poison identification micro-service, a review micro-service, an editing and processing micro-service, an advertisement labeling micro-service and the like.
According to the stream processing method provided by the embodiment of the invention, the micro-service calling topology corresponding to each service is adapted, and the micro-service is orderly called according to the micro-service calling topology, so that the service output object meeting the ex-warehouse requirement is obtained.
Referring to fig. 11, a schematic diagram of an implementation of a stream processing method according to an embodiment of the present invention is shown. The method comprises the steps of placing each original content into a storage space, enabling a content middle station to timely interact with the storage space to obtain a target content object needing to be subjected to stream processing, determining micro-service calling topology corresponding to the target content object, calling each micro-service based on the micro-service calling topology, feeding a calling result back to the target content object to complete stream processing on the target content object, and further obtaining a service output object meeting service requirements based on the target content object. Furthermore, the content center station can provide other services such as monitoring alarm, frequency control, topology management, data reporting and the like for the outside. Specifically, the monitoring alarm service is configured to monitor each stream processing thread running in the content center station, and alarm the found abnormality. The frequency control service may control the frequency of acquiring the target content object. The topology management service can manage the micro-service call topology supported by the content center station service according to different services. And the data reporting service is used for recording the information generated by the operation of the station service in the content and reporting the information.
The stream processing method disclosed in the embodiment of the invention realizes the atomization of stream processing by encapsulating each processing link involved in the stream processing process to obtain the micro-service for executing the atomic processing function. The micro-service calling topology is generated for the business related to the stream processing, and the micro-service is called according to the micro-service calling topology to realize the decoupling of the business and the stream processing, so that the stream processing method can simultaneously meet the stream processing requirements of a plurality of businesses to obtain the stream processing result required by each business. This stream processing method has at least the following significant advantages:
(1) the method has the advantages of expandability: the addition, deletion, or update of a certain processing function in stream processing does not affect other stream processing functions.
(2) The method has maintainability: the addition, deletion or update of a certain processing function in stream processing does not affect the complexity of the whole stream processing function.
(3) Possesses high reliability: the exception generated by a single processing link is easy to be discovered, and the target content object can be ensured to be processed successfully finally through a retry mechanism and the state modification of the target content object.
(4) Possesses high flexibility: different services have corresponding micro-service call topologies, and each service can be served through simple configuration.
(5) And the streaming processing is carried out based on the micro-service call topology, so that the overall delay is reduced.
An embodiment of the present invention further provides a content stream processing apparatus, as shown in fig. 12, the apparatus includes:
an original content processing module 201, configured to obtain original content to be processed, generate a content object corresponding to the original content, and store the content object in a preset storage space;
a target content object module 203, configured to obtain a target content object in the storage space;
a micro service invocation topology set obtaining module 205, configured to obtain at least one service associated with the target content object, and obtain a micro service invocation topology corresponding to the service according to the service, so as to obtain a micro service invocation topology set corresponding to the target content object;
a micro-service invoking module 207, configured to concurrently invoke a relevant micro-service to perform stream processing on the target content object according to each micro-service invoking topology in the micro-service invoking topology set, and store invoking association information generated in a stream processing process in the target content object;
a service output object obtaining module 209, configured to obtain, from the target content object, a service output object corresponding to each service associated with the target content object.
Specifically, the content stream processing apparatus and the method according to the embodiments of the present invention are all based on the same inventive concept. For details, please refer to the method embodiment, which is not described herein.
The embodiment of the invention also provides a computer storage medium, and the computer storage medium can store a plurality of instructions. The instructions may be adapted to be loaded by a processor and to perform a method of content stream processing according to an embodiment of the invention, the method comprising at least the steps of:
a method of content stream processing, the method comprising:
acquiring original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space;
acquiring a target content object in the storage space;
acquiring at least one service associated with the target content object, and acquiring a micro-service call topology corresponding to the service according to the service to obtain a micro-service call topology set corresponding to the target content object;
concurrently calling related micro-services to perform stream processing on the target content object according to each micro-service calling topology in the micro-service calling topology set, and storing calling associated information generated in the stream processing process into the target content object;
and obtaining the service output object corresponding to each service associated with the target content object from the target content object.
In a preferred embodiment, the generating the content object corresponding to the original content includes:
acquiring the grade correlation attribute of the original content;
scoring the original content according to the scoring correlation attributes and a preset scoring algorithm;
generating a content object according to the original content and the score;
an object identification and a current handling shape are generated for the content object.
In a preferred embodiment, the obtaining the target content object in the storage space includes:
traversing the storage space to obtain a set of content objects to be processed, wherein the current processing state of each content object in the set of content objects to be processed is unprocessed;
traversing the content object set to be processed to obtain a target content object set, wherein the score of the content object in the target content object set is greater than a preset score threshold value;
and determining each content object in the target content object set as a target content object.
In a preferred embodiment, the obtaining the target content object in the storage space includes:
traversing the storage space to obtain a set of content objects to be processed, wherein the current processing state of each content object in the set of content objects to be processed is unprocessed;
sequencing each content object in the content object set to be processed in a descending order according to the corresponding score of the content object set to be processed to obtain a target content sequence;
and sequentially determining each content object in the target content sequence as a target content object.
In a preferred embodiment, before the obtaining of the micro-service invocation topology corresponding to the service according to the service, the method further includes a step of constructing the micro-service invocation topology corresponding to each service, where the constructing of the micro-service invocation topology corresponding to each service includes:
packaging each processing link of content stream processing to obtain a micro-service set, wherein each micro-service in the micro-service set is independently called to realize an atomic processing function;
responding to a service subscription request, and acquiring content stream processing logic corresponding to a service;
and acquiring the micro-service call topology corresponding to the service according to the content flow processing logic corresponding to the service and the micro-service set.
In a preferred embodiment, the concurrently invoking, according to each micro-service invocation topology in the micro-service invocation topology set, a related micro-service to perform stream processing on the target content object, and storing invocation associated information generated in a stream processing process into the target content object includes:
modifying the current processing state of the target content object into processing;
generating scheduling service corresponding to each micro-service scheduling topology according to each micro-service scheduling topology in the micro-service scheduling topology set;
and concurrently executing each scheduling service, sequentially calling the micro-services by each scheduling service according to the micro-service calling topology corresponding to each scheduling service, and storing calling associated information of the micro-services into the target content object.
In a preferred embodiment, each scheduling service sequentially invokes the micro services according to its corresponding micro service invocation topology, including:
the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service;
if the current micro service exists, calling the current micro service;
if the current micro service is successfully called, storing a calling result to the target content object, and returning to the step: the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service;
if the current micro service is failed to be called, the current micro service is retried to be called according to preset configuration;
if the micro-service is successfully called in the retry process, storing the calling result to the target content object, and returning to the step: the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service;
and if the micro-services are all failed to be called in the retry process, modifying the current processing state of the target content object to be unprocessed, and ending the scheduling service.
Further, fig. 13 shows a hardware structure diagram of an apparatus for implementing the method provided by the embodiment of the present invention, and the apparatus may participate in forming or containing the device or system provided by the embodiment of the present invention. As shown in fig. 13, the device 10 may include one or more (shown as 102a, 102b, … …, 102 n) processors 102 (the processors 102 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 13 is only an illustration and is not intended to limit the structure of the electronic device. For example, device 10 may also include more or fewer components than shown in FIG. 13, or have a different configuration than shown in FIG. 13.
It should be noted that the one or more processors 102 and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuitry may be a single, stand-alone processing module, or incorporated in whole or in part into any of the other elements in the device 10 (or mobile device). As referred to in the embodiments of the application, the data processing circuit acts as a processor control (e.g. selection of a variable resistance termination path connected to the interface).
The memory 104 may be used for storing software programs and modules of application software, such as program instructions/data storage devices corresponding to the method described in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 104, so as to implement one of the content stream processing methods described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from processor 102, which may be connected to device 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of such networks may include wireless networks provided by the communication provider of the device 10. In one example, the transmission device 106 includes a network adapter (NIC) that can be connected to other network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the device 10 (or mobile device).
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device and server embodiments, since they are substantially similar to the method embodiments, the description is simple, and the relevant points can be referred to the partial description of the method embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (10)

1. A method of content stream processing, the method comprising:
acquiring original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space;
acquiring a target content object in the storage space;
acquiring at least one service associated with the target content object, and acquiring a micro-service call topology corresponding to the service according to the service to obtain a micro-service call topology set corresponding to the target content object;
concurrently calling related micro-services to perform stream processing on the target content object according to each micro-service calling topology in the micro-service calling topology set, and storing calling associated information generated in the stream processing process into the target content object;
and obtaining the service output object corresponding to each service associated with the target content object from the target content object.
2. The method of claim 1, wherein generating the content object corresponding to the original content comprises:
acquiring the grade correlation attribute of the original content;
scoring the original content according to the scoring correlation attributes and a preset scoring algorithm;
generating a content object according to the original content and the score;
an object identification and a current processing state are generated for the content object.
3. The method of claim 2, wherein obtaining the target content object in the storage space comprises:
traversing the storage space to obtain a set of content objects to be processed, wherein the current processing state of each content object in the set of content objects to be processed is unprocessed;
traversing the content object set to be processed to obtain a target content object set, wherein the score of the content object in the target content object set is greater than a preset score threshold value;
and determining each content object in the target content object set as a target content object.
4. The method of claim 2, wherein obtaining the target content object in the storage space comprises:
traversing the storage space to obtain a set of content objects to be processed, wherein the current processing state of each content object in the set of content objects to be processed is unprocessed;
sequencing each content object in the content object set to be processed in a descending order according to the corresponding score of the content object set to be processed to obtain a target content sequence;
and sequentially determining each content object in the target content sequence as a target content object.
5. The method according to claim 1, further comprising a step of constructing a micro service invocation topology corresponding to each service before said obtaining the micro service invocation topology corresponding to the service according to the service, wherein said constructing the micro service invocation topology corresponding to each service comprises:
packaging each processing link of content stream processing to obtain a micro-service set, wherein each micro-service in the micro-service set is independently called to realize an atomic processing function;
responding to a service subscription request, and acquiring content stream processing logic corresponding to a service;
and acquiring the micro-service call topology corresponding to the service according to the content flow processing logic corresponding to the service and the micro-service set.
6. The method according to claim 1, wherein the concurrently invoking relevant micro services according to each micro service invocation topology in the micro service invocation topology set to stream-process the target content object, and storing invocation associated information generated in a stream processing process into the target content object, comprises:
modifying the current processing state of the target content object into processing;
generating scheduling service corresponding to each micro-service scheduling topology according to each micro-service scheduling topology in the micro-service scheduling topology set;
and concurrently executing each scheduling service, sequentially calling the micro-services by each scheduling service according to the micro-service calling topology corresponding to each scheduling service, and storing calling associated information of the micro-services into the target content object.
7. The method of claim 6, wherein each scheduling service sequentially invokes microservices according to its corresponding microservice invocation topology, comprising:
the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service;
if the current micro service exists, calling the current micro service;
if the current micro service is successfully called, storing a calling result to the target content object, and returning to the step: the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service;
if the current micro service is failed to be called, the current micro service is retried to be called according to preset configuration;
if the micro-service is successfully called in the retry process, storing the calling result to the target content object, and returning to the step: the scheduling service determines the current micro service according to the micro service call topology corresponding to the scheduling service;
and if the micro-services are all failed to be called in the retry process, modifying the current processing state of the target content object to be unprocessed, and ending the scheduling service.
8. An apparatus for processing a content stream, the apparatus comprising;
the original content processing module is used for acquiring original content to be processed, generating a corresponding content object for the original content, and storing the content object in a preset storage space;
the target content object module is used for acquiring a target content object in the storage space;
a micro-service call topology set acquisition module, configured to acquire at least one service associated with the target content object, and acquire a corresponding micro-service call topology according to the service, so as to obtain a micro-service call topology set corresponding to the target content object;
the micro-service calling module is used for calling related micro-services to perform stream processing on the target content object according to each micro-service calling topology in the micro-service calling topology set and storing calling associated information generated in the stream processing process into the target content object;
and the service output object acquisition module is used for acquiring the service output object corresponding to each service associated with the target content object from the target content object.
9. An apparatus comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by the processor to implement a content stream processing method according to any one of claims 1 to 7.
10. A computer storage medium having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded by a processor and which performs a method of content stream processing as claimed in any one of claims 1 to 7.
CN201910794944.3A 2019-08-27 2019-08-27 Content stream processing method, device, equipment and medium Active CN110673962B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910794944.3A CN110673962B (en) 2019-08-27 2019-08-27 Content stream processing method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910794944.3A CN110673962B (en) 2019-08-27 2019-08-27 Content stream processing method, device, equipment and medium

Publications (2)

Publication Number Publication Date
CN110673962A true CN110673962A (en) 2020-01-10
CN110673962B CN110673962B (en) 2024-06-14

Family

ID=69075739

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910794944.3A Active CN110673962B (en) 2019-08-27 2019-08-27 Content stream processing method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN110673962B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113259714A (en) * 2021-06-30 2021-08-13 腾讯科技(深圳)有限公司 Content distribution processing method and device, electronic equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108415944A (en) * 2018-01-30 2018-08-17 长安大学 Real time computation system and its implementation based on micro services under a kind of traffic environment
CN109684105A (en) * 2018-12-18 2019-04-26 中国平安人寿保险股份有限公司 Method, equipment and the storage medium that request is controlled under micro services framework
CN109961204A (en) * 2017-12-26 2019-07-02 中国移动通信集团浙江有限公司 Quality of service analysis method and system under a kind of micro services framework

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109961204A (en) * 2017-12-26 2019-07-02 中国移动通信集团浙江有限公司 Quality of service analysis method and system under a kind of micro services framework
CN108415944A (en) * 2018-01-30 2018-08-17 长安大学 Real time computation system and its implementation based on micro services under a kind of traffic environment
CN109684105A (en) * 2018-12-18 2019-04-26 中国平安人寿保险股份有限公司 Method, equipment and the storage medium that request is controlled under micro services framework

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113259714A (en) * 2021-06-30 2021-08-13 腾讯科技(深圳)有限公司 Content distribution processing method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN110673962B (en) 2024-06-14

Similar Documents

Publication Publication Date Title
US11860874B2 (en) Multi-partitioning data for combination operations
US11615084B1 (en) Unified data processing across streaming and indexed data sets
US11151137B2 (en) Multi-partition operation in combination operations
CN112307037B (en) Data synchronization method and device
CN113766035B (en) Service acceptance and consensus method and device
JP6171104B2 (en) Providing access to cloud-based content management systems on mobile devices
US11663242B2 (en) Mass insertion into single-threaded databases
Varia Cloud architectures
US11301425B2 (en) Systems and computer implemented methods for semantic data compression
CN109684036B (en) Container cluster management method, storage medium, electronic device and system
CN106790549B (en) Data updating method and device
CN113779094B (en) Batch-flow-integration-based data processing method and device, computer equipment and medium
CN110716848A (en) Data collection method and device, electronic equipment and storage medium
CN112328448A (en) Zookeeper-based monitoring method, monitoring device, equipment and storage medium
CN111611479B (en) Data processing method and related device for network resource recommendation
CN115185705A (en) Message notification method, device, medium and equipment
CN110673962A (en) Content stream processing method, device, equipment and medium
CN112527519A (en) High-performance local cache method, system, equipment and medium
CN117950850A (en) Data transmission method, device, electronic equipment and computer readable medium
CN116760821A (en) Distributed file downloading method, system, electronic equipment and storage medium
CN112019689A (en) Incoming call show service processing system and method
US11989592B1 (en) Workload coordinator for providing state credentials to processing tasks of a data processing pipeline
Korostelev et al. QoS management in distributed service oriented systems.
CN112988806A (en) Data processing method and device
CN115277610B (en) Message split sending method, device, equipment and medium based on dual-activity environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant