CN113382071A - Link creation method and device based on hybrid cloud architecture - Google Patents

Link creation method and device based on hybrid cloud architecture Download PDF

Info

Publication number
CN113382071A
CN113382071A CN202110643675.8A CN202110643675A CN113382071A CN 113382071 A CN113382071 A CN 113382071A CN 202110643675 A CN202110643675 A CN 202110643675A CN 113382071 A CN113382071 A CN 113382071A
Authority
CN
China
Prior art keywords
log
service
target
cloud
type
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110643675.8A
Other languages
Chinese (zh)
Other versions
CN113382071B (en
Inventor
申阳
贾承昆
王磊
雷冉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Ape Power Future Technology Co Ltd
Original Assignee
Beijing Ape Power Future Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Ape Power Future Technology Co Ltd filed Critical Beijing Ape Power Future Technology Co Ltd
Priority to CN202110643675.8A priority Critical patent/CN113382071B/en
Publication of CN113382071A publication Critical patent/CN113382071A/en
Application granted granted Critical
Publication of CN113382071B publication Critical patent/CN113382071B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/14Session management
    • H04L67/141Setup of application sessions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3089Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
    • G06F11/3093Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3414Workload generation, e.g. scripts, playback
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06Management of faults, events, alarms or notifications
    • H04L41/069Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08Configuration management of networks or network elements
    • H04L41/0803Configuration setting
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Debugging And Monitoring (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The present specification provides a link creation method and device based on a hybrid cloud architecture, wherein the link creation method based on the hybrid cloud architecture comprises: acquiring cloud service information associated with a target service; determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service; creating a log collection node corresponding to the log type according to preset log collection configuration information; generating a log collection link corresponding to the log type based on the log collection node; the log acquisition link under the hybrid cloud architecture is reconstructed, the integrity and timeliness of log acquisition are guaranteed, and the log acquisition efficiency is improved.

Description

Link creation method and device based on hybrid cloud architecture
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a link creation method and apparatus based on a hybrid cloud architecture.
Background
With the development of internet technology, the online business mode becomes an important maintenance object for most business parties. In order to support normal development of services, a service provider needs to provide a stable service processing cluster and a stable processing link, so as to support normal development of front-end services under the condition of stable data volume. However, in some service scenarios, a sudden increase of data volume may be involved, and if the original computing resources are still used, the problem of service blocking may be caused, and even the service may be crashed. Therefore, most business parties can adopt a hybrid cloud architecture to maintain normal business operation in order to support the sudden increase of data volume. Since part of service functions in the hybrid cloud architecture are migrated to the cloud service platform, the existing log collection strategy cannot guarantee the integrity of the collected logs, and an effective scheme is urgently needed to solve the above problems.
Disclosure of Invention
In view of this, the embodiments of the present specification provide a link creation method based on a hybrid cloud architecture. The present specification also relates to a link creation apparatus based on a hybrid cloud architecture, a computing device, and a computer-readable storage medium, so as to solve the technical defects in the prior art.
According to a first aspect of embodiments of the present specification, there is provided a link creation method based on a hybrid cloud architecture, including:
acquiring cloud service information associated with a target service;
determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
creating a log collection node corresponding to the log type according to preset log collection configuration information;
and generating a log collection link corresponding to the log type based on the log collection node.
Optionally, before the step of obtaining cloud service information associated with the target service is executed, the method further includes:
receiving a service migration request submitted aiming at the target service;
extracting sub-services to be migrated from the target services according to the service migration request, and determining a cloud server for bearing the sub-services to be migrated;
and uploading the sub-service to be migrated to the cloud server, and receiving service configuration completion information returned by the cloud server.
Optionally, the obtaining cloud service information associated with the target service includes:
analyzing the service configuration completion information to obtain a service processing type corresponding to the sub-service to be migrated;
and determining the cloud service information according to the service processing type.
Optionally, the determining, according to the cloud service information and the service information of the target service, a log type of at least one log to be collected includes:
acquiring the service information of the target service, and determining a cloud server associated with the target service according to the cloud service information;
under the condition that the target service runs on the cloud server, determining at least one log to be collected, and determining a log type group corresponding to the target service according to the service information;
determining the log type from the log type group based on the at least one log to be collected.
Optionally, the log type comprises at least one of:
database log type, service log type, user behavior log type.
Optionally, the creating a log collection node corresponding to the log type according to preset log collection configuration information includes:
reading the preset log acquisition configuration information based on the log type to obtain node configuration information corresponding to the log type;
and creating an on-cloud log processing node and an off-cloud log processing node according to the node configuration information, and using the on-cloud log processing node and the off-cloud log processing node as the log acquisition node corresponding to the log type.
Optionally, the generating, based on the log collection node, a log collection link corresponding to the log type includes:
determining service structure information corresponding to the target service;
and integrating the log processing nodes on the cloud and the log processing nodes under the cloud according to the service structure information to obtain the log acquisition link corresponding to the log type.
Optionally, after the step of generating the log collection link corresponding to the log type based on the log collection node is executed, the method further includes:
acquiring an initial log corresponding to the log type based on the log acquisition link;
preprocessing the initial log to obtain a target log, and returning the target log;
and determining a log storage address according to the log type, and writing the returned target log into the log storage address.
Optionally, the preprocessing the initial log to obtain a target log includes:
filtering the initial log to obtain an intermediate log;
and merging the intermediate logs, and compressing the merged intermediate logs to obtain the target log.
Optionally, after the step of generating the log collection link corresponding to the log type based on the log collection node is executed, the method further includes:
obtaining a sample log;
performing pressure test on the target service based on the sample log and the log acquisition link to obtain a pressure test result;
and optimizing the cloud server and the local server which bear the target service according to the pressure test result.
Optionally, after the step of receiving the service migration request submitted for the target service is executed, the method further includes:
reading the service weight of the target sub-service contained in the target service according to the service migration request;
and processing each target sub-service contained in the target service based on a preset degradation strategy and the service weight, and updating the target service according to a processing result.
Optionally, the processing each target sub-service included in the target service based on the preset degradation policy and the service weight, and updating the target service according to a processing result includes:
extracting a sub-service set to be closed, a sub-service set to be deleted and/or a sub-service set to be modified in the target service based on the service weight;
and processing the sub-service set to be closed, the sub-service set to be deleted and/or the sub-service set to be modified by using the preset degradation strategy, and updating the target service according to a processing result.
According to a second aspect of embodiments of the present specification, there is provided a link creation apparatus based on a hybrid cloud architecture, including:
the acquisition module is configured to acquire cloud service information associated with the target business;
the determining module is configured to determine the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
the creating module is configured to create a log collecting node corresponding to the log type according to preset log collecting configuration information;
and the generating module is configured to generate a log collection link corresponding to the log type based on the log collection node.
According to a third aspect of embodiments herein, there is provided a computing device comprising:
a memory and a processor;
the memory is to store computer-executable instructions, and the processor is to execute the computer-executable instructions to:
acquiring cloud service information associated with a target service;
determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
creating a log collection node corresponding to the log type according to preset log collection configuration information;
and generating a log collection link corresponding to the log type based on the log collection node.
According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the hybrid cloud architecture-based link creation method.
According to the link creation method based on the hybrid cloud architecture, after the cloud service information associated with the target service is acquired, the log type of at least one log to be collected can be determined according to the cloud service information and the service information of the target service, then the log collection node corresponding to the log type is created according to the preset log collection configuration information, the log collection link corresponding to the log type can be generated based on the log collection node, the latest log collection link can be created aiming at any service architecture, and the link is determined based on the log type of the log to be collected, thereby ensuring that the creation of the generated link is targeted, therefore, the log collection strategy under the current service architecture is met, and the accurate collection of the log corresponding to the log type is realized during service processing.
Drawings
Fig. 1 is a flowchart of a link creation method based on a hybrid cloud architecture according to an embodiment of the present specification;
fig. 2 is a schematic structural diagram before service migration in a link creation method based on a hybrid cloud architecture according to an embodiment of the present specification;
fig. 3 is a schematic structural diagram after service migration in a link creation method based on a hybrid cloud architecture according to an embodiment of the present specification;
fig. 4 is a schematic structural diagram after service migration in another link creation method based on a hybrid cloud architecture according to an embodiment of the present specification;
fig. 5 is a process flow diagram of a hybrid cloud architecture-based link creation method applied in a log collection scenario according to an embodiment of the present specification;
fig. 6 is a schematic structural diagram of a link creation apparatus based on a hybrid cloud architecture according to an embodiment of the present specification;
fig. 7 is a block diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.
The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms to which one or more embodiments of the present specification relate are explained.
Mixing cloud: (Hybrid Cloud) merges public Cloud and private Cloud; private clouds are mainly enterprise-oriented users, and for security reasons, enterprises prefer to store data in the private clouds, but also desire to have access to the computing resources of the public cloud.
Logging: network equipment, a system, a service program and the like can generate an event record called log when in operation; each row is recorded with a description of the date, time, user and action, which will be a log.
Public cloud: the (Public clouds) refers to usable clouds provided by third-party providers for users, the Public clouds are generally available through the Internet and may be free or low-cost, and the core attribute of the Public clouds is shared resource service. There are many instances of such a cloud that can provide services throughout an open public network.
Private cloud: (Private Clouds) is built for one customer to use alone, thus providing the most effective control of data, security and quality of service. The company owns the infrastructure and can control the manner in which applications are deployed on this infrastructure. The private cloud can be deployed in a firewall of an enterprise data center, or can be deployed in a safe host hosting place, and the core attribute of the private cloud is a proprietary resource.
IDC: the Internet Data Center is a service platform with perfect equipment (including high-speed Internet access bandwidth, high-performance local area network, safe and reliable computer room environment and the like), specialized management and perfect application. On the basis of the platform, the IDC service provider provides Internet basic platform services (server hosting, virtual host, mail caching, virtual mail and the like) and various value-added services (site renting services, domain name system services, load balancing systems, database systems, data backup services and the like) for the client.
DB: (Data Base) is a collection of Data organized according to some Data model and stored in secondary storage. Such a data set has the following characteristics: as non-duplicative as possible, multiple applications of a particular organization are served in an optimal manner, with data structures that are independent of the application program using it, and the addition, deletion, modification, and retrieval of data are managed and controlled by unified software.
In the present specification, a link creation method based on a hybrid cloud architecture is provided, and the present specification relates to a link creation apparatus based on a hybrid cloud architecture, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.
In practical application, under a hybrid cloud architecture, in order to ensure normal operation of a service, a service party usually migrates part of service functions to a cloud service platform, and the service operation is supported in a manner of combining a public cloud provided by the cloud service platform and a private cloud provided by the service party, so that the purpose of saving computing resources is achieved, and meanwhile, the service can be ensured to support more complex scenes. However, in some special scenarios, such as an activity scenario (service activity) or a special time node scenario (holiday or holiday start time), all of which may involve a problem of data burst, although normal operation of a service may be supported by a cloud architecture, an original log collection architecture may have a performance bottleneck in a scenario of burst collection of instances, and it is not yet guaranteed that a log collection policy is performed normally, so an effective scheme is urgently needed to solve the problem.
In view of this, according to the link creation method based on the hybrid cloud architecture provided by the present application, after the cloud service information associated with the target service is obtained, the log type of at least one log to be collected may be determined according to the cloud service information and the service information of the target service, and then a log collection node corresponding to the log type is created according to the preset log collection configuration information, so that a log collection link corresponding to the log type may be generated based on the log collection node, and it is achieved that a latest log collection link may be created for any service architecture, and the link is determined based on the log type of the log to be collected, which ensures that the creation of the generated link is targeted, thereby satisfying the log collection policy under the current service architecture, and achieving accurate collection of the log corresponding to the log type during service processing.
Fig. 1 is a flowchart illustrating a link creation method based on a hybrid cloud architecture according to an embodiment of the present specification, which specifically includes the following steps:
step S102, cloud service information associated with the target business is obtained.
Specifically, the target service is a service provided to a service participant, and if the user watches the video to participate in the lottery, the target service may be the lottery at this time, and correspondingly, the service participant is a user watching the video and participating in the lottery; or the service party provides multimedia resources for the user to browse, the target service can be a multimedia resource browsing service, and correspondingly, the service participant is a browser for browsing the multimedia resources; and if the target service is a leased service, the service participant is the enterprise or the user leasing the resource. In practical application, the target service is a service provided by a service direction user/enterprise, and the specific service content of the target service may be set according to a practical application scenario, which is not limited in this embodiment.
Based on this, the cloud service information specifically refers to information related to a part of target sub-services migrated to the cloud service platform in the target service, and the service content of the target sub-services and the service processing logic of the target sub-services can be determined through the cloud service information, so that when the target sub-services are processed, logs related to the sub-services can be accurately collected conveniently.
Further, before creating a log collection link under a hybrid cloud architecture, a service party, in order to support a more complex service processing mechanism, migrates part of sub-services related to a target service to a cloud service platform according to a requirement, so as to ensure normal operation of the target service in a manner of combining a public cloud provided by the cloud service platform and a local private cloud, and therefore, before creating the link, service migration processing needs to be performed, which is specifically implemented in the following manner:
receiving a service migration request submitted by aiming at a target service;
extracting a sub-service to be migrated from a target service according to the service migration request, and determining a cloud server for bearing the sub-service to be migrated;
and uploading the sub-service to be migrated to a cloud server, and receiving service configuration completion information returned by the cloud server.
Specifically, the service migration request refers to a request that a service party migrates part of services related to a target service deployed in a private cloud to a public cloud; correspondingly, the sub-service to be migrated is a part of the service to be migrated to the public cloud, and it should be noted that the sub-service to be migrated belongs to a part of the target service, and the migration of the part of the sub-service does not cause a change in service logic of the target service, but only changes local computing into computing on the cloud. The cloud server is a server providing a public cloud, and public cloud resources provided by the cloud server can be determined in a purchasing manner or a resource replacement manner, which is not limited in this embodiment; correspondingly, the service configuration completion information specifically refers to information fed back after the deployment of the sub-service to be migrated is completed after the deployment of the sub-service to be migrated is deployed to the public cloud provided by the cloud server, so as to inform a service party providing the target service that the deployment of the sub-service to be migrated is completed.
Based on this, when a service side submits a service migration request for a target service, it is indicated that the current computing resources may not support normal operation of the target service, and the processing capability of the target service can be improved only by collecting cloud services. Therefore, the sub-service to be migrated can be determined in the target service according to the service migration request, so as to determine that the sub-service needs to be migrated to the cloud server; meanwhile, a cloud server bearing the sub-service to be migrated is determined according to the service migration request, the sub-service to be migrated is migrated to the cloud server, the cloud server completes deployment of the sub-service to be migrated and is connected with the sub-service not to be migrated so as to maintain normal operation of the target service, and finally the cloud server feeds back service configuration completion information to the service party after configuration is completed so as to inform the service party that deployment of the sub-service to be migrated is completed, the target service at the current stage corresponds to a hybrid cloud architecture, and computing capacity is remarkably improved.
Referring to log collection logic corresponding to the target service before migration shown in fig. 2, although the log collection logic is also a hybrid cloud architecture (cloud server and local IDC), main service content is still completed locally. And the corresponding log collection logic is also that the SLB (Server Load Balancer) provided by the cloud Server controls NGINX to support log collection, and the local IDC realizes log collection by combining the access service Frog with the fluent (log collector) and then transmits the log collection back to kafka so as to write the log collection into the local distributed system HDFS (f104) through the kafka for subsequent business processing and auditing.
Further, after receiving a service migration request submitted by a service party for a target service, determining a sub-service to be migrated in the target service to complete migration; taking a target service as a video-watching lottery activity as an example, in order to support that more users can participate in the video-watching lottery activity on the same day when the activity is open, a service party performs service migration processing on a service migration request created for the video-watching lottery activity at this time, determines sub-services to be migrated as all processing stages of a video-watching and lottery process according to the service migration request submitted by the service party, and a local IDC is only responsible for convergence service control; and then migrating the sub-service to be migrated to the cloud server according to the service migration request, and receiving configuration completion information returned after the cloud server configures the sub-service to be migrated.
Furthermore, after the migration of the sub-service to be migrated is completed, the corresponding log collection logic will also change, so that the log collection of the hybrid cloud architecture can be supported. Based on this, the log collection logic under the hybrid cloud architecture is shown in fig. 3, the SLB provided by the cloud server is directly communicated with the access service Frog, completes log collection by combining with fluent, and then sends the log to kafka, and then preprocesses the collected log according to a preset log processing strategy and then sends the log back to the HDFS for collection, so as to ensure that collection and processing of the log are supported under the hybrid cloud architecture, and normal operation of a target service is ensured; that is to say, the log collection process is implemented according to the schematic diagram shown in fig. 4 (a scenario including two cloud servers), after the logs to be processed are collected by the cloud server 1 and the cloud server 2, the logs to be processed are merged, compressed, and filtered through the EMR module, so as to reduce the volume of the logs to be processed, obtain a target log according to a processing result, and finally return the target log to the target service end by using the dedicated bandwidth resource, so that the target service end can summarize the target logs returned by the cloud server 1 and the cloud server 2, so as to be used and processed by the downstream service.
After the target sub-service is migrated to the cloud server, the cloud service information associated with the target service is acquired at this time, so that a new log acquisition link may be subsequently created for the current hybrid cloud architecture in combination with the cloud service information, in this embodiment, a process of acquiring the cloud service information is as follows:
analyzing the service configuration completion information to obtain a service processing type corresponding to the sub-service to be migrated;
and determining cloud service information according to the service processing type.
Specifically, after the migration processing of the sub-service to be migrated to the cloud server is completed, the service configuration completion information returned by the cloud server is received, and in order to ensure that a subsequently created log acquisition link is matched with the deployment situation of the current migration scenario, the service configuration completion information may be analyzed to obtain a service processing type corresponding to the sub-service to be migrated, where the service processing type specifically refers to a type corresponding to data processed when the sub-service to be migrated runs. The cloud service information can be determined according to the service processing type, so that a log acquisition link capable of acquiring logs under the current hybrid cloud architecture can be created conveniently by combining the cloud service information.
According to the above embodiment, after the cloud server receives the service configuration completion information for watching the video lottery activity, the service configuration completion information can be analyzed, and it is determined that the service processing type corresponding to the sub-service to be migrated comprises the user behavior type, the service processing type and the database type, and then the cloud service information can be determined by combining the user behavior type, the service processing type and the database type, so that log acquisition links corresponding to three different service processing types can be created by combining the cloud service information later, accurate collection of logs generated in the process of watching the video lottery activity is realized, and the service side can collect and audit the logs conveniently.
In conclusion, by determining the cloud service information in combination with the service processing type of the sub-service to be migrated, the conformity between the subsequently created log acquisition link and the current hybrid cloud architecture can be effectively ensured, so that the log can be accurately acquired when the target service runs.
In addition, after the migration of the sub-service to be migrated is completed, the calculation amount that can be supported by the current hybrid architecture is significantly increased, but in order to further increase the data amount that can be processed when the target service runs in a specific time, a degradation policy may be further adopted to modify the sub-service, in this embodiment, the specific implementation manner is as in steps S1022 to S1024:
step S1022, the service weight of the target sub-service included in the target service is read according to the service migration request.
Specifically, the target sub-service specifically refers to a sub-service included in the target service, and the sub-service provides different functions for the user, and if the target service is a video lottery watching activity, the target sub-service may be a log mirror image processing service or a log buffering service; and different target sub-services respectively correspond to different service processing logics when the target service runs. Correspondingly, the service weight specifically represents an index of the importance degree of each target sub-service in the target service, and the larger the service weight is, the larger the function of the target sub-service in the target service is, and the higher the importance degree is; on the contrary, the smaller the service weight is, the smaller the role played by the target sub-service in the target service is, and the lower the importance degree is; the action size is an action for representing whether the sub-service can influence the operation of the main service, and the action is larger when the influence is larger, and the action is smaller when the influence is smaller.
Based on this, in the migration processing process of the sub-services to be migrated, in order to support the normal operation of the target service under the current hybrid cloud architecture, the service weight of each target sub-service may be read according to the request, so that the subsequent selective closing or deletion of part of the sub-services according to the requirements is realized, and thus the resource utilization rate is improved without affecting the main operation logic of the target service.
Step S1024, each target sub-service contained in the target service is processed based on the preset degradation strategy and the service weight, and the target service is updated according to the processing result.
Specifically, the degradation policy specifically refers to a policy corresponding to reducing the influence of part of the sub-services on the target service while improving the system processing capability corresponding to the target service; based on this, since the service weights corresponding to different target sub-services reflect the size of the influence of the different target sub-services on the target service, when the target sub-services are processed, each target sub-service can be processed by combining a preset degradation strategy and the service weights, wherein the processing includes closing, deleting, modifying and the like, so as to avoid that part of the target sub-services occupy excessive computing resources, and meanwhile, the normal operation of the target service is not influenced, and finally, the target is updated according to the processing result.
Further, when the target sub-service is processed by combining the service weight and the degradation policy, different processing manners may be selected for different sub-services to maximize the service processing performance, and in this embodiment, the specific implementation manner is as follows:
extracting a sub-service set to be closed, a sub-service set to be deleted and/or a sub-service set to be modified from the target service based on the service weight;
and processing the sub-service set to be closed, the sub-service set to be deleted and/or the sub-service set to be modified by using a preset degradation strategy, and updating the target service according to a processing result.
Specifically, the sub-service set to be closed specifically refers to a set of target sub-services that can be temporarily closed before service migration processing is performed, and closing the target sub-services does not affect the target services; the sub-service set to be deleted specifically refers to a set formed by target sub-services which can be deleted before service migration processing is performed, and the target sub-services are deleted without influencing the target services; the sub-service specific set to be modified is a set formed by target sub-services which can be modified according to migration requirements before service migration is carried out, so that the target services are prevented from being influenced while the target services are more in fit.
Based on this, after the service weight of each target sub-service is determined, a sub-service set to be closed can be screened from the target service according to the size of the service weight, the sub-service set to be deleted and/or the sub-service set to be modified are/is processed by using a preset degradation strategy, and finally, the target service is updated according to the processing result.
In practical application, the step of processing the target sub-service included in the sub-service set to be closed means that the target sub-service is closed within a set time corresponding to the target service; the step of processing the target sub-service contained in the sub-service set to be deleted means that the part of the target sub-service is deleted within the set time corresponding to the target service; the step of processing the target sub-service contained in the sub-service set to be modified means that the target sub-service is modified within the set time corresponding to the target service, so that the target service under the hybrid cloud architecture is prevented from being affected.
In addition, in consideration of the influence on the deployment of the target service under the hybrid cloud architecture, in order to support the normal operation of the target service under the hybrid cloud architecture, the sub-service can be added, and the added sub-service can improve the service capability of the target service under the hybrid cloud architecture, so that the target service is supported to support a more complex service processing scene.
Along with the above example, before the sub-service to be migrated needs to be migrated to the cloud server, in order to control the improvement of the utilization rate of the dedicated bandwidth during log acquisition, fac (shunt control) sub-services can be added for video lottery activities, so as to achieve the purpose of shunting data transmission; meanwhile, the service weight of each sub-service related to the video lottery activity can be read according to the service migration request, the service weight of the sub-service of the Http log is determined to be 0.6, the service weight of the sub-service of the Graylog (log monitoring system) is determined to be 0.2, and the service weight of the sub-service of the Sentry (log reporting system) is determined to be 0.15; then, based on the service weight of each sub-service, determining the sub-service to be closed (service weight interval [ 0.3-0.7 ]) as an Http log; the sub-services to be deleted (service weight interval [ 0-0.3 ]) are Graylog and Sentry; the sub-service to be added is "fac". The service weight corresponding to each sub-service is used for representing the importance degree of the sub-service in the video lottery activity, the larger the weight value is, the higher the importance degree of the sub-service is represented, and otherwise, the smaller the weight value is, the lower the importance degree of the sub-service is represented; further, the sub-service to be deleted specifically refers to a sub-service that does not affect the video lottery activity, and may be a sub-service that is subjected to deletion processing; the sub-service to be closed specifically refers to the video lottery activity opening time, and the switch and the sub-service to be closed are controlled according to the data volume of the activity, namely the sub-service which can be temporarily closed when the data volume is large and can be opened when the data volume is small.
After the sub-services to be processed are determined, each sub-service is processed according to a preset degradation strategy, namely, the configuration of 'fac' is added in the target service, so that shunting can be performed on the day of activity, Graylog and Sentry are deleted, and an 'Http log' is temporarily closed, so that the consumption of computing resources is reduced, and more resource use is provided for a core processing process on the day of activity.
In summary, in order to ensure that richer computing resources are provided for the target service at the set time, different target sub-services may be processed in combination with the degradation policy and the service weight, and the target service is updated according to the processing result, so as to complete the log collection task at the set time.
Step S104, determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service.
Specifically, on the basis of obtaining the cloud service information associated with the target service, further, since the service content related to the target service may be complex, that is, more service services are provided for users or enterprises, and further, more log types need to be collected in the running process of the target service, in order to create a log collection link adapted to the current hybrid cloud architecture, the log type of at least one log to be collected may be preferentially determined based on the cloud service information and the service information of the target service, so as to achieve that different log collection links may be subsequently created for different log types.
The service information is specifically information related to service provided by a target service to a user or an enterprise, and by integrating cloud service information and the service information, a log type corresponding to a log acquisition link needing to be rebuilt under a hybrid cloud architecture can be determined, namely the log type of the hybrid cloud architecture with log acquisition process change is determined; correspondingly, at least one log to be collected is the log to be collected mounted on the hybrid cloud architecture. Wherein the log type includes at least one of: database log type, service log type, user behavior log type.
Based on the above, the database log type specifically refers to a type corresponding to a log generated by a db (data base) database corresponding to a target sub-service migrated to a cloud server under a hybrid cloud architecture; the service log type specifically refers to a type corresponding to a log generated in the running process of the target service; the user behavior log type specifically refers to a type corresponding to a log generated when a user participates in a target service.
In practical application, the logs of different log types can be collected only by adopting different collecting modes, so that the condition that a business party can collect and audit the logs of different types can be met. Meanwhile, as the target service is operated based on the hybrid cloud architecture, the log acquisition link which needs to be regenerated is also a link influenced by the hybrid cloud architecture, that is, at least one log to be acquired can be determined based on the cloud service information and the service information, and the log type of the at least one log to be acquired is further determined, and the acquisition link corresponding to the log type can be determined to be the acquisition link influenced by the hybrid cloud architecture, so that the acquisition link of the log type needs to be reconstructed.
Further, in the process of determining the log type, considering that the log to be collected is a log influenced by the hybrid cloud architecture and the types of logs related to the target service are more, the log to be collected may be determined when the target service operates in the hybrid cloud architecture, and the log type is determined based on the determined log type, in this embodiment, a specific implementation manner is as follows:
acquiring service information of a target service, and determining a cloud server associated with the target service according to the cloud service information;
under the condition that the target service runs on the cloud server, determining at least one log to be collected, and determining a log type group corresponding to the target service according to the service information;
the log type is determined from the log type group based on at least one log to be collected.
Specifically, the log type group refers to a set formed by types corresponding to logs that can be generated when the target service runs under the hybrid cloud architecture.
Based on the method, in order to accurately determine the type of the target service, firstly, the service information of the target service is obtained, and meanwhile, the cloud server related to the target service is determined according to the cloud service information; secondly, under the condition that the target service runs on the cloud server, determining at least one log to be acquired generated by the target service at the moment, and determining a log type group corresponding to the target service according to the service information; at least one log to be collected generated at the moment is generated based on the hybrid cloud architecture, so that the log type can be determined subsequently; and finally, screening out the log type from the log type group based on at least one log to be collected.
According to the above example, after determining the cloud service information corresponding to the video lottery activity, the video lottery activity may be operated in the hybrid cloud architecture, it is determined according to the operation result that a DB database log, a service log, and a user behavior log are generated, based on three groups of logs generated by the service operation, it is determined that the video lottery activity in the hybrid cloud architecture corresponds to three log types, which are a database log type, a service log type, and a user behavior log type, and subsequently, when a log collection link is created, the three log types may be created, so that the log collection of the video lottery activity may be completed in full on the same day of the activity.
In summary, the log type is determined by combining the service information and the cloud service information, so that the log type corresponding to the log acquisition link needing to be reconstructed is accurately positioned, and the link can be conveniently created from the log type in the follow-up process until the target service under the hybrid cloud architecture normally runs.
And step S106, creating a log collection node corresponding to the log type according to preset log collection configuration information.
Specifically, after the log types are determined, further, because the service contents corresponding to different log types are also different, when creating the log collection link, different log collection links need to be created for different log types. Before this, the nodes of the log collection link to be created need to be determined, so that the log collection link can be formed in a mode of integrating the nodes subsequently.
Based on this, the preset log collection configuration information specifically refers to node-related information which is configured in advance for different log types, and after the log type is determined, the log collection node can be directly determined from the log collection configuration information according to the log type for subsequently creating a log collection link; correspondingly, the log collection node is a node forming a log collection link corresponding to the log type, and different nodes reflect different processing during log collection.
Furthermore, in the process of determining the log collection nodes, the nodes are considered to be important factors for completing log collection and processing, so that the log collection nodes need to be determined one by one; and influenced by the hybrid cloud architecture, it is further required to ensure that the log collection node can effectively communicate the service on the cloud with the service under the cloud, and in this embodiment, the log collection node is determined as follows:
reading preset log acquisition configuration information based on the log type to obtain node configuration information corresponding to the log type;
and creating an on-cloud log processing node and an off-cloud log processing node according to the node configuration information, and using the on-cloud log processing node and the off-cloud log processing node as log acquisition nodes corresponding to log types.
Specifically, the node configuration information specifically refers to information required for creating a node corresponding to the log type, such as determining related information of an initial log collection node, determining related information of a log collection node flow, determining related information of the number of log collection links corresponding to the log type, and the like; correspondingly, the on-cloud log processing node is specifically a log acquisition node deployed in a cloud server under a hybrid cloud architecture; the log processing node under the cloud is a log collection node deployed in the local.
Based on the method, firstly, preset log acquisition configuration information is read according to the log type to determine node configuration information corresponding to the log type, then, an on-cloud log processing node and an off-cloud log processing node are created according to the node configuration information to ensure that a subsequently created log acquisition link can be adapted to a hybrid cloud architecture, and finally, the on-cloud log processing node and the off-cloud log processing node are used as log acquisition nodes corresponding to the log type to be used for subsequently creating the log acquisition link aiming at the log type.
According to the above example, on the basis of determining that the log types include a database log type, a service log type and a user behavior log type, preset log acquisition configuration information can be respectively read based on each log type, and node configuration information corresponding to the database log type is determined to serve as { an initial log processing node-cloud server DB according to a reading result; log collection mode-DTS incremental synchronization; intermediate log processing node 1-kafka on cloud; intermediate log processing node 2-cloud Flink; backhaul-private line bandwidth; end log processing node 1-IDC _ kafka; end log processing node 2-IDC _ flex; summarizing-real-time warehouse }, and determining log collection nodes corresponding to the log collection nodes to comprise { cloud server DB service according to the configuration information of the nodes corresponding to the database log types; DTS increment synchronization; kafka on the cloud; flink on the cloud; the bandwidth of the private line; IDC _ kafka; IDC _ Flink; real time bin }. Correspondingly, determining a corresponding log collection node according to the node configuration information corresponding to the service log type, wherein the log collection node comprises { cloud server k8s service; collecting EMR by using a log fluent; log processing (compression + filtering); the bandwidth of the private line; merging task dependencies; downstream T +1 calculation }; determining log acquisition nodes corresponding to the user behavior log type according to node configuration information corresponding to the user behavior log type, wherein the log acquisition nodes comprise { frog vm; acquiring fluent; EMR; processing logs; the bandwidth of the private line; IDC HDFS; distributing offline data; calculating downstream T + 1; kafka on the cloud; flink on the cloud; IDC _ kafka; IDC _ Flink real-time calculation; DB/OLAP; split log }; after determining the log collection nodes corresponding to the log types respectively, the method can be used for creating subsequent log collection links.
In summary, by creating the log processing node on the cloud and the log processing node under the cloud corresponding to the log type by combining the node configuration information, it can be effectively ensured that the subsequently created log acquisition link is more suitable for the hybrid cloud architecture.
And step S108, generating a log collection link corresponding to the log type based on the log collection node.
Specifically, on the basis of the log collection node for determining the log type, a log collection link corresponding to the log type can be obtained by further integrating the log collection nodes, where the log collection link specifically refers to a link that needs to collect logs generated by a service when a target service operates under a hybrid cloud architecture, and the log collection mode, the log processing process, and the log transmission process can be determined through the log collection link.
Further, in the process of creating the log collection link, because the number of log collection nodes corresponding to different log types is not fixed, and the order of the log collection nodes in the link is also different, the creation of the log collection link needs to be completed in combination with the service structure information of the target service, in this embodiment, the specific implementation manner is as follows:
determining service structure information corresponding to a target service;
and integrating the log processing nodes on the cloud and the log processing nodes under the cloud according to the service structure information to obtain a log acquisition link corresponding to the log type.
Specifically, the service structure information specifically refers to information corresponding to a service function architecture of the target service, and the arrangement order of the log collection nodes can be determined through the service structure information. Based on this, after the log collection node is determined, the service structure information of the target service can be determined, then the log processing node on the cloud and the log processing node under the cloud serving as the log collection node are integrated according to the service structure information, and the log collection link corresponding to the log type can be obtained according to the integration result.
According to the above example, after determining log acquisition nodes corresponding to the database log type, the service log type and the user behavior log type, the log acquisition nodes of the log types can be integrated according to the service structure information for watching the video lottery activity, and a log acquisition link _ DB corresponding to the database log type is determined to be { cloud server DB service- > DTS increment synchronization- > cloud kafka- > cloud Flink- > private line bandwidth- > IDC _ kafka- > IDC _ Flink- > real-time bin }; determining a log collection link _ B corresponding to the service log type as { cloud server k8s service- > log fluent collection EMR- > log processing (compression + filtering) - > dedicated line bandwidth- > merging task dependence- > downstream T +1 calculation }; determining a log collection link _ User1 corresponding to the User behavior log type as { front vm- > fluent collection- > EMR- > log processing- > special line bandwidth- > IDC HDFS- > offline data diversion- > downstream T +1 calculation }, determining a log collection link _ User2 as { front vm- > fluent collection- > on-cloud kafka- > on-cloud flex- > special line bandwidth- > IDC HDFS- > offline data diversion- > downstream T +1 calculation }, determining a log collection link _ User3 as { front vm- > fluent collection- > on-cloud kafka- > on-cloud flex- > special line bandwidth- > IDC _ flex real-time calculation) - > DB/OLAP }, determining a log collection link _ User4 as { front vm- > on-cloud vm- > shunt- > log collection- > IDC- > HDFS- > downstream T +1 calculation }, and determining a log collection link _ User behavior log collection link 4 as { front vm- > on-cloud collection link- > IDC. The log collection link _ User5 is { frog vm- > shunt log- > fluent collection- > private bandwidth- > IDC _ kafka- > IDC _ flex real-time calculation- > DB/OLAP }. Furthermore, after the link is established and the target service runs, the log can be collected and processed according to the log collection links corresponding to the log types, so that the collection of the current full-volume log of the activity is supported.
In addition, after the log collection link is created, the log collection link may be used to collect the log, and since the target service operates under the hybrid cloud architecture, in order to improve the efficiency of log collection, the log collection may be preprocessed after the log collection is completed, so as to improve the log collection efficiency and the log processing efficiency, in this embodiment, the specific implementation manner is as in steps S1082 to S1086:
and step S1082, acquiring an initial log corresponding to the log type based on the log acquisition link.
Step S1084, the initial log is preprocessed to obtain a target log, and the target log is returned.
Specifically, the initial log refers to an initial log corresponding to a log type and acquired according to a log acquisition link; the target log is a log obtained by preprocessing an initial log, wherein the preprocessing is to clean the initial log, namely delete redundant logs and compress and filter the log, so that the volume of the log is reduced.
Based on this, after the log collection link corresponding to the log type is obtained, the target service is operated, that is, the initial log corresponding to the log type can be collected through the log collection link, and since the target service is operated in the hybrid cloud architecture, in order to improve the utilization rate of the dedicated bandwidth and reduce the excessive waste of computing resources by redundant logs, the initial log can be preprocessed to obtain the target log, and then the target log is returned for the processing of the downstream service.
Further, in the process of processing the initial log as the target log, in order to fully utilize the dedicated bandwidth under the hybrid cloud architecture, the initial log may be compressed and filtered, and in this embodiment, the specific implementation manner is as follows:
filtering the initial log to obtain an intermediate log;
and merging the intermediate logs, and compressing the merged intermediate logs to obtain a target log.
Specifically, the intermediate log specifically refers to a log obtained by filtering the initial log. Based on this, after the initial log is obtained, the initial log may be filtered to propose repeated log contents in the initial log to be deleted to obtain an intermediate log, then the intermediate log is merged, and the merged intermediate log is compressed to reduce the volume of the log, and finally the target log may be obtained according to the compression result.
By filtering and compressing the collected initial logs, the volume of the logs can be effectively reduced, so that the utilization rate of the special line bandwidth is reduced when the subsequent transmission is facilitated, and the purpose of saving transmission resources is achieved.
And step S1086, determining a log storage address according to the log type, and writing the returned target log into the log storage address.
Specifically, after the target logs are obtained, because different log types correspond to different target logs, a log storage address is determined according to the log types, so that the returned target logs are written into the log storage address, thereby ensuring that the same logs can be gathered together, and facilitating the processing of downstream services. The log storage address specifically refers to an address for locally storing different types of logs.
Taking the log type as the service log type as an example for explanation, at this time, it is determined that the acquisition link corresponding to the service log type is { cloud server k8s service- > log fluent acquisition EMR- > log processing (compression + filtering) - > dedicated bandwidth- > merging task dependency- > downstream T +1 calculation }; based on this, firstly, the service log corresponding to the service log type is obtained, then the service log is written into the EMR cluster through the fluent on the k8s service instance according to a minute-level writing mode, and then the service log is processed in the EMR cluster, wherein the processing process includes: filtering and compressing the service log to obtain a target service log with a small volume; namely, the service log is 128M, and a target service log with the size of 20M is obtained by merging, filtering and compressing; and then the EMR cluster starts a distcp task, and returns the target service log to the local service system f04 through the dedicated line bandwidth so as to write the target service log into a temporary directory of a service log type corresponding to f04, so that the target service log is used for downstream service to perform calculation processing in a T +1 period.
In summary, in order to improve the utilization rate of the private bandwidth under the hybrid cloud architecture, after the initial log is collected, the initial log is preprocessed, so that the volume of the log is reduced, the log transmission efficiency is improved, and meanwhile, the log processing efficiency of downstream services can be accelerated.
In addition, because the target service is supported by the hybrid cloud architecture to operate, in order to avoid the problem of target service blocking or crashing in a set time, after the log acquisition link is established, a pressure test may be performed on the log acquisition link to implement reasonable optimization of the current system, in this embodiment, a specific implementation manner is as follows:
obtaining a sample log;
performing pressure test on the target service based on the sample log and the log acquisition link to obtain a pressure test result;
and optimizing the cloud server and the local server which bear the target service according to the pressure test result.
Specifically, the sample log refers to a log prepared by performing a pressure test on a target service in combination with a log acquisition link; correspondingly, the pressure test specifically refers to a test that the test target service can bear the load capacity; and the pressure test result is the pressure test information of the target service.
Based on this, when a pressure test is to be performed, a sample log can be obtained according to a requirement, and then the sample log and the log acquisition link are used for performing the pressure test on the target service, that is, the sample log is processed once or many times according to the execution logic of the log acquisition link, so as to reflect the pressure test result of the target service, and it needs to be noted that in the process, the target service can be tested in a mode of continuously increasing the data volume of the sample log; and finally, optimizing the cloud server and the local server which bear the target service according to the pressure test result so as to realize that a hybrid cloud architecture formed by the cloud server and the local server can support more complex service scenes and improve the load capacity of the target service.
In practical application, the index of the pressure test may be selected according to actual requirements, and may be used to reflect the actual situation of the target service, and this embodiment is not limited herein. The sample log used in the process can simulate high concurrency in a multithreading and multiprocessing mode, so that the bearing capacity of the target service in a real scene is simulated.
Further, the cloud server and the local server are optimized, which may be an optimization event that compression cost is high due to data repetition, an optimization event that a CPU preempts, or an optimization event that an SLB load is unbalanced. In specific implementation, the event that can be optimized can be determined according to an actual application scenario, and this embodiment is not limited in any way here.
According to the link creation method based on the hybrid cloud architecture, after the cloud service information associated with the target service is obtained, the log type of at least one log to be collected can be determined according to the cloud service information and the service information of the target service, then the log collection node corresponding to the log type is created according to the preset log collection configuration information, the log collection link corresponding to the log type can be generated based on the log collection node, the latest log collection link can be created aiming at any service architecture, the link is determined based on the log type of the log to be collected, it is guaranteed that the created link is targeted, therefore, the log collection strategy under the current service architecture is met, and accurate collection of the log corresponding to the log type is achieved during service processing.
The following describes, with reference to fig. 5, a link creation method based on a hybrid cloud architecture further by taking an application of the link creation method based on the hybrid cloud architecture provided in this specification in a log collection scenario as an example. Fig. 5 shows a processing flow chart of a link creation method based on a hybrid cloud architecture, which is applied to a log collection scenario and provided by an embodiment of the present specification, and specifically includes the following steps:
step S502, receiving a service transfer request submitted by a service party for the lottery event service.
Step S504, the sub-service to be migrated is lifted in the lottery activity service according to the service migration request, and the cloud server bearing the sub-service to be migrated is determined.
Step S506, the sub-service to be migrated is uploaded to the cloud server, and service configuration completion information returned by the cloud server is received.
Step S508, determining a service processing type corresponding to the sub-service to be migrated according to the service configuration completion information, and determining cloud server information according to the service processing type.
Step S510, acquiring service information corresponding to the lottery activity service, and determining the log type of the log to be collected according to the service information and the cloud service information under the condition that the lottery activity service runs in the cloud server.
And step S512, under the condition that the log type is the user behavior log type, determining node configuration information based on preset log acquisition configuration information.
Step S514, creating a log processing node on the cloud and a log processing node under the cloud according to the node configuration information, and using the log processing nodes on the cloud and the log processing node under the cloud as log acquisition nodes corresponding to the user behavior log types.
Step S516, integrating the log collection nodes according to the service structure information corresponding to the lottery activity service to obtain a log collection link corresponding to the user behavior log type.
Step S518, a user behavior log corresponding to the user behavior log type is collected based on the log collection link.
Step S520, pre-process the user behavior log to obtain a target user behavior log, and return the target user behavior log.
Step S522, determining a log storage address according to the user behavior log type, and writing the returned target user behavior log into the log storage address.
The link creation method based on the hybrid cloud architecture realizes that the latest log collection link can be created for any service architecture, and the link is determined based on the log type of the log to be collected, so that the created link is ensured to be targeted, thereby meeting the log collection strategy under the current service architecture and realizing accurate collection of the log corresponding to the log type during service processing.
Corresponding to the above method embodiment, the present specification further provides a link creation apparatus embodiment based on a hybrid cloud architecture, and fig. 6 shows a schematic structural diagram of a link creation apparatus based on a hybrid cloud architecture provided in an embodiment of the present specification. As shown in fig. 6, the apparatus includes:
an obtaining module 602 configured to obtain cloud service information associated with a target service;
a determining module 604, configured to determine a log type of at least one log to be collected according to the cloud service information and the service information of the target service;
a creating module 606 configured to create a log collection node corresponding to the log type according to preset log collection configuration information;
a generating module 608 configured to generate a log collection link corresponding to the log type based on the log collection node.
In an optional embodiment, the apparatus further comprises:
a migration module configured to receive a service migration request submitted for the target service; extracting sub-services to be migrated from the target services according to the service migration request, and determining a cloud server for bearing the sub-services to be migrated; and uploading the sub-service to be migrated to the cloud server, and receiving service configuration completion information returned by the cloud server.
In an optional embodiment, the obtaining module 602 is further configured to:
analyzing the service configuration completion information to obtain a service processing type corresponding to the sub-service to be migrated; and determining the cloud service information according to the service processing type.
In an optional embodiment, the determining module 604 is further configured to:
acquiring the service information of the target service, and determining a cloud server associated with the target service according to the cloud service information; under the condition that the target service runs on the cloud server, determining at least one log to be collected, and determining a log type group corresponding to the target service according to the service information; determining the log type from the log type group based on the at least one log to be collected.
In an optional embodiment, the log type includes at least one of:
database log type, service log type, user behavior log type.
In an optional embodiment, the creating module 606 is further configured to:
reading the preset log acquisition configuration information based on the log type to obtain node configuration information corresponding to the log type; and creating an on-cloud log processing node and an off-cloud log processing node according to the node configuration information, and using the on-cloud log processing node and the off-cloud log processing node as the log acquisition node corresponding to the log type.
In an optional embodiment, the generating module 608 is further configured to:
determining service structure information corresponding to the target service; and integrating the log processing nodes on the cloud and the log processing nodes under the cloud according to the service structure information to obtain the log acquisition link corresponding to the log type.
In an optional embodiment, the apparatus further comprises:
the log processing module is configured to acquire an initial log corresponding to the log type based on the log acquisition link; preprocessing the initial log to obtain a target log, and returning the target log; and determining a log storage address according to the log type, and writing the returned target log into the log storage address.
In an optional embodiment, the log processing module is further configured to:
filtering the initial log to obtain an intermediate log; and merging the intermediate logs, and compressing the merged intermediate logs to obtain the target log.
In an optional embodiment, the apparatus further comprises:
a pressure testing module configured to obtain a sample log; performing pressure test on the target service based on the sample log and the log acquisition link to obtain a pressure test result; and optimizing the cloud server and the local server which bear the target service according to the pressure test result.
In an optional embodiment, the apparatus further comprises:
the updating module is configured to read the service weight of the target sub-service contained in the target service according to the service migration request; and processing each target sub-service contained in the target service based on a preset degradation strategy and the service weight, and updating the target service according to a processing result.
In an optional embodiment, the update module is further configured to:
extracting a sub-service set to be closed, a sub-service set to be deleted and/or a sub-service set to be modified in the target service based on the service weight; and processing the sub-service set to be closed, the sub-service set to be deleted and/or the sub-service set to be modified by using the preset degradation strategy, and updating the target service according to a processing result.
In the link creation apparatus based on the hybrid cloud architecture provided in this embodiment, after the cloud service information associated with the target service is acquired, the log type of at least one log to be collected can be determined according to the cloud service information and the service information of the target service, then the log collection node corresponding to the log type is created according to the preset log collection configuration information, the log collection link corresponding to the log type can be generated based on the log collection node, the latest log collection link can be created aiming at any service architecture, and the link is determined based on the log type of the log to be collected, thereby ensuring that the creation of the generated link is targeted, therefore, the log collection strategy under the current service architecture is met, and the accurate collection of the log corresponding to the log type is realized during service processing.
The above is an illustrative scheme of a link creation apparatus based on a hybrid cloud architecture according to this embodiment. It should be noted that the technical solution of the link creating apparatus based on the hybrid cloud architecture and the technical solution of the link creating method based on the hybrid cloud architecture belong to the same concept, and details of the technical solution of the link creating apparatus based on the hybrid cloud architecture, which are not described in detail, can be referred to the description of the technical solution of the link creating method based on the hybrid cloud architecture.
Fig. 7 is a block diagram of a computing device 700 provided in accordance with an embodiment of the present disclosure. The components of the computing device 700 include, but are not limited to, memory 710 and a processor 720. Processor 720 is coupled to memory 710 via bus 730, and database 750 is used to store data.
Computing device 700 also includes access device 740, access device 740 enabling computing device 700 to communicate via one or more networks 760. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 740 may include one or more of any type of network interface, e.g., a Network Interface Card (NIC), wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present description, the above-described components of computing device 700, as well as other components not shown in FIG. 7, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 7 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.
Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 700 may also be a mobile or stationary server.
Wherein processor 720 is configured to execute the following computer-executable instructions:
acquiring cloud service information associated with a target service;
determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
creating a log collection node corresponding to the log type according to preset log collection configuration information;
and generating a log collection link corresponding to the log type based on the log collection node.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the link creation method based on the hybrid cloud architecture belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the link creation method based on the hybrid cloud architecture.
An embodiment of the present specification also provides a computer readable storage medium storing computer instructions that, when executed by a processor, are operable to:
acquiring cloud service information associated with a target service;
determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
creating a log collection node corresponding to the log type according to preset log collection configuration information;
and generating a log collection link corresponding to the log type based on the log collection node.
The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the link creation method based on the hybrid cloud architecture belong to the same concept, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the link creation method based on the hybrid cloud architecture.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present disclosure is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present disclosure. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for this description.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the specification and its practical application, to thereby enable others skilled in the art to best understand the specification and its practical application. The specification is limited only by the claims and their full scope and equivalents.

Claims (15)

1. A link creating method based on a hybrid cloud architecture is characterized by comprising the following steps:
acquiring cloud service information associated with a target service;
determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
creating a log collection node corresponding to the log type according to preset log collection configuration information;
and generating a log collection link corresponding to the log type based on the log collection node.
2. The method of claim 1, wherein before the step of obtaining cloud service information associated with the target service is performed, the method further comprises:
receiving a service migration request submitted aiming at the target service;
extracting sub-services to be migrated from the target services according to the service migration request, and determining a cloud server for bearing the sub-services to be migrated;
and uploading the sub-service to be migrated to the cloud server, and receiving service configuration completion information returned by the cloud server.
3. The method of claim 2, wherein the obtaining cloud service information associated with the target business comprises:
analyzing the service configuration completion information to obtain a service processing type corresponding to the sub-service to be migrated;
and determining the cloud service information according to the service processing type.
4. The method according to claim 1, wherein the determining a log type of at least one log to be collected according to the cloud service information and the service information of the target service comprises:
acquiring the service information of the target service, and determining a cloud server associated with the target service according to the cloud service information;
under the condition that the target service runs on the cloud server, determining at least one log to be collected, and determining a log type group corresponding to the target service according to the service information;
determining the log type from the log type group based on the at least one log to be collected.
5. The method of any of claims 1 to 4, wherein the log type comprises at least one of:
database log type, service log type, user behavior log type.
6. The method according to claim 1, wherein the creating of the log collection node corresponding to the log type according to the preset log collection configuration information includes:
reading the preset log acquisition configuration information based on the log type to obtain node configuration information corresponding to the log type;
and creating an on-cloud log processing node and an off-cloud log processing node according to the node configuration information, and using the on-cloud log processing node and the off-cloud log processing node as the log acquisition node corresponding to the log type.
7. The method of claim 6, wherein the generating a log collection link corresponding to the log type based on the log collection node comprises:
determining service structure information corresponding to the target service;
and integrating the log processing nodes on the cloud and the log processing nodes under the cloud according to the service structure information to obtain the log acquisition link corresponding to the log type.
8. The method according to claim 1, wherein after the step of generating the log collection link corresponding to the log type based on the log collection node is executed, the method further comprises:
acquiring an initial log corresponding to the log type based on the log acquisition link;
preprocessing the initial log to obtain a target log, and returning the target log;
and determining a log storage address according to the log type, and writing the returned target log into the log storage address.
9. The method of claim 8, wherein preprocessing the initial log to obtain a target log comprises:
filtering the initial log to obtain an intermediate log;
and merging the intermediate logs, and compressing the merged intermediate logs to obtain the target log.
10. The method according to claim 8, wherein after the step of generating the log collection link corresponding to the log type based on the log collection node is executed, the method further comprises:
obtaining a sample log;
performing pressure test on the target service based on the sample log and the log acquisition link to obtain a pressure test result;
and optimizing the cloud server and the local server which bear the target service according to the pressure test result.
11. The method of claim 2, wherein after the step of receiving the service migration request submitted for the target service is executed, the method further comprises:
reading the service weight of the target sub-service contained in the target service according to the service migration request;
and processing each target sub-service contained in the target service based on a preset degradation strategy and the service weight, and updating the target service according to a processing result.
12. The method according to claim 11, wherein the processing each target sub-service included in the target service based on a preset degradation policy and the service weight, and updating the target service according to a processing result includes:
extracting a sub-service set to be closed, a sub-service set to be deleted and/or a sub-service set to be modified in the target service based on the service weight;
and processing the sub-service set to be closed, the sub-service set to be deleted and/or the sub-service set to be modified by using the preset degradation strategy, and updating the target service according to a processing result.
13. A link creation apparatus based on a hybrid cloud architecture, comprising:
the acquisition module is configured to acquire cloud service information associated with the target business;
the determining module is configured to determine the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
the creating module is configured to create a log collecting node corresponding to the log type according to preset log collecting configuration information;
and the generating module is configured to generate a log collection link corresponding to the log type based on the log collection node.
14. A computing device, comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the method of:
acquiring cloud service information associated with a target service;
determining the log type of at least one log to be collected according to the cloud service information and the service information of the target service;
creating a log collection node corresponding to the log type according to preset log collection configuration information;
and generating a log collection link corresponding to the log type based on the log collection node.
15. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 12.
CN202110643675.8A 2021-06-09 2021-06-09 Link creation method and device based on hybrid cloud architecture Active CN113382071B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110643675.8A CN113382071B (en) 2021-06-09 2021-06-09 Link creation method and device based on hybrid cloud architecture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110643675.8A CN113382071B (en) 2021-06-09 2021-06-09 Link creation method and device based on hybrid cloud architecture

Publications (2)

Publication Number Publication Date
CN113382071A true CN113382071A (en) 2021-09-10
CN113382071B CN113382071B (en) 2022-09-06

Family

ID=77573195

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110643675.8A Active CN113382071B (en) 2021-06-09 2021-06-09 Link creation method and device based on hybrid cloud architecture

Country Status (1)

Country Link
CN (1) CN113382071B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115174366A (en) * 2022-05-30 2022-10-11 阿里巴巴(中国)有限公司 Data processing method and device
WO2023207065A1 (en) * 2022-04-26 2023-11-02 华为云计算技术有限公司 Cloud service integration method and apparatus and related device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105681472A (en) * 2016-03-28 2016-06-15 微梦创科网络科技(中国)有限公司 Log transmission method and log transmission device
CN107426023A (en) * 2017-07-21 2017-12-01 携程旅游信息技术(上海)有限公司 Cloud platform log collection and retransmission method, system, equipment and storage medium
CN107612730A (en) * 2017-09-18 2018-01-19 山东浪潮云服务信息科技有限公司 A kind of log collection analysis method, device and system
CN112069145A (en) * 2020-08-31 2020-12-11 北京易捷思达科技发展有限公司 Log collection system and method suitable for large-scale cloud deployment environment
WO2020258290A1 (en) * 2019-06-28 2020-12-30 京东方科技集团股份有限公司 Log data collection method, log data collection apparatus, storage medium and log data collection system
CN112527618A (en) * 2020-12-17 2021-03-19 中国农业银行股份有限公司 Log collection method and log collection system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105681472A (en) * 2016-03-28 2016-06-15 微梦创科网络科技(中国)有限公司 Log transmission method and log transmission device
CN107426023A (en) * 2017-07-21 2017-12-01 携程旅游信息技术(上海)有限公司 Cloud platform log collection and retransmission method, system, equipment and storage medium
CN107612730A (en) * 2017-09-18 2018-01-19 山东浪潮云服务信息科技有限公司 A kind of log collection analysis method, device and system
WO2020258290A1 (en) * 2019-06-28 2020-12-30 京东方科技集团股份有限公司 Log data collection method, log data collection apparatus, storage medium and log data collection system
CN112069145A (en) * 2020-08-31 2020-12-11 北京易捷思达科技发展有限公司 Log collection system and method suitable for large-scale cloud deployment environment
CN112527618A (en) * 2020-12-17 2021-03-19 中国农业银行股份有限公司 Log collection method and log collection system

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023207065A1 (en) * 2022-04-26 2023-11-02 华为云计算技术有限公司 Cloud service integration method and apparatus and related device
CN115174366A (en) * 2022-05-30 2022-10-11 阿里巴巴(中国)有限公司 Data processing method and device
CN115174366B (en) * 2022-05-30 2023-10-20 浙江天猫技术有限公司 Data processing method and device

Also Published As

Publication number Publication date
CN113382071B (en) 2022-09-06

Similar Documents

Publication Publication Date Title
US8612406B1 (en) Sharing business data across networked applications
Saripalli et al. Madmac: Multiple attribute decision methodology for adoption of clouds
CN113382071B (en) Link creation method and device based on hybrid cloud architecture
US10684990B2 (en) Reconstructing distributed cached data for retrieval
CN110716744A (en) Data stream processing method, system and computer readable storage medium
CN106649088A (en) Distributed pressure test method and system
CN102150150A (en) Techniques for resource location and migration across data centers
CN105959168A (en) Techniques for network replication
CN113010565B (en) Server real-time data processing method and system based on server cluster
CN105468442A (en) Application and information movement in a cloud environment
CN106850761A (en) Journal file storage method and device
CN105515836A (en) Log processing method, device and server
CN114780301A (en) Disaster recovery method and system supporting multi-cloud production environment
CN113965587A (en) Data acquisition method, device, equipment and medium of artificial intelligence platform
CN115996230B (en) Cross-cloud data synchronization method and device, computer equipment and readable storage medium
CN107357922A (en) A kind of NFS of distributed file system accesses auditing method and system
CN113407491A (en) Data processing method and device
CN106534234A (en) Relationship chain processing system, method and device
CN115277853A (en) Data processing method and device based on edge cloud
CN115426216A (en) Internet of things node interconnection method based on complex virtual instrument software
CN115706727A (en) Cloud desktop data migration method, node and server
KR102093764B1 (en) Managment server for managing the server and storage
CN112583667A (en) Content delivery network link evaluation method and device
CN113391985A (en) Resource allocation method and device
CN111988368A (en) Data interaction system and interaction method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant