CN115510045A

CN115510045A - AI decision-based big data acquisition configuration method and intelligent scene system

Info

Publication number: CN115510045A
Application number: CN202211267590.5A
Authority: CN
Inventors: 周正涛; 吴正红
Original assignee: Individual
Current assignee: Individual
Priority date: 2022-04-13
Filing date: 2022-04-13
Publication date: 2022-12-23
Also published as: CN114691664A; CN114691664B

Abstract

The embodiment of the application provides a big data acquisition configuration method based on AI decision and a smart scene system, a target noise penetration area obtained by performing noise penetration decision on loaded big data of a target smart scene based on a target noise penetration decision model is obtained, big data cleaning field distribution of a big data acquisition process associated with the big data of the target smart scene is determined based on the target noise penetration area, big data cleaning configuration is performed on the big data acquisition process by combining the big data cleaning field distribution, big data cleaning configuration data is output, the big data acquisition process is updated by combining the big data cleaning configuration data, and therefore the updated big data acquisition process is loaded to a smart scene service server to perform big data acquisition configuration. Therefore, the big data cleaning configuration corresponding to the big data cleaning configuration rule is executed in combination with the noise penetration area, the big data acquisition configuration is updated in a targeted manner, the big data cleaning is carried out in the later big data acquisition process, and the generation of partial noise data is avoided.

Description

AI decision-based big data acquisition configuration method and intelligent scene system

The application is a divisional application of Chinese application with the application number of 202210381557.9, the application date of 2022, 13.04.13 entitled "cleaning method of intelligent scene big data based on AI prediction and intelligent scene system".

Technical Field

The application relates to the technical field of big data cleaning, in particular to a big data acquisition configuration method based on AI decision and an intelligent scene system.

Background

The intelligent scene service is oriented to scenes such as cities, enterprises and industries, and improves the internet service level through the fission effect generated by the deep fusion and innovation of ICT technologies such as 5G, cloud computing and AI and industry knowledge, so that the internet experience of users is stronger, the production efficiency of enterprises is higher, and the industrial creativity is stronger.

In the wisdom scene information service in-process, for the demand of accurate analysis follow-up user, enterprise, need carry out big data acquisition and further big data mining, however the noise data that influences the big data precision can be gathered to the big data acquisition in-process, causes the demand analysis probably to have the error, consequently in big data acquisition, needs carry out the big data washing in real time, and one of big data washing's thinking is carrying out the noise elimination, needs to analyze out the noise infiltration district. In the related art, the output of the noise penetration area by combining AI prediction is always in the research process, but the current difficulties are that it is difficult to better adapt to the limited data labels under the training data, the noise penetration learning performance is low, and the configuration precision guarantee of the subsequent big data cleaning process is also influenced.

Disclosure of Invention

In order to overcome at least the above disadvantages in the prior art, the present application provides a big data collection configuration method based on AI decision and an intelligent scene system.

In a first aspect, the present application provides an AI prediction-based intelligent scene big data cleaning method, which is applied to an intelligent scene system, where the intelligent scene system is in communication connection with a plurality of intelligent scene service servers, and the method includes:

calling basic template scene big data from an intelligent scene service database bound with the intelligent scene system in advance, wherein the basic template scene big data covers a plurality of basic template scene data with a root template noise penetration area calibrated;

performing noise penetration point mining on the basic template scene data in the basic template scene big data by combining a noise penetration decision model of initialization training, and outputting a noise penetration point cluster;

combining the noise penetration point clusters, and configuring a noise penetration relation space by taking each basic template scene data as an independent penetration noise penetration positioning branch;

performing infiltration area derivation on a root template noise infiltration area of the basic template scene data by combining the noise infiltration relation space, and outputting derived template scene big data;

and performing decision function functional layer adjustment on the noise penetration decision model of the initialization training based on the derived template scene big data, and performing noise penetration decision on the loaded target intelligent scene big data based on the target noise penetration decision model matched with the model deployment condition.

For example, a target noise penetration decision-making model based on the deployment condition of the matching model is obtained to perform noise penetration decision-making on the loaded target intelligent scene big data to obtain a target noise penetration area;

determining big data cleaning field distribution of a big data acquisition process related to the big data of the target intelligent scene based on the target noise penetration area;

performing big data cleaning configuration on the big data acquisition process by combining the big data cleaning field distribution, and acquiring traversal data acquisition node distribution obtained by performing traversal data acquisition on big data updating activities associated with the big data acquisition process in a big data updating state by service acquisition activities of the big data acquisition process, wherein the traversal data acquisition node distribution comprises a plurality of traversal data acquisition nodes generated according to the latest acquisition configuration sequence of a big data acquisition template;

aiming at each traversal data acquisition node in the traversal data acquisition node distribution, acquiring a target traversal data acquisition node in each single process and determining the target traversal data acquisition node as a target traversal data acquisition node, and acquiring a traversal data acquisition node linked with the target traversal data acquisition node and determining the target traversal data acquisition node as a linked traversal data acquisition node;

acquiring big data cleaning configuration information associated with a target big data updating process needing big data cleaning configuration;

acquiring field updating is carried out on the target traversal data acquisition node and the linkage traversal data acquisition node acquired in each single process by combining the big data cleaning configuration information, so that the big data cleaning configuration of the service acquisition activity for the big data updating process of each big data updating activity is carried out, big data cleaning configuration data are output, and the big data cleaning configuration data comprise noise cleaning field data corresponding to each traversal data acquisition table item;

and updating the big data acquisition process by combining the big data cleaning configuration data, and loading the updated big data acquisition process to the intelligent scene service server for big data acquisition configuration.

For example, the big data cleaning configuration for updating the acquisition field of the target traversal data acquisition node and the linkage traversal data acquisition node acquired in each individual process by combining the big data cleaning configuration information to perform traversal data acquisition on the big data update process of each big data update activity of the service acquisition activity includes:

in combination with the big data cleaning configuration information, respectively performing data cleaning index distribution on the target traversal data acquisition node and the linkage traversal data acquisition node, and outputting a target data cleaning index of the target traversal data acquisition node and a linkage data cleaning index of the linkage traversal data acquisition node, wherein the target data cleaning index has a corresponding cleaning index between a big data updating flow associated with the target traversal data acquisition node and the big data cleaning configuration information, and the linkage data cleaning index has a corresponding cleaning index between a big data updating flow associated with the linkage traversal data acquisition node and the big data cleaning configuration information;

shared data cleaning index distribution is carried out on the target data cleaning index and the linkage data cleaning index, and shared data cleaning indexes of the target traversal data acquisition nodes are output;

and performing traversal data acquisition on the big data updating flow of each big data updating activity of the service acquisition activity by combining the shared data cleaning index.

For example, in combination with the big data cleansing configuration information, data cleansing index distribution is performed on the target traversal data acquisition node, and a target data cleansing index of the target traversal data acquisition node is output, including:

extracting cleaning elements of the big data cleaning configuration information, and outputting cleaning element clusters of the big data cleaning configuration information;

extracting the acquisition elements of a plurality of traversal data acquisition sub-nodes in the target traversal data acquisition node, and outputting the acquisition elements of the plurality of traversal data acquisition sub-nodes;

outputting cleaning element distributions of the plurality of traversal data acquisition sub-nodes based on the cleaning element clusters and the acquisition elements of the plurality of traversal data acquisition sub-nodes, the cleaning element distribution of the traversal data acquisition child node refers to mapping variable information between the traversal data acquisition child node and the big data cleaning configuration information;

and associating the acquisition elements of the plurality of traversal data acquisition sub-nodes based on the cleaning element distribution of the plurality of traversal data acquisition sub-nodes, and outputting the target data cleaning index.

In a second aspect, the embodiment of the present application further provides an AI prediction based intelligent scene big data cleansing system, where the AI prediction based intelligent scene big data cleansing system includes an intelligent scene system and a plurality of intelligent scene service servers in communication connection with the intelligent scene system;

the intelligent scene system is used for:

For the above embodiment, the noise penetration relationship space can be configured by combining the noise penetration point clusters, and the penetration relationship of the noise penetration points in the basic template scene data is adopted for penetration region derivation, so that the data labels of the basic template scene data are expanded, and in addition, the noise penetration learning performance of the basic template scene data can be effectively enhanced based on the penetration region derivation, so that the noise penetration decision performance of the noise penetration decision network can be further enhanced, and the configuration precision of the subsequent big data cleaning process can be ensured.

Drawings

FIG. 1 is a schematic flowchart illustrating a method for cleaning big data of an intelligent scene based on AI prediction according to an embodiment of the present disclosure;

fig. 2 is a schematic block diagram of a smart scene system for implementing the AI prediction based smart scene big data cleansing method according to an embodiment of the present disclosure.

Detailed Description

The architecture of the AI-forecast based smart scene big data cleansing system 10 according to an embodiment of the present application is described below, and the AI-forecast based smart scene big data cleansing system 10 may include a smart scene system 100 and a smart scene service server 200 communicatively connected to the smart scene system 100. The intelligent scene system 100 and the intelligent scene service server 200 in the AI forecast based intelligent scene big data cleansing system 10 can cooperate to execute the AI forecast based intelligent scene big data cleansing method described in the following method embodiments, and the detailed description of the method embodiments can be referred to in the following steps of the intelligent scene system 100 and the intelligent scene service server 200.

The AI-prediction-based intelligent scene big data cleaning method provided by the present embodiment can be executed by the intelligent scene system 100, and the AI-prediction-based intelligent scene big data cleaning method is described in detail below with reference to fig. 1.

And Node101, calling basic template scene big data from an intelligent scene service database bound with the intelligent scene system in advance.

For example, the basic template scene big data may include a plurality of basic template scene data labeled with a root template noise penetration region, where the root template noise penetration region may be an initial noise penetration region manually labeled in the basic template scene data, and the initial noise penetration region represents that the positioning labeling information of the noise data exists in the basic template scene data.

Node102, combining the noise penetration decision model of the initialization training to perform noise penetration point mining on the basic template scene data in the basic template scene big data, and outputting a noise penetration point cluster.

The noise penetration point mining structure of the noise penetration decision model of the initialization training can be combined to extract the noise penetration points of each basic template scene data in the basic template scene big data, and a noise penetration point cluster is output.

For example, in some examples, the noise penetration point mining structure may employ DenseNet or the like.

Node103, combining with the noise penetration point cluster, configuring noise penetration relation space for independent penetration noise penetration positioning branch by using each said basic template scene data.

For example, in some examples, the noise penetration relationship space is a relationship map expressing noise penetration relationships between an penetrating noise penetration localization branch and its associated penetrating noise penetration localization branch. For example, an embodiment of configuring the noise penetration relationship space may be:

for example, in a noise penetration point cluster, noise penetration points associated with each piece of basic template scene data are selected, noise penetration connectivity between pieces of basic template scene data is determined based on the noise penetration points of the pieces of basic template scene data, connected basic template scene data of the pieces of basic template scene data is determined in the piece of basic template scene big data in combination with the noise penetration connectivity, the connected basic template scene big data of the pieces of basic template scene data is output, and a noise penetration relation space is configured by taking each piece of basic template scene data as an independent penetration noise penetration positioning branch in combination with the connected basic template scene big data.

For example, in some examples, the noise penetration connectivity between the base template scene data may be determined by determining a penetration correlation value between two noise penetration points in conjunction with a feature distance function, thereby obtaining the noise penetration connectivity between the base template scene data.

For example, in some examples, in combination with the connected basic template scene big data, an implementation scheme for configuring the noise penetration relationship space may be selected based on actual requirements, for example, noise penetration connection information between the basic template scene data and the basic template scene data in the associated connected basic template scene big data may be retrieved from an intelligent scene service database previously bound to the intelligent scene system, connection data of the basic template scene data is output, in combination with the connection data, an initial noise penetration relationship space is configured with each basic template scene data as an independent penetration noise penetration positioning branch, and a repeated relationship elimination is performed on the initial noise penetration relationship space, so as to output the noise penetration relationship space.

For example, in some examples, the connectivity data may be information indicating noise penetration connectivity information between the base template scene data and base template scene data in the connectivity base template scene big data. The scheme for obtaining the connected data can be flexibly configured, for example, the target noise penetration connectivity between the basic template scene data and the basic template scene data in the associated connected basic template scene big data can be determined in the noise penetration connectivity, the target noise penetration connectivity is aggregated, the noise penetration connectivity information between the basic template scene data and the basic template scene data in the connected basic template scene big data is output, and the connected data of the basic template scene data is generated by combining the noise penetration connectivity information.

For example, in some examples, a scheme for aggregating the target noise penetration connectivity and outputting the noise penetration connectivity information between the basic template scene data and the basic template scene data in the connected basic template scene big data may be selected based on actual requirements, for example, the target noise penetration connectivity may be aggregated, the aggregated noise penetration connectivity may be output, the aggregated noise penetration connectivity may be sorted, and the noise penetration connectivity information between the basic template scene data may be determined by combining the sorting information.

For example, in some examples, in combination with the connected data, the manner of configuring the initial noise penetration relationship space for the independent penetration noise penetration positioning branch by using each piece of the basic template scene data may be to configure, in combination with the connected data, a relationship space in which each relationship member represents noise penetration connection information between two pieces of basic template scene data, so as to use the configured relationship space as the initial noise penetration relationship space.

After the initial noise penetration relation space is configured, the repeated relation elimination can be carried out on the initial noise penetration relation space, and the noise penetration relation space is output.

Node104, combining the noise penetration relation space to perform penetration region derivation on the root template noise penetration region of the basic template scene data, and outputting the derived template scene big data.

For example, the root template noise penetration region of the basic template scene data may be subjected to penetration derivative analysis between penetration noise penetration positioning branches of the noise penetration relationship space, derivative noise penetration region information of the basic template scene data is output, the root template noise penetration region of the basic template scene data is subjected to penetration region derivation by combining the derivative noise penetration region information, and derivative template scene big data is output, which may be implemented by, for example, the following steps:

a1, performing penetration derivative analysis on a root template noise penetration area of the basic template scene data among penetration noise penetration positioning branches of a noise penetration relation space, and outputting derivative noise penetration area information of the basic template scene data.

For example, in some examples, the derived noise penetration region information may be information of a noise penetration region distribution obtained after penetration derivative analysis of a noise penetration region distribution associated with the root template noise penetration region in a noise penetration relationship space.

For example, in some examples, the scheme for performing the penetration derivation analysis on the root template noise penetration region of the basic template scene data may be selected based on actual requirements, and may be, for example:

for example, a root template noise penetration region distribution associated with basic template scene big data is generated by combining a root template noise penetration region of basic template scene data, a penetration derivation analysis is performed on the root template noise penetration region between penetration noise penetration positioning branches of a noise penetration relation space by combining a penetration derivation analysis rule specified by a developer, and derivation noise penetration region information of the basic template scene data is output, for example, the implementation manner may be:

(1) And generating root template noise penetration area distribution related to the big data of the basic template scene by combining the root template noise penetration area of the basic template scene data.

For example, in some examples, the root template noise penetration region distribution may be a noise penetration region distribution obtained by performing weight fusion of respective corresponding coefficients on root template noise penetration regions of all pieces of basic template scene data in the basic template scene big data.

For example, in some examples, the scheme for configuring the root template noise penetration region distribution associated with the basic template scene big data may be selected based on actual requirements, for example, the implementation may be:

for example, a noise penetration region distribution M of P × L is constructed by combining the number P of the basic template scene data and the number L of the root template noise penetration regions of the basic template scene data, where the penetration regions in the noise penetration region distribution are used to represent the root template noise penetration regions of the associated basic template scene data, and all penetration regions in the xth row may be combined into a feature cluster, and the feature cluster is used as the root template noise penetration region associated with the basic template scene data x. Therefore, the root template noise penetration region associated with each piece of basic template scene data may be included in the root template noise penetration region distribution.

(2) And distributing the root template noise penetration region between penetration noise penetration positioning branches of the noise penetration relation space by combining a penetration derivative analysis rule specified by a developer to perform penetration derivative analysis, and outputting the derivative noise penetration region information of the basic template scene data.

For example, the noise relationship matching degree between the basic template scene data may be determined by combining the noise penetration relationship space, a target coefficient associated with the noise relationship matching degree is obtained, a root template noise penetration region of the basic template scene data is derived based on the target coefficient, the derived noise penetration regions are aggregated, and the derived noise penetration region information of the basic template scene data is output.

For example, in some examples, a scheme for determining the noise relationship matching degree between the basic template scene data in combination with the noise penetration relationship space may be selected based on actual requirements, for example, noise penetration connection information between penetration noise penetration positioning branches may be mined in the noise penetration relationship space, the penetration deviation degree of the penetration noise penetration positioning branches may be generated in combination with the noise penetration connection information, the penetration deviation degree is converted into the noise relationship matching degree, and the smaller the penetration deviation degree, the larger the noise relationship matching degree of the basic template scene data may be represented.

For example, in some examples, the scheme of aggregating the derived noise penetration zones to obtain the derived noise penetration zone information of the basic template scene data may be selected based on actual requirements, for example, the derived noise penetration zones may be aggregated to obtain a new noise penetration zone distribution, the noise penetration zone distribution M is updated in combination with the new noise penetration zone distribution, the updated noise penetration zone distribution M is output, and the updated noise penetration zone distribution M is used as the derived noise penetration zone information.

And A2, combining the information of the derivative noise penetration area, carrying out penetration area derivative on the root template noise penetration area of the basic template scene data, and outputting the derivative template scene big data.

For example, the derivative noise penetration characteristics of the basic template scene data are analyzed in the derivative noise penetration area information, the derivative noise penetration area of the basic template scene data is generated by combining the derivative noise penetration characteristics, the penetration area derivation is performed on the root template noise penetration area of the basic template scene data by combining the derivative noise penetration area, and the derivative template scene big data is output.

For example, in some examples, a scheme for mining the derivative noise penetration features of the basic template scene data in the derivative noise penetration region information may be selected based on actual requirements, for example, taking the basic template scene data x as an example, all penetration regions in the xth row are determined in the updated noise penetration region distribution, and feature combination is performed on all penetration regions respectively, so that the derivative noise penetration features of the basic template scene data may be obtained, or feature combination may be performed after the penetration regions are derived, and the derivative noise penetration features of the basic template scene data may also be obtained.

After the derivative noise penetration features of the basic template scene data are extracted, a derivative noise penetration area of the basic template scene data can be determined, and the determination scheme can be selected based on actual requirements, for example, a noise penetration feature point with the largest support parameter value can be determined in the derivative noise penetration features, penetration positioning field information of the noise penetration feature point is mined in the derivative noise penetration features, a target noise penetration area associated with the penetration positioning field information is obtained, the target noise penetration area is used as the derivative noise penetration area of the basic template scene data, for example, the node of the noise penetration feature point is the y-th column, and then the noise penetration area y associated with the y-th column can be used as the derivative noise penetration area of the basic template scene data.

When the derivative noise penetration area of the basic template scene data is determined, penetration area derivation can be performed on a root template noise penetration area of the basic template scene data, and derivative template scene big data is output.

Node105, combining the derived template scene big data to perform decision function functional layer adjustment on the noise penetration decision model of the initialization training, and performing noise penetration decision on the loaded target intelligent scene big data based on the target noise penetration decision model of the matched model deployment condition.

For example, an implementation of Node105 may include the following scheme, for example.

And C1, adjusting a decision function layer of the noise penetration decision model subjected to initial training by combining with the big data of the derivative template scene.

For example, a decision function functional layer of a noise penetration decision model in initial training is adjusted by combining noise penetration points and noise penetration areas in the derived template scene big data, noise penetration point mining is performed on basic template scene data in the derived template scene big data by combining the noise penetration decision model in the initial training, penetration area derivation is performed on the noise penetration area of the basic template scene data based on a target noise penetration point cluster, the noise penetration points and the noise penetration areas of the basic template scene data in the derived template scene big data are returned, a decision function functional layer adjusting process is performed on the noise penetration decision model in the initial training until the noise penetration decision model in the initial training matches a model deployment condition, and a target noise penetration decision model matching the model deployment condition is output. For example, the implementation mode can be as follows:

(1) And adjusting a decision function layer of the noise penetration decision model subjected to initial training by combining the noise penetration point and the noise penetration area of the basic template scene data in the big data of the derived template scene.

For example, a first penetration decision error value of a noise penetration region of basic template scene data in the derived template scene big data may be determined in combination with the noise penetration region of the basic template scene data in the derived template scene big data, a second penetration decision error value of a noise penetration point of the basic template scene data may be determined in combination with the noise penetration point of the basic template scene data in the derived template scene big data, the first penetration decision error value and the second penetration decision error value may be subjected to weight fusion of respective corresponding coefficients, and a decision function layer of the noise penetration decision model initially trained may be adjusted based on the fusion penetration decision error value.

For example, in some examples, a scheme for determining the first penetration decision error value of the noise penetration region of the base template scene data may be selected based on actual requirements, for example, the root template noise penetration region of the base template scene data may be compared with the noise penetration region after penetration derivation analysis, an error rate of noise penetration decision performed by the noise penetration decision model that is initially trained is determined, and error rate calculation is performed based on a cross entropy loss function, so as to obtain the first penetration decision error value.

For example, in some examples, a scheme for determining the second penetration decision error value of the noise penetration point of the basic template scene data may be selected based on actual requirements, for example, the noise penetration area of the basic template scene data in the derived template scene big data may be combined, the basic template scene data may be clustered, a basic template scene data cluster associated with each noise penetration area may be output, a target noise penetration point associated with the basic template scene data cluster may be determined in combination with the noise penetration point of the basic template scene data in the basic template scene data cluster, the noise penetration point of the basic template scene data and the target noise penetration point associated with the basic template scene data cluster may be subjected to weight fusion of respective corresponding coefficients, and the second penetration decision error value of the noise penetration point of the basic template scene data may be output.

For example, in some examples, the scheme for determining the target noise penetration point associated with the basic template scene data cluster may be selected based on actual requirements, for example, a noise penetration difference mean of the noise penetration points of the basic template scene data in the basic template scene data cluster may be calculated, and the noise penetration difference mean may be used as the target noise penetration point associated with the basic template scene data cluster.

After calculating the target noise penetration point associated with the basic template scene data cluster, the noise penetration point of the basic template scene data and the target noise penetration point associated with the basic template scene data cluster may be weighted and fused by respective corresponding coefficients, and a second penetration decision error value of the noise penetration point of the basic template scene data may be output.

For example, in some examples, a first penetration decision error value and a second penetration decision error value are subjected to weight fusion of respective corresponding coefficients, and a noise penetration decision model subjected to initial training is subjected to decision function layer adjustment based on the weighted penetration decision error value, a specific training scheme may be selected based on actual requirements, for example, a target coefficient associated with the first penetration decision error value and the second penetration decision error value may be obtained, the first penetration decision error value and the second penetration decision error value are respectively derived in combination with the target coefficient, the derived first penetration decision error value and the derived second penetration decision error value in a noise penetration region are subjected to weight fusion of respective corresponding coefficients, a weighted penetration decision error value is output, and a model function layer in the noise penetration decision model subjected to initial training is updated based on the weighted penetration decision error value to converge the noise penetration decision model subjected to initial training, or the first penetration decision error value and the second penetration decision error value are directly subjected to weight fusion of respective corresponding coefficients, and a gradient descent decision algorithm is combined to update the noise penetration decision model subjected to noise penetration decision configuration, and a noise penetration decision layer is configured.

(2) And (4) carrying out noise penetration point mining on the basic template scene data in the big data of the derived template scene by combining with a noise penetration decision model of initialization training, and outputting a target noise penetration point cluster.

For example, noise penetration points of each basic template scene data in the derived template scene big data can be extracted by combining with a noise penetration point mining structure of the noise penetration decision model which is initially trained, and a noise penetration point cluster is output.

(3) And carrying out infiltration region derivation on the noise infiltration region of the scene data of the basic template based on the target noise infiltration point cluster.

For example, based on the target noise penetration point cluster, configuring a target noise penetration relationship space for each basic template scene data as an independent penetration noise penetration positioning branch, performing penetration derivation analysis on a noise penetration region of the basic template scene data among the penetration noise penetration positioning branches of the target noise penetration relationship space, outputting target derived noise penetration region information of the basic template scene data, performing penetration region derivation on the noise penetration region of the basic template scene data by combining the target derived noise penetration region information, and outputting derived template scene big data.

(4) And returning to execute the step of adjusting the decision function layer of the noise penetration decision model subjected to initial training by combining the noise penetration point and the noise penetration area of the basic template scene data in the big data of the derived template scene until the noise penetration decision model subjected to initial training matches the model deployment condition, and outputting the target noise penetration decision model matched with the model deployment condition.

For example, after performing the penetration region derivation on the noise penetration region of the basic template scene data, the step of performing the decision function layer adjustment on the noise penetration decision model initially trained may be performed by returning to the step of updating the noise penetration point and the noise penetration region of the basic template scene data in the basic template scene big data, and the specific adjustment manner may refer to the description of the above embodiment.

And C2, carrying out noise penetration decision on the loaded target intelligent scene big data by a target noise penetration decision model based on the matched model deployment condition.

For example, the loaded target smart scene big data may be obtained, the loaded target smart scene big data has a plurality of target smart scene acquisition data, noise penetration point mining is performed on the loaded target smart scene big data by combining with a noise penetration point mining structure of a target noise penetration decision model matching with the model deployment condition, noise penetration points of the target smart scene acquisition data are output, noise penetration decision is performed on the noise penetration points by combining with the target noise penetration decision model matching with the model deployment condition, and noise penetration decision information of the target smart scene acquisition data is output, where the noise penetration decision information may include noise penetration decision area distribution information of the target smart scene acquisition data.

For example, in some examples, the scheme for performing noise penetration decision on the noise penetration point may be selected based on actual requirements, for example, noise penetration location may be performed on the noise penetration point in combination with a noise penetration location branch of the target noise penetration decision model matching the model deployment condition, a support parameter value associated with each noise penetration location partition is output, and noise penetration area distribution of the loaded target smart scene big data is determined in the noise penetration location partition in combination with the support parameter value, or noise penetration area classification may be performed on the noise penetration point in combination with the target noise penetration decision model matching the model deployment condition, and an output result of the loaded target smart scene big data is determined in combination with the noise penetration area distribution.

The following describes a big data collection configuration method based on AI decision according to another embodiment of the present application, including the following steps.

STEP110, obtaining a target noise penetration region obtained by performing noise penetration decision on the loaded target intelligent scene big data by a target noise penetration decision model based on the matched model deployment condition.

In detail, the target noise penetration decision model may have a noise penetration region positioning function, so as to perform noise penetration decision on the loaded target smart scene big data to obtain a target noise penetration region.

And STEP120, which determines the big data cleaning field distribution of the big data acquisition process related to the big data of the target intelligent scene based on the target noise penetration area.

In detail, for each target noise penetration area, the corresponding big data cleaning field distribution can be configured according to the data field related to the target noise penetration area, so that the big data cleaning field distribution of the big data acquisition process related to the big data of the target intelligent scene can be obtained from the configured big data cleaning field distribution. Wherein, the big data washing field distribution can be used for participating in the field allocation in the subsequent big data washing configuration process.

And the STEP130 is used for performing big data cleaning configuration on the big data acquisition process by combining the big data cleaning field distribution and outputting big data cleaning configuration data.

STEP140, updating the big data acquisition process by combining the big data cleaning configuration data, thereby loading the updated big data acquisition process to the intelligent scene service server for big data acquisition configuration.

By adopting the above embodiment, in this embodiment, a target noise penetration region obtained by performing a noise penetration decision on loaded target smart scene big data based on a target noise penetration decision model is obtained, the big data cleansing field distribution of a big data acquisition process associated with the target smart scene big data is determined based on the target noise penetration region, the big data cleansing configuration is performed on the big data acquisition process in combination with the big data cleansing field distribution, the big data cleansing configuration data is output, the big data acquisition process is updated in combination with the big data cleansing configuration data, and thus the updated big data acquisition process is loaded to the smart scene service server to perform the big data acquisition configuration. Therefore, the big data cleaning configuration corresponding to the big data cleaning configuration rule is executed in combination with the noise penetration area, and the big data acquisition configuration is updated in a targeted manner, so that the big data is cleaned in the process of carrying out the big data acquisition in the later period, and the generation of part of noise data is avoided.

For the STEP130, for example, the following technical solutions may be implemented.

And the SS101 is used for performing big data cleaning configuration on the big data acquisition process by combining the big data cleaning field distribution, acquiring traversal data acquisition node distribution obtained by performing traversal data acquisition on the big data updating activity associated with the big data acquisition process in a big data updating state by the service acquisition activity of the big data acquisition process, wherein the traversal data acquisition node distribution comprises a plurality of traversal data acquisition nodes generated according to the latest acquisition configuration sequence of the big data acquisition template.

For example, the traversal data acquisition nodes and each of the traversal data acquisition nodes in the traversal data acquisition nodes may be a traversal data acquisition node associated with a global traversal data acquisition process or a traversal data acquisition node associated with different splitting processes included in a global traversal data acquisition process.

And the SS102 acquires a target traversal data acquisition node in each single process aiming at each traversal data acquisition node in the traversal data acquisition node distribution and determines the target traversal data acquisition node as a target traversal data acquisition node, and acquires a traversal data acquisition node linked with the target traversal data acquisition node and determines the target traversal data acquisition node as a linked traversal data acquisition node.

For example, the distribution of the traversal data collection nodes may be represented as: { collection.1, collection.2, collection.3, ·. In other words, the traversal data collection node distribution may include several traversal data collection nodes, collection.1 through collection.n. Then, in the first single flow, collection.1 can be determined as a target traversal data collection node, and collection.2 serves as a linkage traversal data collection node; in a later independent flow, the collection.2 can be determined as a target traversal data collection node, and the collection.3 can be used as a linkage traversal data collection node.

For another example, the number of the linkage traversal data acquisition nodes may also be multiple, for example, taking that each individual process includes two linkage traversal data acquisition nodes as an example, in the first individual process, collection.1 may be determined as a target traversal data acquisition node, and collection.2 and collection.3 are taken as linkage traversal data acquisition nodes; in a later single flow, collection.2 can be determined as a target traversal data collection node, and collection.3 and collection.4 can be used as linkage traversal data collection nodes.

The traversal data acquisition nodes comprise at least one traversal data acquisition sub-node related to the traversal data acquisition nodes of the big data updating process.

And the SS103 acquires big data cleaning configuration information associated with a target big data updating process needing to be subjected to big data cleaning configuration.

For example, in the big data cleansing configuration process, the big data cleansing configuration may be performed on the traversal data collection node of one big data updating process.

And the SS104 is used for updating the acquisition field of the target traversal data acquisition node and the linkage traversal data acquisition node acquired in each single process by combining the big data cleaning configuration information, so as to perform the big data cleaning configuration of the service acquisition activity on the big data updating process of each big data updating activity and acquire the traversal data, and output big data cleaning configuration data, wherein the big data cleaning configuration data comprises noise cleaning field data corresponding to each traversal data acquisition table item.

For example, for the SS104, the following steps may be included in a manner of performing acquisition field update on the target traversal data acquisition node and the linkage traversal data acquisition node acquired in each individual process in combination with the big data cleaning configuration information to implement big data cleaning configuration for performing traversal data acquisition on the big data update process of each big data update activity by the service acquisition activity.

And the SS141 is used for respectively distributing data cleaning indexes to the target traversal data acquisition nodes and the linkage traversal data acquisition nodes by combining the big data cleaning configuration information, and outputting the target data cleaning indexes of the target traversal data acquisition nodes and the linkage data cleaning indexes of the linkage traversal data acquisition nodes.

For example, the target data cleansing index has a corresponding cleansing index between a big data updating process associated with the target traversal data acquisition node and the big data cleansing configuration information, and the linkage data cleansing index has a corresponding cleansing index between a big data updating process associated with the linkage traversal data acquisition node and the big data cleansing configuration information.

The target data cleaning index of the target traversal data acquisition node may include cleaning index information of a big data updating process described in the target traversal data acquisition node, the linkage data cleaning index of the linkage traversal data acquisition node may include cleaning index information of a big data updating process described in the linkage traversal data acquisition node, the target data cleaning index may further include a corresponding cleaning index between the big data updating process associated with the target traversal data acquisition node and the big data cleaning configuration information, and the linkage data cleaning index may also include a corresponding cleaning index between the big data updating process associated with the linkage traversal data acquisition node and the big data cleaning configuration information.

In combination with the above, data cleaning index distribution is performed on the target traversal data acquisition node based on the big data cleaning configuration information, so that the cleaning element cluster of the big data cleaning configuration information is merged into the target data cleaning index configured by big data cleaning, and meanwhile, data cleaning index distribution is performed on the linkage traversal data acquisition node based on the big data cleaning configuration information, so that the cleaning element cluster of the big data cleaning configuration information is also merged into the extracted linkage data cleaning index.

For example, the SS141 refers to the following description.

And the SS1411 extracts the cleaning elements of the big data cleaning configuration information and outputs the cleaning element clusters of the big data cleaning configuration information.

The cleaning element cluster of the big data cleaning configuration information is used for representing the characteristic variable information of the specific big data cleaning field of the big data cleaning configuration information.

And the SS1412 is used for extracting the acquisition elements of a plurality of traversal data acquisition sub-nodes in the target traversal data acquisition node and outputting the acquisition elements of the plurality of traversal data acquisition sub-nodes.

For example, the traversal data acquisition nodes may be divided, a plurality of traversal data acquisition sub-nodes may be output, then acquisition element extraction may be performed on each traversal data acquisition sub-node, and an acquisition element of each traversal data acquisition sub-node may be output.

For example, for the big data cleansing configuration information, candidate big data cleansing configuration information associated with a plurality of different candidate big data update flows of the big data cleansing configuration that needs to perform traversal data acquisition may be obtained, and the currently obtained big data cleansing configuration information may be any one big data cleansing configuration information obtained from the plurality of candidate big data cleansing configuration information.

And SS1413, outputting cleaning element distribution of the plurality of traversal data acquisition sub-nodes based on the cleaning element cluster and the acquisition elements of the plurality of traversal data acquisition sub-nodes.

For example, the cleaning element distribution of the traversal data collection child node refers to mapping variable information between the traversal data collection child node and the big data cleaning configuration information.

And the SS1414 associates the acquisition elements of the plurality of traversal data acquisition sub-nodes based on the cleaning element distribution of the plurality of traversal data acquisition sub-nodes, and outputs the target data cleaning index.

And the SS142 is used for distributing the shared data cleaning indexes of the target data cleaning indexes and the linkage data cleaning indexes and outputting the shared data cleaning indexes of the target traversal data acquisition nodes.

For example, in order to enable the target data cleansing index to include more information related to the big data updating process, the target data cleansing index and the linkage data cleansing index may be distributed to share the data cleansing index, and the share data cleansing index of the target traversal data acquisition node is output.

By adopting the technical scheme, the embodiment of the application can combine the noise penetration point cluster to configure the noise penetration relation space, and the penetration relation of the noise penetration points in the basic template scene data is adopted to carry out penetration zone derivation, so that the data label of the basic template scene data is expanded, and in addition, the noise penetration learning performance of the basic template scene data can be effectively enhanced based on the penetration zone derivation, so that the noise penetration decision performance of a noise penetration decision network can be further enhanced, and the configuration precision of the follow-up big data cleaning process can be ensured.

Fig. 2 illustrates a hardware structure of the AI prediction based intelligent scene big data cleansing system 100 according to an embodiment of the present disclosure, and as shown in fig. 2, the intelligent scene system 100 may include a processor 110, a machine-readable storage medium 120, a bus 130, and a communication unit 140.

The processor 110 may perform various suitable actions and processes according to a program stored in the machine-readable storage medium 120, such as program instructions corresponding to the AI prediction based intelligent scene big data washing method described in the foregoing embodiments. The processor 110, the machine-readable storage medium 120, and the communication unit 140 perform signal transmission through the bus 130.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication unit 140, and when executed by the processor 110, performs the above-described functions defined in the methods of the embodiments of the present disclosure.

Yet another embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the computer-readable storage medium is used for implementing the AI prediction based intelligent scene big data cleaning method according to any one of the above embodiments.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (LAM), a read-only memory (LOM), an erasable programmable read-only memory (EPLOM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-LOM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, LM (radio frequency), etc., or any suitable combination of the foregoing.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

Yet another embodiment of the present disclosure further provides a computer program product, which includes a computer program, and when being executed by a processor, the computer program implements the AI-prediction-based intelligent scene big data cleaning method according to any one of the above embodiments.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of Network, including a Local Area Network (LAN) or a wide Area Network (PAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as LOM, LAM, magnetic or optical disks, etc.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present disclosure, and not for limiting the same; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.

Claims

1. A big data acquisition and configuration method based on AI decision is applied to an intelligent scene system, and the method comprises the following steps:

acquiring a target noise penetration decision-making model based on a matched model deployment condition, and performing noise penetration decision-making on target intelligent scene big data added by an intelligent scene service server based on task requirements to obtain a target noise penetration area;

performing big data cleaning configuration on the big data acquisition process by combining the big data cleaning field distribution, and outputting big data cleaning configuration data;

2. The AI decision-based big data collection configuration method according to claim 1, wherein the step of performing big data cleansing configuration on the big data collection process in combination with the big data cleansing field distribution and outputting big data cleansing configuration data comprises:

acquiring big data cleaning configuration information associated with a target big data updating process needing big data cleaning configuration by combining the big data cleaning field distribution;

and performing acquisition field updating on the target traversal data acquisition node and the linkage traversal data acquisition node acquired in each single process by combining the big data cleaning configuration information, so as to perform the big data cleaning configuration of the service acquisition activity on the big data updating process of each big data updating activity, and output big data cleaning configuration data, wherein the big data cleaning configuration data comprises noise cleaning field data corresponding to each traversal data acquisition table item.

3. The AI decision-based big data collection and configuration method according to claim 2, wherein the big data cleaning and configuration for performing collection field update on the target traversal data collection node and the linkage traversal data collection node acquired in each individual process in combination with the big data cleaning and configuration information to perform traversal data collection on the big data update process of the business collection activity for each big data update activity comprises:

combining the big data cleaning configuration information, respectively distributing data cleaning indexes to the target traversal data acquisition node and the linkage traversal data acquisition node, and outputting a target data cleaning index of the target traversal data acquisition node and a linkage data cleaning index of the linkage traversal data acquisition node, wherein the target data cleaning index has a corresponding cleaning index between a big data updating flow associated with the target traversal data acquisition node and the big data cleaning configuration information, and the linkage data cleaning index has a corresponding cleaning index between a big data updating flow associated with the linkage traversal data acquisition node and the big data cleaning configuration information;

shared data cleaning index distribution is carried out on the target data cleaning index and the linkage data cleaning index, and the shared data cleaning index of the target traversal data acquisition node is output;

4. The AI decision-based big data collection and configuration method according to claim 2, wherein the step of performing data cleansing index assignment on the target traversal data collection node in combination with the big data cleansing configuration information and outputting a target data cleansing index of the target traversal data collection node comprises:

outputting cleaning element distribution of the plurality of traversal data acquisition sub-nodes based on the cleaning element cluster and the acquisition elements of the plurality of traversal data acquisition sub-nodes, wherein the cleaning element distribution of the traversal data acquisition sub-nodes refers to mapping variable information between the traversal data acquisition sub-nodes and the big data cleaning configuration information;

and associating the acquisition elements of the plurality of traversal data acquisition subnodes based on the distribution of the cleaning elements of the plurality of traversal data acquisition subnodes, and outputting the target data cleaning index.

5. The AI decision-based big data collection configuration method according to claim 1, wherein the step of adjusting the decision function functional layer of the target noise penetration decision model comprises:

calling basic template scene big data from a smart scene service database which is bound with the smart scene system in advance, wherein the basic template scene big data covers a plurality of basic template scene data with root template noise penetration areas calibrated, the root template noise penetration areas are initial noise penetration areas manually marked in the basic template scene data, and the initial noise penetration areas indicate that positioning marking information of noise data exists in the basic template scene data;

and performing decision function layer adjustment on the noise penetration decision model subjected to initialization training based on the derived template scene big data, performing noise penetration decision on target smart scene big data added by the smart scene service server based on task requirements based on a target noise penetration decision model matched with model deployment conditions, and executing a big data cleaning process according to a target noise penetration area obtained by the noise penetration decision.

6. The AI decision-based big data collection configuration method according to claim 5, wherein the configuring a noise penetration relationship space for an independent penetration noise penetration localization branch with each of the basic template scene data in combination with the noise penetration point clusters comprises:

determining noise penetration points associated with each basic template scene data in the noise penetration point cluster, and determining the noise penetration connectivity between the basic template scene data based on the noise penetration points of the basic template scene data;

determining connected basic template scene data of the basic template scene data in the basic template scene big data by combining the noise penetration connectivity, and outputting the connected basic template scene big data of the basic template scene data;

determining a target noise penetration connectivity between the basic template scene data and basic template scene data in associated connected basic template scene big data in the noise penetration connectivity;

aggregating the target noise penetration connectivity, and outputting noise penetration connectivity information between the basic template scene data and basic template scene data in the connected basic template scene big data;

generating connected data of the scene data of the basic template by combining the noise penetration connected information;

and combining the communication data, configuring an initial noise penetration relation space by taking each basic template scene data as an independent penetration noise penetration positioning branch, performing repeated relation elimination on the initial noise penetration relation space, and outputting the noise penetration relation space.

7. The AI decision-based big data collection and configuration method according to claim 5, wherein the performing a decision function functional layer adjustment on the initially trained noise penetration decision model based on the derived template scene big data comprises:

based on the noise penetration point and the noise penetration area of the basic template scene data in the big data of the derived template scene, carrying out decision function functional layer adjustment on a noise penetration decision model which is initially trained;

performing noise penetration point mining on basic template scene data in the derived template scene big data by combining the initialized and trained noise penetration decision model, and outputting a target noise penetration point cluster;

based on the target noise penetration point cluster, carrying out penetration zone derivation on a noise penetration zone of the basic template scene data;

returning to execute the noise penetration point and the noise penetration area based on the basic template scene data in the derived template scene big data, and performing decision function layer adjustment on the initialized and trained noise penetration decision model until the initialized and trained noise penetration decision model matches the model deployment condition, and outputting a target noise penetration decision model matching the model deployment condition;

wherein the adjusting the decision function layer of the initially trained noise penetration decision model based on the noise penetration point and the noise penetration region of the basic template scene data in the derived template scene big data comprises:

determining a first penetration decision error value of a noise penetration area of basic template scene data in the derived template scene big data based on the noise penetration area of the basic template scene data;

clustering the basic template scene data based on the noise penetration area of the basic template scene data in the big data of the derived template scene, and outputting a basic template scene data cluster associated with each noise penetration area;

acquiring a target noise penetration point associated with the basic template scene data cluster based on the noise penetration point of the basic template scene data in the basic template scene data cluster;

determining noise penetration difference information between basic template scene data in the basic template scene data cluster based on the noise penetration points of the basic template scene data, and outputting first noise penetration difference information;

determining noise penetration difference information between the basic template scene data clusters based on target noise penetration difference information associated with the basic template scene data clusters, and outputting second noise penetration difference information;

calculating noise penetration difference information between the first noise penetration difference information and the second noise penetration difference information, outputting third noise penetration difference information, performing weight fusion of respective corresponding coefficients on the third noise penetration difference information and a preset penetration template variable, and outputting fourth noise penetration difference information;

if the fourth noise penetration difference information is larger than a preset difference value through analysis, determining a noise penetration difference mean value of the fourth noise penetration difference information, and outputting a second penetration decision error value of a noise penetration point of the basic template scene data;

and performing weight fusion of respective corresponding coefficients on the first penetration decision error value and the second penetration decision error value, and performing decision function layer adjustment on the initially trained noise penetration decision model based on the fusion penetration decision error value.

8. The AI decision-based big data collection and configuration method according to claim 5, wherein the performing infiltration region derivation on a root template noise infiltration region of the base template scene data in combination with the noise infiltration relationship space and outputting derived template scene big data comprises:

performing penetration derivative analysis between penetration noise penetration positioning branches of the noise penetration relation space by combining a root template noise penetration area of the basic template scene data, and outputting the derivative noise penetration area information of the basic template scene data;

and carrying out infiltration region derivation on a root template noise infiltration region of the basic template scene data based on the derived noise infiltration region information, and outputting derived template scene big data.

9. The AI decision-based big data collection configuration method according to claim 8, wherein the combining the root template noise penetration region of the base template scene data performs penetration derivation analysis between penetration noise penetration localization branches of the noise penetration relationship space, and outputting the derived noise penetration region information of the base template scene data includes:

generating root template noise penetration area distribution related to the basic template scene big data based on the root template noise penetration area of the basic template scene data;

distributing the root template noise penetration region between penetration noise penetration positioning branches of the noise penetration relation space for penetration derivative analysis by combining a penetration derivative analysis rule specified by a developer, and outputting derivative noise penetration region information of the basic template scene data;

the distributing of the root template noise penetration regions includes root template noise penetration regions associated with each basic template scene data, and the distributing of the root template noise penetration regions among penetration noise penetration positioning branches of the noise penetration relationship space is performed with penetration derivative analysis by combining a penetration derivative analysis rule specified by a developer, and the outputting of the derivative noise penetration region information of the basic template scene data includes:

determining the noise relation matching degree between the basic template scene data by combining the noise penetration relation space;

acquiring a target coefficient associated with the noise relationship matching degree, and deriving a root template noise penetration area of the basic template scene data based on the target coefficient;

and aggregating the derived noise penetration areas, and outputting the derived noise penetration area information of the basic template scene data.

10. An intelligent scenario system, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the AI decision-based big data collection configuration method of any one of claims 1-9.