CN110874645A

CN110874645A - Data reduction method

Info

Publication number: CN110874645A
Application number: CN201911113392.1A
Authority: CN
Inventors: 周金龙
Original assignee: Beijing Shouqi Zhixing Technology Co Ltd
Current assignee: Beijing Shouqi Zhixing Technology Co Ltd
Priority date: 2019-11-14
Filing date: 2019-11-14
Publication date: 2020-03-10

Abstract

The invention discloses a data reduction method, relates to the technical field of computers, and can effectively reduce various types of service data by adopting a machine learning model and a regression model.

Description

Data reduction method

Technical Field

The invention belongs to the technical field of computers, and particularly relates to a data reduction method.

Background

Some types of data are collected in a buried point mode at present, the collection mode needs to be transmitted through a plurality of nodes, and data can be lost due to data transmission failure of a certain node for some reasons. There is currently no relative recovery measure for lost data.

Disclosure of Invention

Aiming at the defects in the prior art, the embodiment of the invention provides a data reduction method, which comprises the following steps:

before data loss, acquiring data characteristics of various service data to generate a first data characteristic set;

extracting partial data features from the first feature set to generate a second data feature set;

inputting the first data feature set and the second data feature set into a machine learning model respectively, and calculating correlation coefficients between the data features of the first data set and the data features of the second data set respectively;

selecting data features with the maximum correlation coefficient with each data feature of the first data feature set from the second data feature set to generate a third data feature set;

after data loss, simultaneously inputting the remaining data features of the first data feature set and the third data feature set into a regression model, and restoring the lost service data of the first data feature set.

Preferably, the remaining data features of the first data feature set and the third data feature set are simultaneously input into a regression model, and the restoring the lost service data of the first data feature set includes:

and predicting the service data corresponding to each data feature of the first data feature set by the regression model through the trend of each data feature in the third data feature set to obtain the lost service data of each data feature of the first data feature set.

The data reduction method provided by the embodiment of the invention has the following beneficial effects:

by adopting the machine learning model and the regression model, various types of business data can be effectively restored.

Detailed Description

The present invention will be described in detail with reference to the following embodiments.

The data recovery method provided by the embodiment of the invention comprises the following steps:

s101, before data loss, data characteristics of various types of service data are obtained, and a first data characteristic set is generated.

As a specific embodiment, for the shared automobile industry, the first data feature set includes data features of the number of network points in 5 kilometers nearby, the number of network points in 1 kilometer nearby, the total number of vehicles at the network points, the number of vehicles to be rented at each network point, the total number of orders placed at each time, the number of miles traveled by each vehicle, the number of miles ordered, the time length of placing the orders, and the number of scanned vehicles at each time, which are 9.

S102, extracting partial data features from the first feature set to generate a second data feature set.

As a specific embodiment, for the shared automobile industry, the second data feature set includes 6 data features of the number of vehicles to be rented at each website, the total number of orders placed at each time, the mileage traveled by each vehicle, the mileage ordered, the duration of placing the orders, and the number of scanned vehicles at each time.

S103, inputting the first data feature set and the second data feature set into the machine learning model respectively, and calculating correlation coefficients between the data features of the first data set and the data features of the second data set respectively.

And S104, selecting the data features with the maximum correlation coefficient with the data features in the first data feature set from the second data feature set, and generating a third data feature set.

As a specific embodiment, in the second data feature set, the data features with the highest correlation coefficient with respect to each data feature of the first data feature set are: and the number of the vehicles to be rented of each website, the total number of the orders placed at each moment, the travel mileage of each vehicle and the order placing time length, and the generated third data feature set comprises 4 data features of the number of the vehicles to be rented of each website, the total number of the orders placed at each moment, the travel mileage of each vehicle and the order placing time length.

And S105, after the data are lost, simultaneously inputting the residual data features of the first data feature set and the third data feature set into a regression model, and restoring the lost service data of the first data feature set.

Optionally, the step of simultaneously inputting the remaining data features of the first data feature set and the third data feature set into the regression model, and the restoring the lost service data of the first data feature set includes:

and predicting the service data corresponding to each data feature of the first data feature set by the regression model through the trend of each data feature in the third data feature set to obtain the service data lost by each data feature of the first data feature set.

The data reduction method provided by the embodiment of the invention can effectively reduce various types of service data by adopting the machine learning model and the regression model.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

It will be appreciated that the relevant features of the method and apparatus described above are referred to one another. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In addition, the memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

It should be noted that the above-mentioned embodiments do not limit the present invention in any way, and all technical solutions obtained by using equivalent alternatives or equivalent variations fall within the protection scope of the present invention.

Claims

1. A method of data reduction, comprising:

2. The data reduction method according to claim 1, wherein the remaining data features of the first data feature set and the third data feature set are input into a regression model at the same time, and the reduction of the lost traffic data of the first data feature set comprises:

3. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of claims 1-2 are implemented when the computer program is executed by the processor.