CN117390098A - Data analysis method, device, computer equipment and storage medium - Google Patents
Data analysis method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117390098A CN117390098A CN202311108474.3A CN202311108474A CN117390098A CN 117390098 A CN117390098 A CN 117390098A CN 202311108474 A CN202311108474 A CN 202311108474A CN 117390098 A CN117390098 A CN 117390098A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- features
- processed
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000007405 data analysis Methods 0.000 title claims abstract description 39
- 238000004458 analytical method Methods 0.000 claims abstract description 152
- 238000013506 data mapping Methods 0.000 claims abstract description 54
- 238000013507 mapping Methods 0.000 claims abstract description 54
- 238000010191 image analysis Methods 0.000 claims abstract description 44
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 38
- 238000004590 computer program Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 13
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 7
- 230000004927 fusion Effects 0.000 claims description 5
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Image Analysis (AREA)
Abstract
The application relates to a data analysis method, a data analysis device, computer equipment and a storage medium. Can be used in the field of artificial intelligence. The method comprises the following steps: acquiring a data table corresponding to a target object in a target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image; acquiring a trained target image analysis model, and extracting features of a target image based on the target image analysis model to obtain various features corresponding to the target image; and based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result. By adopting the method, the efficiency of data analysis in the database can be improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data analysis method, apparatus, computer device, and storage medium.
Background
At present, the analysis of data in a database is mainly to search the data based on query sentences and then analyze the characteristics of a large amount of searched data. However, this way of data analysis takes time and is computationally intensive, resulting in a low efficiency of data analysis in the database.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data analysis method, apparatus, computer device, and storage medium capable of converting a data table into an image for analysis, which improves the efficiency of data analysis in a database.
A method of data analysis, the method comprising:
acquiring a data table corresponding to a target object in a target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image;
acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image;
And based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result.
In one embodiment, determining each data mapping range corresponding to each field type in the data table includes:
performing outlier processing on the data table to obtain a data table to be converted;
and determining a plurality of data mapping ranges corresponding to the field types based on the change relation of the data corresponding to the field types in the data table to be converted.
In one embodiment, based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result includes:
based on a preset similarity algorithm, respectively calculating the difference values between the feature vectors corresponding to the various features and the feature vectors corresponding to the various reference features, and respectively carrying out fusion operation on the difference values corresponding to the various features and the various reference features to obtain the similarity values corresponding to the various features and the various reference features;
When the similarity value is larger than a preset similarity threshold value, the target analysis result is that the target object belongs to a target type object corresponding to the target analysis service type;
and when the similarity value is not greater than the preset similarity threshold value, the target analysis result is that the target object does not belong to the target type object corresponding to the target analysis service type.
In one embodiment, before acquiring the trained image analysis model, the method further comprises:
acquiring a to-be-processed data table set corresponding to a target analysis service type, dividing the to-be-processed data table set based on a preset dimension to obtain a plurality of to-be-processed data sub-tables, determining a data mapping range corresponding to each field type in each to-be-processed data sub-table, and mapping data corresponding to unit cells in each to-be-processed data sub-table into corresponding mapping pixel values based on a corresponding relation between each data mapping range and a mapping pixel value to obtain to-be-processed images corresponding to each to-be-processed data sub-table;
inputting each image to be processed into an image analysis model, and extracting the characteristics of each image to be processed based on the image analysis model to obtain various comprehensive characteristics corresponding to each image to be processed;
Based on the preset similarity algorithm, various to-be-synthesized features corresponding to the to-be-processed images are compared, the to-be-synthesized features corresponding to the to-be-processed images with similar comparison are fused, various to-be-synthesized features are obtained, and the various to-be-synthesized features are used as various reference features corresponding to the target analysis service types.
In one embodiment, based on a preset similarity algorithm, various features corresponding to the target image and various reference features corresponding to the target analysis service type are matched, and after a target analysis result is obtained, the method further includes:
and when the target object belongs to a target type object corresponding to the target analysis service type, sending the associated content corresponding to the target analysis service type to a terminal corresponding to the target object.
A data analysis device, the device comprising:
the conversion module is used for acquiring a data table corresponding to the target object in the target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each unit cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image;
The extraction module is used for acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image; and the matching module is used for matching various features corresponding to the target image with various reference features corresponding to the target analysis service type based on a preset similarity algorithm to obtain a target analysis result.
In one embodiment, the data analysis device further includes a processing module, configured to obtain a set of to-be-processed data tables corresponding to the target analysis service type, divide the set of to-be-processed data tables based on a preset dimension to obtain a plurality of to-be-processed data sub-tables, determine a data mapping range corresponding to each field type in each to-be-processed data sub-table, and map data corresponding to cells in each to-be-processed data sub-table into corresponding mapping pixel values based on a corresponding relationship between each data mapping range and a mapping pixel value to obtain to-be-processed images corresponding to each to-be-processed data sub-table; inputting each image to be processed into an image analysis model, and extracting the characteristics of each image to be processed based on the image analysis model to obtain various comprehensive characteristics corresponding to each image to be processed; based on the preset similarity algorithm, various to-be-synthesized features corresponding to the to-be-processed images are compared, the to-be-synthesized features corresponding to the to-be-processed images with similar comparison are fused, various to-be-synthesized features are obtained, and the various to-be-synthesized features are used as reference similar features corresponding to the target analysis service types.
In one embodiment, the data analysis device further includes a sending module, configured to send, when the target object belongs to a target type object corresponding to the target analysis service type, associated content corresponding to the target analysis service type to a terminal corresponding to the target object.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring a data table corresponding to a target object in a target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image;
acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image;
and based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring a data table corresponding to a target object in a target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image;
acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image;
and based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result.
According to the data analysis method, the data analysis device, the computer equipment and the storage medium, the data table corresponding to the target object in the target analysis service type is obtained, each data mapping range corresponding to each field type in the data table is determined, and the data corresponding to each cell in the data table is mapped into the corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value, so that a target image is obtained; acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image; based on a preset similarity algorithm, various features corresponding to the target image are matched with various reference features corresponding to the target analysis service type, so that a target analysis result is obtained, a large amount of data is converted into an image for analysis, the time and calculation force spent when acquiring the data according to a database query statement and analyzing whether certain specific features or other types of features exist in the large amount of data are well reduced, and the efficiency of analyzing the data in the database is well improved.
Drawings
FIG. 1 is a diagram of an application environment for a data analysis method in one embodiment;
FIG. 2 is a flow chart of a method of data analysis in one embodiment;
FIG. 3 is a block diagram showing the structure of a data analysis device according to one embodiment;
FIG. 4 is an internal block diagram of a computer device in one embodiment;
fig. 5 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The data analysis method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The server 104 is configured to obtain a data table corresponding to the target object in the target analysis service type, determine each data mapping range corresponding to each field type in the data table, and map data corresponding to each cell in the data table to a corresponding mapping pixel value based on a corresponding relationship between each data mapping range and the mapping pixel value, so as to obtain a target image; acquiring a trained target image analysis model, and extracting features of a target image based on the target image analysis model to obtain various features corresponding to the target image; and based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.
The data analysis methods, apparatus, computer devices, and storage media of the present application include, but are not limited to, use in the big data arts, the artificial intelligence arts, and the financial arts.
In one embodiment, as shown in fig. 2, a data analysis method is provided, and the method is applied to the server in fig. 1 for illustration, and includes the following steps:
step S200, a data table corresponding to the target object in the target analysis service type is obtained, each data mapping range corresponding to each field type in the data table is determined, and data corresponding to each cell in the data table is mapped into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value, so that the target image is obtained.
Wherein the target object refers to an object to be analyzed and may be a certain user. The objective analysis service type refers to an analysis service type or an application scenario, such as analyzing whether a certain object is a good customer of bank credit, whether the object is a fraudulent customer, etc., and applying the analysis result to the credit evaluation scenario. A data table refers to a table in a database. The data mapping range refers to a data section in which data defining a certain section is mapped to a certain pixel value. Mapping pixel values refers to pixel values used to map table data. The target image refers to a two-dimensional image formed by converting data in a data table for predictive analysis into pixel values.
Specifically, because the data volume stored in the database is huge and various, each object storing data has corresponding data information corresponding to various application scenarios, when predicting whether an object accords with an object feature corresponding to an application scenario or a service type, a data table corresponding to the target object when the service type is analyzed by the target object can be obtained from the database, the obtained data information corresponding to each field in the data table is the data information corresponding to the analysis target analysis service type, the data type corresponding to each field is converted into a numerical value form, the data corresponding to each field type is divided into a plurality of data mapping ranges, each data mapping range has a corresponding mapping pixel value, and then the data corresponding to each unit cell in the data table is mapped into a corresponding mapping pixel value, so that a corresponding two-dimensional image, namely a target image, is obtained, and therefore the time and the computing power for obtaining the data from the database and analyzing the data in the prior art are reduced.
Step S202, a trained target image analysis model is obtained, and feature extraction is performed on a target image based on the target image analysis model, so that various features corresponding to the target image are obtained.
The target image analysis model refers to a model for analyzing image characteristics, and can be a CNN (Convolutional Neural Networks) neural network model.
Specifically, in order to obtain the data characteristics reflected by the image features corresponding to the target image and to subsequently judge whether the target object accords with the object features corresponding to the target analysis service type, the feature extraction is performed on the target image through a target image analysis model which is obtained by training a data set corresponding to the target analysis service type in advance, so as to obtain various features in the target image, wherein the various features commonly reflect the data rule represented by the target image.
Step S204, based on a preset similarity algorithm, various features corresponding to the target image are matched with various reference features corresponding to the target analysis service type, and a target analysis result is obtained.
The preset similarity algorithm refers to an algorithm for analyzing similarity between different image features, and may be a manhattan distance algorithm. The reference feature refers to a feature reflecting the data characteristic or data rule corresponding to the target analysis service type, and is used as a reference standard for judging whether the data of the prediction object accords with the characteristic corresponding to the target analysis service type. The target analysis result refers to an analysis result of data corresponding to the target object, and may be a result of whether the target analysis result belongs to an object type corresponding to the target analysis service type.
Specifically, the preset similarity algorithm can judge the similarity between image features by calculating the Manhattan distance between corresponding feature vectors in different images, so that when whether an analysis target object accords with the object characteristics corresponding to the target analysis service type or not, the similarity degree between various features corresponding to the target image and various reference features corresponding to the target analysis service type can be calculated based on the preset similarity algorithm, and if the similarity degree between a certain type of feature corresponding to the target image and a certain type of reference feature in the target analysis service type reaches the preset similarity degree, or the similarity degree between multiple types of features corresponding to the target image and multiple types of reference features in the target analysis service type reaches the preset similarity degree, the condition that the target object has the characteristics corresponding to the target analysis service type is indicated; if none of the various features corresponding to the target image are similar to any of the reference features in the target analysis service type, the fact that the target object does not have the characteristics corresponding to the target analysis service type is indicated.
In the data analysis method, the data table corresponding to the target object in the target analysis service type is obtained, the data mapping range corresponding to each field type in the data table is determined, the data corresponding to each cell in the data table is mapped into the corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value, the target image is obtained, the trained target image analysis model is obtained, the feature extraction is carried out on the target image based on the target image analysis model, various features corresponding to the target image are obtained, various features corresponding to the target image are matched with various reference features corresponding to the target analysis service type based on a preset similarity algorithm, the target analysis result is obtained, a large amount of data is converted into images for analysis, the time and the calculation force spent in obtaining the data according to database query sentences and analyzing whether certain specific features or other types of features exist in the large amount of data are reduced, and accordingly the data analysis efficiency in the database is improved well.
In one embodiment, step S200 includes:
and step S300, performing outlier processing on the data table to obtain a data table to be converted.
Step S302, determining a plurality of data mapping ranges corresponding to each field type based on the change relation of each data corresponding to each field type in the data table to be converted.
The data table to be converted refers to a data table obtained after the abnormal value processing is performed on the original data table.
Specifically, in order to ensure that the converted two-dimensional image can better represent the condition of the data in the original database, before the data table is converted into the image, the abnormal value processing can be performed on the data table, and operations such as deleting, changing or supplementing the abnormal data can be performed, so that the data table to be converted with more perfect data information can be obtained. In addition, for the data corresponding to different field types, the corresponding data can be converted into the corresponding mapping value according to the rule of the preset mapping, for example, the data in the field A is a specific value or two values or three values, and at the moment, different values can be mapped into different pixel values; for example, after the data corresponding to a certain field B is changed, the data of the field B may be sequenced in advance, the sequenced data may be divided into a plurality of data mapping ranges, and the data of different data mapping ranges are mapped into different pixel values; for example, for a certain non-numeric field C, non-numeric data in the field C may be converted into numeric values according to a preset conversion algorithm, and then the converted numeric values may be mapped into corresponding pixel values according to a preset division rule.
In the above embodiment, the data table to be converted is obtained by performing outlier processing on the data table, and the plurality of data mapping ranges corresponding to the field types are determined based on the change relation of the data corresponding to the field types in the data table to be converted, so that the data is perfected and the data mapping ranges are divided, and the accurate and effective data information is provided for the execution of the subsequent steps, thereby improving the accuracy of the data analysis result to a certain extent.
In one embodiment, step S204 includes:
step S400, based on a preset similarity algorithm, respectively calculating the difference values between the feature vectors corresponding to the various features and the feature vectors corresponding to the various reference features, and respectively carrying out fusion operation on the difference values corresponding to the various features and the various reference features to obtain the similarity values corresponding to the various features and the various reference features.
In step S402, when the similarity value is greater than the preset similarity threshold, the target analysis result is that the target object belongs to the target type object corresponding to the target analysis service type.
In step S404, when the similarity value is not greater than the preset similarity threshold, the target analysis result is that the target object is not a target type object corresponding to the target analysis service type.
The preset similarity threshold value refers to a critical value for judging feature similarity. The target type object refers to an object processed corresponding to the target analysis service type.
Specifically, various reference features corresponding to the target analysis service type jointly reflect the features of the image obtained by conversion corresponding to a large number of data sets corresponding to the target analysis service type, all the features of the target analysis service type are integrated, the similarity value between various features of the target image and various reference features can be calculated based on a preset similarity algorithm, namely a Manhattan distance algorithm, a specific calculation process can be shown as a formula (1), A in the formula (1) is a feature vector of a certain type of feature corresponding to the target image, B is a feature vector of a certain type of reference feature corresponding to the target analysis service type, and F is a similarity value of A and B. If the calculated similarity value is larger than the preset similarity threshold value, the fact that the data characteristics corresponding to the target object have characteristics corresponding to the target service analysis type is indicated, and the target analysis result is that the target object is a target type object corresponding to the target analysis service type; if the calculated similarity values are smaller than or equal to the preset similarity threshold value, the data characteristics corresponding to the target object are not consistent with the characteristics corresponding to the target service analysis type, and the target analysis result is that the target object is a target type object corresponding to the target analysis service type. For example, after data (such as customer finance and flow data) of credit analysis business corresponding to a certain customer of a bank is converted into a target image and corresponding various characteristics are analyzed, similarity value calculation is carried out on various characteristics and various reference characteristics which belong to better credit in the credit analysis business, and when the calculated similarity value of a certain type of characteristics and a certain type of reference characteristics is greater than a preset similarity threshold value, the customer is indicated to be the customer with better credit.
F＝sum(abs(A-B)) (1)
In the above embodiment, based on a preset similarity algorithm, differences between feature vectors corresponding to various features and feature vectors corresponding to various reference features are calculated respectively, and fusion operation is performed on the differences between various features and various reference features to obtain similarity values corresponding to various features and various reference features, when the similarity values are larger than a preset similarity threshold, the target analysis result is that the target object belongs to the target type object corresponding to the target analysis service type, and when the similarity values are not larger than the preset similarity threshold, the target analysis result is that the target object does not belong to the target type object corresponding to the target analysis service type, so that similarity matching between various features corresponding to the target image and various reference features corresponding to the target analysis service type is realized, data characteristics corresponding to the target image are analyzed efficiently, and the data analysis efficiency corresponding to various service types is improved.
In one embodiment, before step S202, the method further includes:
step S500, a data table set to be processed corresponding to the target analysis service type is obtained, the data table set to be processed is divided based on a preset dimension to obtain a plurality of data sub-tables to be processed, a data mapping range corresponding to each field type in each data sub-table to be processed is determined, and data corresponding to the unit cells in each data sub-table to be processed are mapped to corresponding mapping pixel values based on the corresponding relation between each data mapping range and the mapping pixel values to obtain a data image to be processed corresponding to each data sub-table to be processed.
Step S502, inputting each image to be processed into an image analysis model, and extracting the characteristics of each image to be processed based on the image analysis model to obtain various comprehensive characteristics corresponding to each image to be processed.
Step S504, based on a preset similarity algorithm, comparing various comprehensive features corresponding to the images to be processed, fusing the comprehensive features corresponding to the images to be processed with similar comparison, obtaining various comprehensive features, and taking the various comprehensive features as various reference features corresponding to the target analysis service types.
The data table set to be processed refers to a data set for analyzing data characteristics corresponding to the target analysis service type. The preset dimension refers to a dimension for dividing the data table. The to-be-processed data sub-table refers to a data table obtained after dividing the data tables in the to-be-processed data table set. The image to be processed refers to an image for analyzing the characteristics corresponding to the target analysis service type. The image analysis model refers to a model for analyzing image characteristics, and may be a CNN network model. The feature to be integrated refers to the image feature corresponding to each image to be processed.
Specifically, in order to improve the analysis efficiency of the characteristics, before analyzing the data of a certain object in the data table corresponding to the target analysis service type, a set of to-be-processed data tables corresponding to the target analysis service type may be acquired first, and various data characteristics corresponding to the target analysis service type may be trained and learned from the set of to-be-processed data tables. In the process of analyzing various data characteristics corresponding to the target analysis service types, the data of the data table in the data table set to be processed can be firstly ordered according to preset dimensions, then the data belonging to the same object is divided into a data sub-table to be processed, the data corresponding to various data types in the data sub-table to be processed are converted into numerical values, then a plurality of different data mapping ranges can be preset for different field types, each data mapping range has a corresponding mapping pixel value, each cell in the data sub-table to be processed is mapped into a corresponding mapping pixel value, and accordingly a two-dimensional image to be processed corresponding to each data sub-table to be processed is obtained. Further, each image to be processed is input into an image analysis model to obtain various features to be synthesized corresponding to each image to be processed, in order to synthesize the characteristics corresponding to the target analysis service types, similarity among the features to be synthesized can be calculated based on a preset similarity algorithm, the similar features to be synthesized are fused to obtain comprehensive features, and the comprehensive features are further used as reference features corresponding to the target analysis service types.
In one embodiment, dissimilar characteristics in the characteristics to be integrated and various integrated characteristics obtained after the similar characteristics to be integrated are fused can be used as reference characteristics corresponding to the target analysis service type together, so that the influence of new characteristic types in the target analysis service type or the influence of the characteristic characteristics obtained by data analysis corresponding to the object types in the target analysis service type on the final analysis result is ensured.
In the above embodiment, a set of to-be-processed data tables corresponding to the target analysis service type is obtained, the set of to-be-processed data tables is divided based on a preset dimension to obtain a plurality of to-be-processed data sub-tables, a data mapping range corresponding to each field type in each to-be-processed data sub-table is determined, based on a corresponding relation between each data mapping range and a mapping pixel value, data corresponding to a unit cell in each to-be-processed data sub-table is mapped to a corresponding mapping pixel value to obtain to-be-processed images corresponding to each to-be-processed data sub-table, each to-be-processed image is input into an image analysis model, feature extraction is performed on each to-be-processed image based on the image analysis model to obtain each to-be-processed comprehensive feature corresponding to each to-be-processed image, based on a preset similarity algorithm, each to-be-processed comprehensive feature corresponding to each to-be-processed image is fused to obtain each comprehensive feature, each comprehensive feature corresponding to each to the comparison is used as each reference feature corresponding to the target analysis service type, analysis of data characteristics corresponding to the target analysis service type is completed, important comparison data is provided for the target object to the analysis service type, and the data corresponding to the target object is accurate in the analysis result is ensured to the target object.
In one embodiment, after step S204, further includes: and when the target object belongs to a target type object corresponding to the target analysis service type, transmitting the associated content corresponding to the target analysis service type to a terminal corresponding to the target object.
Specifically, when the target object belongs to a target type object corresponding to the target analysis service type, service content corresponding to the target analysis service type can be recommended to the target object, for example, whether a certain user is a financial attention person can be analyzed, and if the user is finally analyzed to belong to a financial fan, a banking financial product or content can be pushed to a terminal corresponding to the user.
In one embodiment, a description will be given of an example of whether a certain customer is a fraudulent customer in a banking analysis service. Before analysis, the data table sets belonging to the cheated clients in the bank, such as running water, bills and the like, can be converted into corresponding image sets to be trained in advance, then the image sets to be trained are analyzed through a CNN model capable of analyzing image features, various features corresponding to the images in the image sets to be trained are further obtained through analysis, the similarity degree among the various features corresponding to the images to be trained is calculated based on a Manhattan distance algorithm, the similar features among the images to be trained are fused, and therefore fused comprehensive features and unfused features are obtained, the comprehensive features represent the features shared by the fused images to be trained, the features corresponding to the cheated clients can be represented more strongly, and when the features of the similar comprehensive features are analyzed from the images converted from the data table corresponding to the predicted clients, the predicted clients are more likely to be the cheated clients. In addition, unfused features can also be applied to the judgment of predicting whether the client is a fraudulent client so as to improve the influence of few features on the accuracy of the judgment result.
In one embodiment, the data of the target object corresponding to the target analysis service type may be further analyzed at the cut-off time, the updated recorded data is obtained at this time, the updated data is converted into the corresponding two-dimensional image, the features of the two-dimensional image are analyzed, and if the new features are analyzed, the new features may be used as new comparison features, or the expansion of other service types may be analyzed from the new features.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data analysis device for realizing the data analysis method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data analysis device provided below may refer to the limitation of the data analysis method hereinabove, and will not be repeated herein.
In one embodiment, as shown in fig. 3, there is provided a data analysis apparatus including: a conversion module 300, an extraction module 302, and a matching module 304, wherein:
the conversion module 300 is configured to obtain a data table corresponding to a target object in a target analysis service type, determine each data mapping range corresponding to each field type in the data table, and map data corresponding to each cell in the data table to a corresponding mapping pixel value based on a corresponding relationship between each data mapping range and the mapping pixel value, so as to obtain a target image.
The extracting module 302 is configured to obtain a trained target image analysis model, and perform feature extraction on the target image based on the target image analysis model to obtain various features corresponding to the target image.
And the matching module 304 is configured to match various features corresponding to the target image with various reference features corresponding to the target analysis service type based on a preset similarity algorithm, so as to obtain a target analysis result.
In one embodiment, the conversion module 300 is further configured to perform outlier processing on the data table to obtain a data table to be converted; and determining a plurality of data mapping ranges corresponding to the field types based on the change relation of the data corresponding to the field types in the data table to be converted.
In one embodiment, the matching module 304 is further configured to calculate differences between feature vectors corresponding to the various features and feature vectors corresponding to the various reference features based on a preset similarity algorithm, and perform a fusion operation on the differences between the various features and the various reference features to obtain similarity values corresponding to the various features and the various reference features; when the similarity value is larger than a preset similarity threshold value, the target analysis result is that the target object belongs to a target type object corresponding to the target analysis service type; and when the similarity value is not greater than the preset similarity threshold value, the target analysis result is that the target object does not belong to the target type object corresponding to the target analysis service type.
In one embodiment, the data analysis device further includes a processing module 306, configured to obtain a set of to-be-processed data tables corresponding to the target analysis service type, divide the set of to-be-processed data tables based on a preset dimension to obtain a plurality of to-be-processed data sub-tables, determine a data mapping range corresponding to each field type in each to-be-processed data sub-table, and map data corresponding to a cell in each to-be-processed data sub-table to a corresponding mapping pixel value based on a corresponding relationship between each data mapping range and a mapping pixel value, so as to obtain a to-be-processed image corresponding to each to-be-processed data sub-table; inputting each image to be processed into an image analysis model, and extracting the characteristics of each image to be processed based on the image analysis model to obtain various comprehensive characteristics corresponding to each image to be processed; based on the preset similarity algorithm, various to-be-synthesized features corresponding to the to-be-processed images are compared, the to-be-synthesized features corresponding to the to-be-processed images with similar comparison are fused, various to-be-synthesized features are obtained, and the various to-be-synthesized features are used as reference similar features corresponding to the target analysis service types.
In one embodiment, the data analysis device further includes a sending module 308, configured to send, when the target object belongs to a target type object corresponding to the target analysis service type, the associated content corresponding to the target analysis service type to a terminal corresponding to the target object.
The respective modules in the above-described data analysis apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 4. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store data related to the execution process. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data analysis method.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data analysis method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the structures shown in fig. 4 and 5 are block diagrams of only portions of structures associated with the present application and are not intended to limit the computer device to which the present application is applied, and that a particular computer device may include more or less elements than those shown, or may be combined with certain elements, or have different arrangements of elements.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, storing a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
In one embodiment, a computer program product or computer program is provided that includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-described method embodiments.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method of data analysis, the method comprising:
acquiring a data table corresponding to a target object in a target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image;
Acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image;
and based on a preset similarity algorithm, matching various features corresponding to the target image with various reference features corresponding to the target analysis service type to obtain a target analysis result.
2. The method of claim 1, wherein determining each data mapping range corresponding to each field type in the data table comprises:
performing outlier processing on the data table to obtain a data table to be converted;
and determining a plurality of data mapping ranges corresponding to the field types based on the change relation of the data corresponding to the field types in the data table to be converted.
3. The method of claim 1, wherein the matching, based on a preset similarity algorithm, the various features corresponding to the target image with the various reference features corresponding to the target analysis service type to obtain a target analysis result includes:
based on a preset similarity algorithm, respectively calculating the difference values between the feature vectors corresponding to the various features and the feature vectors corresponding to the various reference features, and respectively carrying out fusion operation on the difference values corresponding to the various features and the various reference features to obtain the similarity values corresponding to the various features and the various reference features;
When the similarity value is larger than a preset similarity threshold value, the target analysis result is that the target object belongs to a target type object corresponding to the target analysis service type;
and when the similarity value is not greater than the preset similarity threshold value, the target analysis result is that the target object does not belong to the target type object corresponding to the target analysis service type.
4. The method of claim 3, further comprising, prior to the acquiring the trained image analysis model:
acquiring a to-be-processed data table set corresponding to a target analysis service type, dividing the to-be-processed data table set based on a preset dimension to obtain a plurality of to-be-processed data sub-tables, determining a data mapping range corresponding to each field type in each to-be-processed data sub-table, and mapping data corresponding to unit cells in each to-be-processed data sub-table into corresponding mapping pixel values based on a corresponding relation between each data mapping range and a mapping pixel value to obtain to-be-processed images corresponding to each to-be-processed data sub-table;
inputting each image to be processed into an image analysis model, and extracting the characteristics of each image to be processed based on the image analysis model to obtain various comprehensive characteristics corresponding to each image to be processed;
Based on the preset similarity algorithm, various to-be-synthesized features corresponding to the to-be-processed images are compared, the to-be-synthesized features corresponding to the to-be-processed images with similar comparison are fused, various to-be-synthesized features are obtained, and the various to-be-synthesized features are used as various reference features corresponding to the target analysis service types.
5. The method according to claim 3, wherein the matching, based on a preset similarity algorithm, the various features corresponding to the target image with the various reference features corresponding to the target analysis service type, to obtain a target analysis result, further includes:
and when the target object belongs to a target type object corresponding to the target analysis service type, sending the associated content corresponding to the target analysis service type to a terminal corresponding to the target object.
6. A data analysis device, the device comprising:
the conversion module is used for acquiring a data table corresponding to the target object in the target analysis service type, determining each data mapping range corresponding to each field type in the data table, and mapping data corresponding to each unit cell in the data table into a corresponding mapping pixel value based on the corresponding relation between each data mapping range and the mapping pixel value to obtain a target image;
The extraction module is used for acquiring a trained target image analysis model, and extracting features of the target image based on the target image analysis model to obtain various features corresponding to the target image;
and the matching module is used for matching various features corresponding to the target image with various reference features corresponding to the target analysis service type based on a preset similarity algorithm to obtain a target analysis result.
7. The apparatus of claim 6, further comprising a processing module, configured to obtain a set of to-be-processed data tables corresponding to a target analysis service type, divide the set of to-be-processed data tables based on a preset dimension to obtain a plurality of to-be-processed data sub-tables, determine a data mapping range corresponding to each field type in each to-be-processed data sub-table, and map data corresponding to a cell in each to-be-processed data sub-table to a corresponding mapping pixel value based on a corresponding relation between each data mapping range and a mapping pixel value to obtain a to-be-processed image corresponding to each to-be-processed data sub-table; inputting each image to be processed into an image analysis model, and extracting the characteristics of each image to be processed based on the image analysis model to obtain various comprehensive characteristics corresponding to each image to be processed; based on the preset similarity algorithm, various to-be-synthesized features corresponding to the to-be-processed images are compared, the to-be-synthesized features corresponding to the to-be-processed images with similar comparison are fused, various to-be-synthesized features are obtained, and the various to-be-synthesized features are used as reference similar features corresponding to the target analysis service types.
8. The apparatus of claim 6, wherein the conversion module is further configured to perform outlier processing on the data table to obtain a data table to be converted; and determining a plurality of data mapping ranges corresponding to the field types based on the size relation among the data corresponding to the field types in the data table to be converted.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311108474.3A CN117390098A (en) | 2023-08-30 | 2023-08-30 | Data analysis method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311108474.3A CN117390098A (en) | 2023-08-30 | 2023-08-30 | Data analysis method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117390098A true CN117390098A (en) | 2024-01-12 |
Family
ID=89436238
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311108474.3A Pending CN117390098A (en) | 2023-08-30 | 2023-08-30 | Data analysis method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117390098A (en) |
-
2023
- 2023-08-30 CN CN202311108474.3A patent/CN117390098A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9454580B2 (en) | Recommendation system with metric transformation | |
| CN115795000A (en) | Joint similarity algorithm comparison-based enclosure identification method and device | |
| CN116630630B (en) | Semantic segmentation method, semantic segmentation device, computer equipment and computer readable storage medium | |
| CN116030466B (en) | Image text information identification and processing method and device and computer equipment | |
| CN117035980A (en) | Resource borrowing evaluation method, device, computer equipment and storage medium | |
| CN115758271A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN117390098A (en) | Data analysis method, device, computer equipment and storage medium | |
| CN116894802B (en) | Image enhancement method, device, computer equipment and storage medium | |
| CN117576645B (en) | Parking space detection method and device based on BEV visual angle and computer equipment | |
| CN115601550B (en) | Model determination method, model determination device, computer equipment and computer readable storage medium | |
| CN117238017A (en) | Face recognition method, device, computer equipment and storage medium | |
| CN116342242A (en) | Abnormality detection method, abnormality detection device, computer device, and storage medium | |
| CN115630973A (en) | User data processing method, device, computer equipment and storage medium | |
| CN117407418A (en) | Information acquisition method, information acquisition device, computer apparatus, storage medium, and program product | |
| CN117150311A (en) | Data processing method, device, equipment and storage medium | |
| CN116703544A (en) | Resource interaction track prediction method and device, computer equipment and storage medium | |
| CN116719994A (en) | Media object recommendation method, device, computer equipment and storage medium | |
| CN116127183A (en) | Service recommendation method, device, computer equipment and storage medium | |
| CN116866419A (en) | Information pushing method, device, computer equipment and storage medium | |
| CN116343013A (en) | Image processing acceleration method, device, computer equipment and storage medium | |
| CN115658899A (en) | Text classification method and device, computer equipment and storage medium | |
| CN116932935A (en) | Address matching method, device, equipment, medium and program product | |
| CN117915272A (en) | Terminal movement track display method, device, equipment and medium | |
| CN118154300A (en) | Mortgage parameter processing method, mortgage parameter processing device, computer equipment and storage medium | |
| CN117710107A (en) | Resource value prediction method, apparatus, device, storage medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |