CN109241231A - The accurately pretreatment unit and method of diagram data - Google Patents
The accurately pretreatment unit and method of diagram data Download PDFInfo
- Publication number
- CN109241231A CN109241231A CN201811046182.0A CN201811046182A CN109241231A CN 109241231 A CN109241231 A CN 109241231A CN 201811046182 A CN201811046182 A CN 201811046182A CN 109241231 A CN109241231 A CN 109241231A
- Authority
- CN
- China
- Prior art keywords
- matrix
- data
- original
- covariance
- original characteristic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 22
- 238000010586 diagram Methods 0.000 title abstract description 4
- 239000011159 matrix material Substances 0.000 claims abstract description 81
- 230000009467 reduction Effects 0.000 claims abstract description 30
- 239000013598 vector Substances 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims description 18
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 230000002159 abnormal effect Effects 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 230000000875 corresponding effect Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000002547 anomalous effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses the pretreatment units and method of a kind of accurately diagram data.Wherein, which comprises obtain initial data；Primitive character matrix X is constructed according to the M primitive character values and N number of data record；Covariance matrix C is constructed according to the primitive character matrix X；Calculate the covariance eigenvalue and feature vector of the covariance matrix C；Dimensionality reduction matrix P is constructed according to the covariance eigenvalue of the covariance matrix C and described eigenvector；The product of primitive character matrix X and the dimensionality reduction matrix X are calculated as the data set after the initial data dimensionality reduction.The present invention carries out dimension-reduction treatment to initial data, reduces data processing difficulty, improves the efficiency for constructing and updating to accurately diagram data.
Description
Technical Field
The invention relates to the field of high-precision maps, in particular to a high-precision map data preprocessing device and method.
Background
In the field of high-precision maps, a large amount of original vision data related to road environments can be generated through sensing equipment such as laser radars, cameras and vehicle-mounted sensors. And identifying traffic elements such as marking lines, road edges, guideboards, traffic lights, stop lines and the like of the passage.
The premise for realizing the data fusion is to preprocess the original-view data with different data types and formats. Since the preprocessing of the raw data is directed to complex road environments, incomplete or inconsistent or anomalous difference data exists in large amounts of raw data. The difference data can cause problems in the data preprocessing process; when the original data relate to too many characteristic values, the dimensionality disaster of the data can be caused, and the abnormal consumption of the data processing capacity of the computer is caused.
Disclosure of Invention
The embodiment of the invention at least provides a high-precision map data preprocessing method, which can solve the problem of low data processing efficiency caused by excessive characteristic values of high-precision map data in the prior art.
The specific implementation of the above embodiment is as follows.
The method comprises the following steps:
step100, acquiring N original data including M original characteristic values for establishing a high-precision map, wherein N and M are positive integers greater than or equal to 1;
step200, constructing an original characteristic matrix X according to the M original characteristic values and the N data records;
step300, constructing a covariance matrix C according to the original characteristic matrix X;
step400, calculating a covariance eigenvalue and an eigenvector of the covariance matrix C;
step500, constructing a dimensionality reduction matrix P according to the covariance eigenvalue and the eigenvector;
and Step600, calculating the product of the original characteristic matrix X and the dimension reduction matrix X to be the data set of the original data after dimension reduction.
In an embodiment, Step100 preferably includes:
step110, acquiring environment data of analog quantity through an environment sensing element;
and Step220, performing analog-to-digital conversion on the environment data.
In an embodiment, the raw data is vehicle trajectory data or vehicle state data or traffic sign data or road sign data or vehicle anomaly data or road environment data or city POI data.
Preferably, in the embodiment, Step300 includes:
step310, solving an original characteristic mean value of all the original characteristic values;
step320, performing zero equalization processing on all the original characteristic values of the original characteristic matrix X according to the original characteristic average value;
step330, constructing a covariance matrix C,
preferably, Step400 is configured to perform singular value decomposition on the covariance matrix C to obtain covariance eigenvalues and eigenvectors.
Preferably, in the embodiment, Step500 includes:
step510, arranging the eigenvectors corresponding to the covariance eigenvalues according to the size of the covariance eigenvalues;
step520, establishing a characteristic vector matrix Z according to the arranged characteristic vectors;
and Step530, selecting the first K rows of the feature vector matrix Z to construct a dimensionality reduction matrix P, wherein K is smaller than N and is a positive integer.
In an embodiment, after Step600, the method further includes:
and Step700, compressing binary symbols of the data set by using Huffman coding.
The embodiment of the invention at least provides another high-precision map data preprocessing device, which comprises:
the acquisition module is used for acquiring N pieces of original data including M original characteristic values for establishing a high-precision map, wherein N and M are positive integers greater than or equal to 1;
the matrix module is used for constructing an original characteristic matrix X according to the M original characteristic values and the N data records, constructing a covariance matrix C according to the original characteristic matrix X, calculating a covariance characteristic value and a characteristic vector of the covariance matrix C, and constructing a dimension reduction matrix P according to the covariance characteristic value and the characteristic vector;
and the dimension reduction module is used for calculating the product of the original characteristic matrix X and the dimension reduction matrix X to be the data set after dimension reduction of the original data.
In an embodiment, preferably, the constructing a covariance matrix C according to the original feature matrix X includes:
the matrix module is used for solving the original characteristic mean value of all the original characteristic values; performing zero equalization processing on all the original characteristic values of the original characteristic matrix X according to the original characteristic average value; a covariance matrix C is constructed which,
in an embodiment, the apparatus preferably includes a storage module that compresses binary symbols of the data set according to a selected huffman code.
In view of the above, other features and advantages of the disclosed exemplary embodiments will become apparent from the following detailed description of the disclosed exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a flow chart of the method of the present embodiment;
FIG. 2 is a schematic diagram of the apparatus of the present embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment provides a method special for dimensionality reduction of original data with more characteristic values in high-precision map data. Wherein, the original data is generally obtained by various sensing elements or artificial indexing; the raw data of this embodiment may be at least the following.
In order to perform dimension reduction processing on the raw data including a large number of feature values, the method of this embodiment includes the following steps:
step100, acquiring M groups of data records of any one of the original data. And determining that the original data comprises M original characteristic values by using a table look-up or according to the output description of the sensing element. N and M are positive integers greater than or equal to 1;
step200, constructing an original characteristic matrix X according to the M original characteristic values and the N data records; wherein,
and Step300, calculating a covariance matrix of the original characteristic matrix X.
The general covariance formula is that,is the characteristic mean.
X, Y sample and when the covariance formula is positive, it indicates that X and Y are positively correlated; when the covariance formula is negative, X and Y are negative correlation relations, and when the covariance formula is 0, X and Y are independent.
In consideration of the data characteristics of the original data matrix X, the present embodiment simplifies the covariance as follows.
Step310, calculating the original characteristic mean value of all the original characteristic values.
Step320, performing zero equalization processing on all original characteristic values of the original characteristic matrix X according to the original characteristic average value, even if the original characteristic average value is 0,
the simplified covariance formula is then,
step330, since the original data is N-dimensional data and belongs to symmetric covariance, the covariance matrix C is,
step400, in this embodiment, singular value decomposition is performed on the covariance matrix C, and the covariance eigenvalue and eigenvector of the covariance matrix C are calculated.
Step500, constructing a dimension reduction matrix P according to the covariance eigenvalue and the eigenvector;
step510, carrying out sequence arrangement on the eigenvectors corresponding to the covariance eigenvalues according to the size of the covariance eigenvalues;
step520, establishing a characteristic vector matrix Z according to the arranged characteristic vectors;
and Step530, selecting the first K rows of the feature vector matrix Z to construct a dimensionality reduction matrix P, wherein K is smaller than N and is a positive integer.
Step600, calculating the product of the original characteristic matrix X and the dimensionality reduction matrix X, namely PX is used as the data set of original data after dimensionality reduction.
The acquired data set is obtained by reducing the dimension of the original data.
Furthermore, the original data after dimensionality reduction is stored or even encrypted before being used for constructing high-precision map data, so that the storage capacity and the safety of a unit memory can be improved.
And Step700, compressing binary symbols of the data set by using Huffman coding.
Step810, encrypting the compressed data set into a data file by selecting a DES key;
and Step820, selecting an RSA public key to encrypt the DES secret key and writing the encrypted data file, and storing the written data file in the unit memory.
By combining the scheme, the embodiment encrypts the original data after binary compression, and decrypts the original data by the RSA private key and the DES key matched with the RSA public key before the geographic information service system calls the stored data file, so that the storage efficiency and the data security are effectively improved.
This embodiment additionally discloses a preprocessing device of high accuracy map data, and the device includes:
the acquisition module is used for acquiring N pieces of original data including M original characteristic values for establishing a high-precision map, wherein N and M are positive integers greater than or equal to 1;
the matrix module is used for solving the original characteristic mean value of all the original characteristic values; performing zero equalization processing on all original characteristic values of the original characteristic matrix X according to the original characteristic mean value; a covariance matrix C is constructed which,calculating a covariance eigenvalue and an eigenvector of the covariance matrix C, and constructing a dimension reduction matrix P according to the covariance eigenvalue and the eigenvector;
and the dimension reduction module is used for calculating the product of the original characteristic matrix X and the dimension reduction matrix X to be the data set after dimension reduction of the original data.
And the storage module is used for compressing the binary symbols of the data set according to the selected Huffman codes.
The encryption module is used for encrypting the compressed data set into a data file by selecting a DES key; and selecting an RSA public key to encrypt the DES secret key and write the encrypted data file into the DES secret key, and storing the written data file in the unit memory.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A preprocessing method of high-precision map data is used for reducing dimension of original data in a preprocessing process and is characterized in that,
the method comprises the following steps:
step100, acquiring N original data including M original characteristic values for establishing a high-precision map, wherein N and M are positive integers greater than or equal to 1;
step200, constructing an original characteristic matrix X according to the M original characteristic values and the N data records;
step300, constructing a covariance matrix C according to the original characteristic matrix X;
step400, calculating a covariance eigenvalue and an eigenvector of the covariance matrix C;
step500, constructing a dimensionality reduction matrix P according to the covariance eigenvalue and the eigenvector;
and Step600, calculating the product of the original characteristic matrix X and the dimension reduction matrix X to be the data set of the original data after dimension reduction.
2. The preprocessing method of high precision map data according to claim 1,
before Step100, the method comprises the following steps:
step110, acquiring environment data of analog quantity through an environment sensing element;
and Step220, performing analog-to-digital conversion on the environment data.
3. The preprocessing method of high precision map data according to claim 2,
the original data is vehicle track data or vehicle state data or traffic marking data or road marking data or vehicle abnormal data or road environment data or city POI data.
4. The preprocessing method of high precision map data according to claim 1,
the Step300 comprises the following steps:
step310, solving an original characteristic mean value of all the original characteristic values;
step320, performing zero equalization processing on all the original characteristic values of the original characteristic matrix X according to the original characteristic average value;
step330, constructing a covariance matrix C,。
5. the preprocessing method of high precision map data according to claim 4,
and Step400, performing singular value decomposition on the covariance matrix C to obtain covariance eigenvalues and eigenvectors.
6. The preprocessing method of high precision map data according to claim 1,
the Step500 comprises the following steps:
step510, arranging the eigenvectors corresponding to the covariance eigenvalues according to the size of the covariance eigenvalues;
step520, establishing a characteristic vector matrix Z according to the arranged characteristic vectors;
and Step530, selecting the first K rows of the feature vector matrix Z to construct a dimensionality reduction matrix P, wherein K is smaller than N and is a positive integer.
7. The preprocessing method of high precision map data according to claim 1,
after Step600, the method comprises the following steps:
and Step700, compressing binary symbols of the data set by using Huffman coding.
8. An apparatus for preprocessing high-precision map data, characterized by comprising:
the acquisition module is used for acquiring N pieces of original data including M original characteristic values for establishing a high-precision map, wherein N and M are positive integers greater than or equal to 1;
the matrix module is used for constructing an original characteristic matrix X according to the M original characteristic values and the N data records, constructing a covariance matrix C according to the original characteristic matrix X, calculating a covariance characteristic value and a characteristic vector of the covariance matrix C, and constructing a dimension reduction matrix P according to the covariance characteristic value and the characteristic vector;
and the dimension reduction module is used for calculating the product of the original characteristic matrix X and the dimension reduction matrix X to be the data set after dimension reduction of the original data.
9. The apparatus for preprocessing high precision map data according to claim 8,
the constructing of the covariance matrix C according to the original feature matrix X comprises:
the matrix module is used for solving the original characteristic mean value of all the original characteristic values; performing zero equalization processing on all the original characteristic values of the original characteristic matrix X according to the original characteristic average value; a covariance matrix C is constructed which,
10. the apparatus for preprocessing high precision map data according to claim 8,
the device comprises a storage module, wherein the storage module compresses binary symbols of the data set according to the selected Huffman codes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811046182.0A CN109241231A (en) | 2018-09-07 | 2018-09-07 | The accurately pretreatment unit and method of diagram data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811046182.0A CN109241231A (en) | 2018-09-07 | 2018-09-07 | The accurately pretreatment unit and method of diagram data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109241231A true CN109241231A (en) | 2019-01-18 |
Family
ID=65060183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811046182.0A Pending CN109241231A (en) | 2018-09-07 | 2018-09-07 | The accurately pretreatment unit and method of diagram data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109241231A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112241748A (en) * | 2019-07-16 | 2021-01-19 | 广州汽车集团股份有限公司 | Data dimension reduction method and device based on multi-source information entropy difference |
| CN114723922A (en) * | 2022-02-24 | 2022-07-08 | 北京深势科技有限公司 | Three-dimensional structure data contrast presentation method and device based on data dimension reduction |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090028455A1 (en) * | 2004-06-03 | 2009-01-29 | Japan Science And Technology Agency | High-speed high-accuracy matrix singular value decomposition method, program, and device |
| CN106331615A (en) * | 2016-08-22 | 2017-01-11 | 何颖 | Intelligent traffic information acquisition system and static information acquisition method thereof |
| CN106407363A (en) * | 2016-09-08 | 2017-02-15 | 电子科技大学 | Ultra-high-dimensional data dimension reduction algorithm based on information entropy |
| CN107977468A (en) * | 2017-12-21 | 2018-05-01 | 横琴国际知识产权交易中心有限公司 | A kind of transmission method and system of sparse type data file |
| CN108106500A (en) * | 2017-12-21 | 2018-06-01 | 中国舰船研究设计中心 | A kind of missile target kind identification method based on multisensor |
-
2018
- 2018-09-07 CN CN201811046182.0A patent/CN109241231A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090028455A1 (en) * | 2004-06-03 | 2009-01-29 | Japan Science And Technology Agency | High-speed high-accuracy matrix singular value decomposition method, program, and device |
| CN106331615A (en) * | 2016-08-22 | 2017-01-11 | 何颖 | Intelligent traffic information acquisition system and static information acquisition method thereof |
| CN106407363A (en) * | 2016-09-08 | 2017-02-15 | 电子科技大学 | Ultra-high-dimensional data dimension reduction algorithm based on information entropy |
| CN107977468A (en) * | 2017-12-21 | 2018-05-01 | 横琴国际知识产权交易中心有限公司 | A kind of transmission method and system of sparse type data file |
| CN108106500A (en) * | 2017-12-21 | 2018-06-01 | 中国舰船研究设计中心 | A kind of missile target kind identification method based on multisensor |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112241748A (en) * | 2019-07-16 | 2021-01-19 | 广州汽车集团股份有限公司 | Data dimension reduction method and device based on multi-source information entropy difference |
| CN114723922A (en) * | 2022-02-24 | 2022-07-08 | 北京深势科技有限公司 | Three-dimensional structure data contrast presentation method and device based on data dimension reduction |
| CN114723922B (en) * | 2022-02-24 | 2023-04-18 | 北京深势科技有限公司 | Three-dimensional structure data contrast presentation method and device based on data dimension reduction |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104751055B (en) | A kind of distributed malicious code detecting method, apparatus and system based on texture | |
| CN102930192B (en) | Security information generates and acquisition methods and relevant apparatus | |
| CN101706946B (en) | Digital image encryption method based on DNA sequence and multi-chaotic mapping | |
| EP3172700B1 (en) | Improved method for fingerprint matching and camera identification, device and system | |
| CN110770725B (en) | Data processing method and device | |
| CN109241231A (en) | The accurately pretreatment unit and method of diagram data | |
| CN110913092B (en) | Reversible information hiding method for encrypted image | |
| US10083194B2 (en) | Process for obtaining candidate data from a remote storage server for comparison to a data to be identified | |
| CN109359276B (en) | Anti-counterfeiting code generation method, device, equipment and storage medium | |
| CN112073444B (en) | Data set processing method and device and server | |
| CN109145639A (en) | File encrypting method, decryption method and device | |
| CN114298211A (en) | Feature binning method and device, electronic equipment and storage medium | |
| CN112819156A (en) | Data processing method, device and equipment | |
| JP5714785B2 (en) | How to encrypt multiple data in a protected set | |
| CN116611107A (en) | Data processing method, device and equipment, chip, board card and medium | |
| CN115002196B (en) | Data processing method and device and vehicle end acquisition equipment | |
| CN112669068B (en) | Market research data transmission method and system based on big data | |
| CN105654156A (en) | Examination paper collection method and system for computer examinations | |
| CN100403765C (en) | Information processing apparatus and its control method, computer program and storage medium | |
| CN112434316A (en) | Data encryption method, data retrieval method and device for database | |
| CN112364367A (en) | Object processing method, device and equipment based on privacy protection | |
| CN107992555B (en) | Method for storing and reading raster data | |
| Vijayanandh et al. | Tamper detection of medical images using statistical moments against various attacks | |
| CN111680722B (en) | Content identification method, device, equipment and readable storage medium | |
| CN105184733B (en) | A kind of improved high-fidelity vulnerable watermark method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190118 |
|
| RJ01 | Rejection of invention patent application after publication |