CN109471860B - Large-scale charging pile data processing method and device for electric vehicle charging network - Google Patents
Large-scale charging pile data processing method and device for electric vehicle charging network Download PDFInfo
- Publication number
- CN109471860B CN109471860B CN201811284988.3A CN201811284988A CN109471860B CN 109471860 B CN109471860 B CN 109471860B CN 201811284988 A CN201811284988 A CN 201811284988A CN 109471860 B CN109471860 B CN 109471860B
- Authority
- CN
- China
- Prior art keywords
- charging pile
- pile data
- data
- storage node
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/18—Payment architectures involving self-service terminals [SST], vending machines, kiosks or multimedia terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/30—Payment architectures, schemes or protocols characterised by the use of specific devices or networks
- G06Q20/34—Payment architectures, schemes or protocols characterised by the use of specific devices or networks using cards, e.g. integrated circuit [IC] cards or magnetic cards
- G06Q20/341—Active cards, i.e. cards including their own processing means, e.g. including an IC or chip
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T90/00—Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation
- Y02T90/10—Technologies relating to charging of electric vehicles
- Y02T90/16—Information or communication technologies improving the operation of electric vehicles
- Y02T90/167—Systems integrating technologies related to power network operation and communication or information technologies for supporting the interoperability of electric or hybrid vehicles, i.e. smartgrids as interface for battery charging of electric vehicles [EV] or hybrid vehicles [HEV]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S30/00—Systems supporting specific end-user applications in the sector of transportation
- Y04S30/10—Systems supporting the interoperability of electric or hybrid vehicles
- Y04S30/12—Remote or cooperative charging
Abstract
The application provides a large-scale charging pile data processing method and device for an electric vehicle charging network, which are applied to a server and comprise the following steps: receiving charging pile data sent by a charging pile; determining a database storage node corresponding to the charging pile data according to a first attribute value corresponding to a preset key field; performing hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to charging pile data; determining a target storage area of the charging pile data in a database storage node according to the hash value corresponding to the charging pile data; and storing the charging pile data into the corresponding target storage area. The key fields and the distribution key fields of the charging pile data are selected, the target storage area of the charging pile data in the database storage node is located, any field information of the charging pile data can be flexibly read and processed, a user does not need to carry out conversion configuration, the operation complexity is low, the flexibility is high, and the method and the device are suitable for frequently updating scenes of certain attribute fields.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a large-scale charging pile data processing method and device for an electric vehicle charging network.
Background
The charging pile real-time data comprises information of a charging pile such as real-time state and real-time interaction, and is one of important information for knowing the running condition of the charging network of the electric automobile. However, the charging pile is large in scale, the real-time state information is more, the updating frequency is high, and the charging pile data cannot be stored by adopting the traditional relational database storage.
At present, a mode of adopting json character string storage is provided to save charging pile data, and the concrete storage process is as follows: the method comprises the steps of selecting a charging pile number of charging pile data as a key to be stored by using the simplest key-value mode, organizing all other fields in the charging pile number into json format to be stored in a value, and obtaining the value spliced by all other fields by using a command get 11101900000000000025.
However, in the above storage method, it takes a long time to store and analyze the json string, and when a certain field is updated, it is not possible to directly read the value and update the certain field, and all the value fields must be updated at once.
Disclosure of Invention
In view of this, an embodiment of the present application aims to provide a method and an apparatus for processing large-scale charging pile data for an electric vehicle charging network, where any field information of the charging pile data can be flexibly read and updated in a hash storage manner, a user does not need to perform conversion configuration, and the method and the apparatus are low in operation complexity and high in flexibility.
In a first aspect, an embodiment of the present application provides a data processing method for a large-scale charging pile facing an electric vehicle charging network, which is applied to a server and includes:
receiving charging pile data sent by a charging pile; each charging pile data comprises attribute values corresponding to the fields respectively;
determining a database storage node corresponding to the charging pile data according to a first attribute value corresponding to a preset key field;
performing hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to the charging pile data;
determining a target storage area of the charging pile data in the database storage node according to the hash value corresponding to the charging pile data;
and storing the charging pile data into the corresponding target storage area.
With reference to the first aspect, an embodiment of the present application provides a first possible implementation manner of the first aspect, where the method includes:
receiving DML operation which is sent by a user terminal and carries an operation field; wherein the DML operation further carries the first attribute value and the second attribute value;
performing hash operation on the second attribute value in the DML operation to obtain a hash value of the second attribute value;
determining a target database storage node according to the first attribute;
sending the DML operation carrying the hash value of the operation field and the second attribute value to the target database storage node, so that the target database storage node operates the first target charging pile data corresponding to the operation field in a target storage area corresponding to the second attribute value;
and receiving an operation result returned by the storage node of the target database, and forwarding the operation result to the user terminal.
With reference to the first aspect, an embodiment of the present application provides a second possible implementation manner of the first aspect, where each charging pile data stored in a target storage area of the database storage node corresponds to an aging time; after the charging pile data are stored in the corresponding target storage areas, the method further comprises the following steps:
monitoring the aging time of charging pile data stored in a target storage area of a database storage node;
and if the second target charging pile data with the aging time reaching the preset threshold value is detected, deleting the second target charging pile data.
In combination with the first aspect, the present examples provide a third possible implementation manner of the first aspect, wherein,
after receiving the charging pile data sent by the charging pile, the method further comprises the following steps:
judging whether the received charging pile data meet the current service identification;
if yes, responding to the received charging pile data;
and if not, discarding the charging pile data.
In combination with the third possible implementation manner of the first aspect, the present application provides an example of a fourth possible implementation manner of the first aspect, wherein,
the number of the servers is multiple, and the method further comprises the following steps:
and for any server, judging whether the charging pile data are stored in the database storage node or not after the server receives the charging pile data, and responding to the received charging pile data if the charging pile data are not stored in the database storage node.
In a second aspect, an embodiment of the present application provides an electric vehicle charging network-oriented large-scale charging pile data processing apparatus, including:
the receiving module is used for receiving charging pile data sent by the charging pile; each charging pile data comprises attribute values corresponding to the fields respectively;
the determining module is used for determining a database storage node corresponding to the charging pile data according to a first attribute value corresponding to a preset key field;
the hash operation module is used for carrying out hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to the charging pile data;
the determining module is further configured to determine a target storage area of the charging pile data in the database storage node according to the hash value corresponding to the charging pile data;
and the storage module is used for storing the charging pile data into the corresponding target storage area.
In combination with the second aspect, embodiments of the present application provide a first possible implementation manner of the second aspect, where the apparatus further includes:
the receiving module is further configured to receive a DML operation which is sent by the user terminal and carries an operation field; wherein the DML operation further carries the first attribute value and the second attribute value;
the hash operation module is further configured to perform hash operation on the second attribute value in the DML operation to obtain a hash value of the second attribute value;
the determining module is further used for determining a target database storage node according to the first attribute;
the sending module is further configured to send the DML operation carrying the hash value of the operation field and the second attribute value to the target database storage node, so that the target database storage node operates the first target charging pile data corresponding to the operation field in the target storage area corresponding to the second attribute value;
the receiving module is further configured to receive an operation result returned by the target database storage node;
the sending module is further configured to forward the operation result to the user terminal.
With reference to the second aspect, embodiments of the present application provide a second possible implementation manner of the second aspect, where the apparatus further includes:
each charging pile data stored in a target storage area of the database storage node corresponds to aging time; the device further comprises:
the monitoring module is used for monitoring the aging time of the charging pile data stored in the target storage area of the database storage node;
and the deleting module is used for deleting the second target charging pile data when the second target charging pile data with the aging time reaching the preset threshold value are detected.
In combination with the second aspect, embodiments of the present application provide a third possible implementation manner of the second aspect, where the apparatus further includes:
the device further comprises:
the judging module is used for judging whether the received charging pile data meet the current service identification;
the response module is used for responding to the received charging pile data when the received charging pile data meets the current service identification;
and the discarding module is used for discarding the charging pile data when the received charging pile data does not meet the current service identifier.
With reference to the third possible implementation manner of the second aspect, the present application provides a fourth possible implementation manner of the second aspect, where the apparatus further includes:
the judging module is further used for judging whether the charging pile data is stored in the database storage node;
the response module is further configured to respond to the received charging pile data when the charging pile data does not exist in the database storage node.
The application provides a large-scale charging pile data processing method and device for an electric vehicle charging network, a target storage area of charging pile data in a database storage node is located by selecting a key field and a distribution key field of the charging pile data, so that the charging pile data are stored, and meanwhile, the stored charging pile data are stored based on an attribute field of a two-dimensional table structure.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 shows a flowchart of a data processing method for a large-scale charging pile of an electric vehicle charging network according to an embodiment of the present application.
Fig. 2 shows a flowchart of another data processing method for a large-scale charging pile of an electric vehicle charging network according to an embodiment of the present application.
Fig. 3 shows a flowchart of another data processing method for a large-scale charging pile of an electric vehicle charging network according to an embodiment of the present application.
Fig. 4 shows a flow chart of a service sequence provided by an embodiment of the present application.
Fig. 5 shows a schematic structural diagram of a distributed service architecture provided in an embodiment of the present application.
Fig. 6 shows a schematic diagram of displaying a real-time TPS of a redis cluster according to an embodiment of the present application.
Fig. 7 shows a flowchart of a data processing device for a large-scale charging pile facing an electric vehicle charging network according to an embodiment of the present application.
Fig. 8 shows a schematic structural diagram of a computer device 40 provided in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
Along with electric automobile rapid development and popularization, electric automobile public charging stake coverage is wider and wider, and coverage density is also bigger and bigger. The electric automobile charging pile is a direct source for providing electric energy for the electric automobile, and the charging pile real-time data reflects information such as real-time state and real-time interaction of the charging pile, and is one of the most important information of data interaction and data processing. The national grid electric vehicle service company takes a charging and battery replacing service as a main operation service, proposes to construct an open, intelligent, interactive and efficient electric vehicle charging and battery replacing service network, and the processing of real-time data of a charging pile is a key step for completing the aim.
The real-time state data of the charging pile truly reflects the state of more than two hundred of charging piles in the charging process, and provides the most basic numerical information for data mining work such as charging pile fault early warning and charging pile aging analysis. Fill electric pile real-time status data have the frequency height, measure characteristics many. According to measurement and calculation, a charging network is connected to 100 ten thousand electric vehicle charging piles in the next 5 years, calculation is carried out according to the charging pile data uploading period being 10 seconds, and 10 ten thousand pieces of state data are generated in the charging network every second. Fill electric pile about 240 measurations, one fills electric pile state information and is about 1KB, fills the data bulk of electric pile real-time status data big, update frequency is high.
Meanwhile, the charging pile ordered flow information is important information for real-time interaction of charging operation, and the problems of network delay, network interruption and the like in an unreliable network environment can cause message loss, message disorder or message uploading for multiple times, and a reasonable method is needed for solving the problems. The real-time alarm data of the charging pile can also cause information disorder due to an unreliable network environment, and a reasonable method is also needed to solve the problem.
The embodiment of the application provides a large-scale charging pile data processing method for an electric vehicle charging network, charging pile data are stored by using a high-efficiency memory database redis, any field information of the charging pile data can be flexibly read and updated in a hash storage mode, a user does not need to perform conversion configuration, the operation complexity is low, and the flexibility is high.
Among them, redis an open source memory data storage, can be used as database, cache, etc., and is widely used in the internet industry. Because of its small size and high usability, it is often used to store user identification information, website context information, and the like. Has the following characteristics:
(1) the reading and writing speed is high. The official single-process redis performance indexes are as follows: write 8.1 ten thousand bars/second, read 11 ten thousand bars/second.
(2) And (5) lasting. The redis service writes memory data into a disk file periodically (at a second level) according to a configuration file, loads a disk archive file after the redis service is stopped or a fault is down, recovers information in the memory, and ensures that the service does not lose data in fault and normal start-stop states.
(3) A master-slave mechanism. The main-standby relation can be configured, the data of the standby service is synchronized with the main service in real time, and when the main service fails, the system automatically switches the main service and the standby service.
(4) Multiple data types are supported: string, list, set, hash, ordered set.
(5) Multiple programming languages are supported. And about 40 programming languages are supported, so that the use by users is convenient.
(6) Supporting both clustering functions and transaction processing functions.
The redis has more data types, supports more characteristics, and is more suitable for being used in a production environment with higher requirements on data safety.
As shown in fig. 1, a first embodiment of the present application provides a data processing method for a large-scale charging pile facing an electric vehicle charging network, which is applied to a server, and includes:
s101, receiving charging pile data sent by a charging pile; wherein, each fills electric pile data and all includes the attribute value that corresponds with a plurality of fields respectively.
In the embodiment of the application, it includes a plurality of fields to fill electric pile data, if fill electric pile serial number: 1110190000000025, respectively; the working state is as follows: working; charging type: charging by direct current; and (3) user identification: 8820190000012077, respectively; charging A phase voltage: 380 of the raw material; charging phase a current: 11. specifically, as shown in table 2.
Table 2 charging pile real-time status data information
The charge state information is shown in the table: including information such as charging pile number, operating condition, charging type. The data is a common two-dimensional table relational type data, comprises a plurality of fields, is suitable for being stored in a traditional relational database from the structural point of view, and needs to be stored in a memory database in order to adapt to real-time updating and reading of charging pile state data.
Because the real-time state data of the charging pile belong to the data of the two-dimensional table relationship, a specific storage mode needs to be designed by utilizing the redis memory database to store the data, so that the operations of storing, reading, updating and the like of the real-time state data of the charging pile are met.
The bottom layer of the redis in-memory database is to store key-value type key-value pairs, such as: < number of charging pile: 1110190000000025, charging type: and D, charging by direct current. The loose data organization relation is not easy to store and read, a two-dimensional relation table needs to be split into redis for storage by a design method, and meanwhile various actual operations such as updating of real-time state data of the charging pile are met. At present, three centralized storage modes, namely json character string storage, key value pair storage, hash type storage and the like, are mainly provided, and the storage modes and the advantages and the disadvantages thereof are respectively described below.
S102, determining a database storage node corresponding to the charging pile data according to a first attribute value corresponding to a preset key field.
In the embodiment of the application, the number of the charging pile is selected as the key field, and the index of the key field and the database storage node is established in advance. Here, an index is searched according to a first attribute value (e.g., 1110190000000025) corresponding to the charging pile number in the charging pile data, and a database storage node corresponding to the first attribute value is determined according to the index.
S103, carrying out hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to the charging pile data.
In the embodiment of the application, the hash value of the distribution key field and the index of the storage area in the database storage node are stored in the server in advance. Here, performing hash operation on the second attribute value corresponding to the distribution key field in the charging pile data to obtain a hash value corresponding to the charging pile data, so as to determine a target storage area corresponding to the charging pile data through the hash value.
And S104, determining a target storage area of the charging pile data in the database storage node according to the hash value corresponding to the charging pile data.
In the embodiment of the application, the hash value of the distribution key field and the index of the storage area in the database storage node are stored in the server in advance. Therefore, by calculating the hash value of the second attribute value corresponding to the distribution key field in the charging pile data, the target storage area corresponding to the charging pile data can be determined.
And S105, storing the charging pile data into the corresponding target storage area.
The application provides a large-scale charging pile data processing method for an electric vehicle charging network, a target storage area of charging pile data in a database storage node is located by selecting a key field and a distribution key field of the charging pile data, so that the charging pile data are stored, and meanwhile, the stored charging pile data are stored based on an attribute field of a two-dimensional table structure.
Further, as shown in fig. 2, in the data processing method for the large-scale charging pile facing the electric vehicle charging network provided by the embodiment of the application, the method includes:
s201, receiving DML operation which is sent by a user terminal and carries an operation field; wherein the DML operation further carries the first attribute value and the second attribute value.
The operation field can be any one or more types of field information of the charging pile data.
S202, carrying out hash operation on the second attribute value in the DML operation to obtain a hash value of the second attribute value.
In the embodiment of the application, the hash value of the distribution key field and the index of the storage area in the database storage node are stored in the server in advance. Therefore, by calculating the hash value of the second attribute value in the DML operation, the target storage area of the charging pile data in the database storage node can be determined.
S203, determining a target database storage node according to the first attribute.
In the embodiment of the application, the server stores the index of the key field and the database storage node in advance, so that the target database storage node can be determined by querying the index based on the first attribute value carried by the DML operation.
And S204, sending the DML operation carrying the hash value of the operation field and the second attribute value to the target database storage node, so that the target database storage node operates the first target charging pile data corresponding to the operation field in the target storage area corresponding to the second attribute value.
In the embodiment of the application, after receiving the DML operation, the target database storage node determines a target storage area according to the hash value of the second attribute value, and then executes the first target charging pile data corresponding to the operation field from the target storage area.
S205, receiving the operation result returned by the target database storage node, and forwarding the operation result to the user terminal.
Further, as shown in fig. 3, in the method for processing data of large-scale charging piles oriented to the electric vehicle charging network provided by the embodiment of the application, each piece of charging pile data stored in the target storage area of the database storage node corresponds to an aging time; step 105, after the charging pile data is stored in the corresponding target storage area, further comprising:
s301, monitoring the aging time of charging pile data stored in a target storage area of the database storage node.
S302, if the second target charging pile data with the aging time reaching the preset threshold value is detected, deleting the second target charging pile data.
Combining the steps 301 to 302, each piece of charging pile data stored in the database storage node corresponds to aging time, and when the aging time of any piece of charging pile data is reduced to 0, the charging pile data is deleted.
In the embodiment of the application, when new charging pile data are inserted into the redis database, the timeout time is set for the charging pile data. The purpose of this is: (1) the overtime is cleared without a large amount of useless information in long-term operation. (2) And the redis information is deleted after the overtime time, so that the charging pile and the user are prompted to restart a new charging process, and the charging pile and the user cannot be interfered by the disorder information for too long. The real-time ordered flow information processing of the charging pile is well represented in an actual system, the in-out of the ordered flow information is strictly controlled, and a reliable foundation is provided for the ordered flow business.
Further, in the data processing method for the large-scale charging pile facing the electric vehicle charging network provided by the embodiment of the application, in step 101, after the charging pile data sent by the charging pile is received, the method further includes: judging whether the received charging pile data meet the current service identification; if yes, responding to the received charging pile data; and if not, discarding the charging pile data.
In the embodiment of the application, the specific method for judging whether the received charging pile data meets the current service identifier includes: and judging whether the received charging pile data carries the service identification of the current service, and responding to the charging pile data in the manner from step 101 to step 105 if the received charging pile data carries the service identification of the current service. And if the received charging pile data does not carry the service identification of the current service, discarding the charging pile data.
Further, in the data processing method for the large-scale charging pile facing the electric vehicle charging network provided by the embodiment of the application, a plurality of servers are provided, and the method further includes:
and for any server, judging whether the charging pile data are stored in the database storage node or not after the server receives the charging pile data, and responding to the received charging pile data if the charging pile data are not stored in the database storage node.
In the embodiment of the application, a plurality of servers are deployed or a plurality of service instances are deployed on one server, and the plurality of servers or the plurality of service instances respectively and independently execute work; here, the work performed includes: and storing charging pile data or processing the DML operation of a user. After receiving the task, each service instance searches whether the current task is executed in the database node, and if yes, executes the task based on the method from step 101 to step 105 or from step 201 to step 205; and if not, discarding the received task.
In this application embodiment, electric automobile fills electric pile as the direct interface of the operation of charging, needs to handle multiple orderly flow work, including the user flow of charging of punching the card, and the code is swept to the two-dimensional code and the flow of charging, the charging billing model change etc.. In the charging pile information uploading process, due to network factors such as network delay, network interruption and network retransmission, the message is lost or is uploaded for multiple times, so that the normal processing sequence of the flow work is disturbed. As shown in fig. 4, the process of charging a user by swiping a card, starting charging from the user by swiping a card, includes: the method comprises the following processes of producing orders, freezing user accounts, starting charging, finishing charging, unfreezing merchants, deducting account, completing charging and the like. In the actual business processing process, due to network problems, information such as order generation and account deduction is sent for multiple times, so that multiple orders and multiple account deductions are generated by a user through one-time charging, or ordered flow information is lost, and the whole work processing fails.
A distributed service architecture is adopted in the data processing system, namely: the data processing services adopt a distributed architecture, each data processing service is stateless, a plurality of same services run simultaneously, and when a request needs to be processed, the data processing services can be forwarded to any service for processing. The design has the advantages that the service node can be dynamically expanded, and the service processing capacity can be simply and conveniently improved.
But the distributed service architecture brings inconvenience to the ordered information processing. Since the distributed service does not have a processing context (context), it cannot record the previously processed contents and processing steps, and when the ordered information is transmitted into the distributed data processing system in an out-of-order manner, the data is confused and the operation is repeated.
This disadvantage of distributed services can be handled correctly by a redis single-threaded processing mechanism.
As shown in fig. 5, the charging pile data processing service is a distributed service, and there are n service instances, and each service is a stateless process. The charging pile data processing service receives the ordered information processing requests, each piece of information is randomly sent to the service instance, and the service instance receiving the service requests processes the information. The distributed coordination of ordered information processing is completed by a Redis real-time database, and each request for accessing the Redis database is changed into serial processing due to a single-thread processing machine of the Redis database, so that the information sent out in an out-of-order mode is changed into an order mode.
The distributed service architecture solves the problem of ordered information processing, and a distributed coordination mechanism is also needed. In the embodiment of the application, a redis database is adopted for distributed coordination.
For example: the 4 message processing requests are sent in the order of 1, 2, 4 and 3 respectively. And taking the information 'charging pile mark-serial number' as a key and the information 'time' as a value, and sequentially processing each piece of information according to the serial number of the ordered flow. The information is processed and recorded according to the flow sequence, and the information which is not in the flow sequence is discarded.
The information 1 and the information 2 are processed, the mark of the information 2 is recorded in a redis database, then the information 4 request is sent, step3 information of the charging pile 001 does not exist in the redis database, and the information 4 is discarded if the time of arrival of the information 4 is judged to cause disorder. And then the information 3 comes, and the flow sequence is met, the information of the charging pile 001-step02 is deleted, and the content of the information 3 is written.
In the embodiment of the application, in actual engineering deployment, the electric vehicle charging pile real-time processing redis cluster is based on 18 virtual machines, a cpu of each virtual machine is an Intel Xeon E7-4830V2, a core is 4 cores, a core frequency bit is 2.20GHZ, a memory is 4GB, and a hard disk is 100 GB. The cluster work service adopts a one-master-two-standby architecture, the cluster is divided into 6 groups, and all data reading and writing work of the load-balancing cluster is balanced. The maximum stable TPS of the cluster is 25 ten thousand per second, and the storage capacity can reach 20 GB.
As shown in fig. 6, a real-time TPS of a redis cluster displays that the real-time data processing service of the electric vehicle charging pile adopts a redis official java driver package to interact with the redis cluster, and fig. 6 shows that the real-time data processing service accesses the redis cluster at a certain moment, the instantaneous processing exceeds 11 ten thousand per second, the load of the real-time data processing service and the redis cluster is lower than 30%, and the operation is stable. The electric automobile charging pile is currently connected to 5 thousands of charging piles, 643563 data in a redis cluster occupy 468.81MB of storage capacity, the storage accounts for 2.3% of the total capacity, and the charging pile real-time data processing service and the redis cluster can stably operate for a long time.
According to the embodiment of the application, the real-time state data of the electric automobile charging pile is stored by the memory database redis, the problems of large data storage capacity and high data updating frequency are solved, and aiming at the characteristics of the real-time state data, several storage formats are designed and respective defects are analyzed. And then, providing a real-time ordered flow information processing flow of the charging pile, and designing an ordered flow information coordination mechanism combined with a redis database.
The method provided by the embodiment of the application runs normally and efficiently on the electric vehicle networking platform, provides rich functions and processing convenience for real-time data processing, and provides a borrowable method for real-time data processing of the electric vehicle charging network.
As shown in fig. 7, a second embodiment of the present application provides an electric vehicle charging network-oriented large-scale charging pile data processing apparatus, where the apparatus includes:
the receiving module 11 is used for receiving charging pile data sent by a charging pile; each charging pile data comprises attribute values corresponding to the fields respectively;
the determining module 12 is configured to determine, according to a first attribute value corresponding to a preset key field, a database storage node corresponding to the charging pile data;
the hash operation module 13 is configured to perform hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to the charging pile data;
the determining module 12 is further configured to determine a target storage area of the charging pile data in the database storage node according to the hash value corresponding to the charging pile data;
and the storage module 14 is configured to store the charging pile data in the corresponding target storage area.
Further, the electric automobile charging network-oriented large-scale charging pile data processing device provided by the embodiment of the application further comprises:
the receiving module 11 is further configured to receive a DML operation which is sent by the user terminal and carries an operation field; wherein the DML operation further carries the first attribute value and the second attribute value;
the hash operation module 13 is further configured to perform hash operation on the second attribute value in the DML operation to obtain a hash value of the second attribute value;
the determining module 12 is further configured to determine a target database storage node according to the first attribute;
the sending module is further configured to send the DML operation carrying the hash value of the operation field and the second attribute value to the target database storage node, so that the target database storage node operates the first target charging pile data corresponding to the operation field in the target storage area corresponding to the second attribute value;
the receiving module 11 is further configured to receive an operation result returned by the target database storage node;
the sending module is further configured to forward the operation result to the user terminal.
Further, in the large-scale charging pile data processing device for the electric vehicle charging network provided by the embodiment of the application, each piece of charging pile data stored in the target storage area of the database storage node corresponds to an aging time; the device further comprises:
the monitoring module is used for monitoring the aging time of the charging pile data stored in the target storage area of the database storage node;
and the deleting module is used for deleting the second target charging pile data when the second target charging pile data with the aging time reaching the preset threshold value are detected.
Further, the electric automobile charging network-oriented large-scale charging pile data processing device provided by the embodiment of the application further comprises:
the judging module is used for judging whether the received charging pile data meet the current service identification;
the response module is used for responding to the received charging pile data when the received charging pile data meets the current service identification;
and the discarding module is used for discarding the charging pile data when the received charging pile data does not meet the current service identifier.
Further, in the large-scale charging pile data processing device for the electric vehicle charging network provided by the embodiment of the application, the judging module is further configured to judge whether the charging pile data is stored in the database storage node;
the response module is further configured to respond to the received charging pile data when the charging pile data does not exist in the database storage node.
The application provides a towards electric automobile charging network large-scale electric pile data processing apparatus that fills, through selecting key field and the distribution key field of filling electric pile data, the target storage area of electric pile data in database storage node is filled in the location, like this, realize filling electric pile data and save, simultaneously, the electric pile data that fills of storage is based on the attribute field of two-dimensional table structure and is saved, like this, when user terminal to the electric pile data execution data manipulation language DML operation of storage, can read in a flexible way and fill electric pile data's arbitrary field information and handle, need not the user and carry out the conversion configuration in advance, the operation complexity is low, the flexibility is high, be applicable to the scene of some attribute fields of frequent more renew.
Fig. 8 is a schematic structural diagram of a computer device 40 according to an embodiment of the present application, and as shown in fig. 5, the computer device is configured to execute the method for processing data of a large-scale charging pile for an electric vehicle charging network in fig. 2, where the device includes a memory 401, a processor 402, and a computer program stored in the memory 401 and capable of running on the processor 402, where the processor 402 implements the steps of the method for processing data of a large-scale charging pile for an electric vehicle charging network when executing the computer program.
Specifically, the memory 401 and the processor 402 can be general memories and processors, which are not limited in particular, and when the processor 402 runs a computer program stored in the memory 401, the data processing method for the large-scale charging pile facing the electric vehicle charging network can be executed.
Corresponding to the method for processing the large-scale charging pile data for the electric vehicle charging network in fig. 2, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for processing the large-scale charging pile data for the electric vehicle charging network are executed.
Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, and the like, and when a computer program on the storage medium is executed, the data processing method for the large-scale charging pile facing the electric vehicle charging network can be executed.
The large-scale charging pile data processing device for the electric automobile charging network can be specific hardware on equipment or software or firmware installed on the equipment. The device provided by the embodiment of the present application has the same implementation principle and technical effect as the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments where no part of the device embodiments is mentioned. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the foregoing systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the spirit and scope of the present disclosure, which should be construed in light of the above teachings. Are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (8)
1. A data processing method for a large-scale charging pile facing an electric vehicle charging network is applied to a server and is characterized by comprising the following steps:
receiving charging pile data sent by a charging pile; each charging pile data comprises attribute values corresponding to the fields respectively;
determining a database storage node corresponding to the charging pile data according to a first attribute value corresponding to a preset key field;
performing hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to the charging pile data;
determining a target storage area of the charging pile data in the database storage node according to the hash value corresponding to the charging pile data;
storing the charging pile data into the corresponding target storage area;
the method comprises the following steps:
receiving DML operation which is sent by a user terminal and carries an operation field; wherein the DML operation further carries the first attribute value and the second attribute value;
performing hash operation on the second attribute value in the DML operation to obtain a hash value of the second attribute value;
determining a target database storage node according to the first attribute;
sending the DML operation carrying the hash value of the operation field and the second attribute value to the target database storage node, so that the target database storage node operates the first target charging pile data corresponding to the operation field in a target storage area corresponding to the second attribute value;
and receiving an operation result returned by the storage node of the target database, and forwarding the operation result to the user terminal.
2. The large-scale charging pile data processing method oriented to the electric vehicle charging network is characterized in that each piece of charging pile data stored in the target storage area of the database storage node corresponds to an aging time; after the charging pile data are stored in the corresponding target storage areas, the method further comprises the following steps:
monitoring the aging time of charging pile data stored in a target storage area of a database storage node;
and if the second target charging pile data with the aging time reaching the preset threshold value is detected, deleting the second target charging pile data.
3. The large-scale charging pile data processing method oriented to the electric vehicle charging network of claim 1, wherein after receiving charging pile data sent by the charging pile, the method further comprises:
judging whether the received charging pile data meet the current service identification;
if yes, responding to the received charging pile data;
and if not, discarding the charging pile data.
4. The data processing method for the large-scale charging pile facing the electric vehicle charging network according to any one of claims 1 to 3, wherein the number of the servers is multiple, and the method further comprises the following steps:
and for any server, judging whether the charging pile data are stored in the database storage node or not after the server receives the charging pile data, and responding to the received charging pile data if the charging pile data are not stored in the database storage node.
5. The utility model provides a fill electric pile data processing device on a large scale towards electric automobile charging network which characterized in that includes:
the receiving module is used for receiving charging pile data sent by the charging pile; each charging pile data comprises attribute values corresponding to the fields respectively;
the determining module is used for determining a database storage node corresponding to the charging pile data according to a first attribute value corresponding to a preset key field;
the hash operation module is used for carrying out hash operation on a second attribute value corresponding to a preset distribution key field to obtain a hash value corresponding to the charging pile data;
the determining module is further configured to determine a target storage area of the charging pile data in the database storage node according to the hash value corresponding to the charging pile data;
the storage module is used for storing the charging pile data into the corresponding target storage area;
the device further comprises:
the receiving module is further configured to receive a DML operation which is sent by the user terminal and carries an operation field; wherein the DML operation further carries the first attribute value and the second attribute value;
the hash operation module is further configured to perform hash operation on the second attribute value in the DML operation to obtain a hash value of the second attribute value;
the determining module is further used for determining a target database storage node according to the first attribute;
the sending module is further configured to send the DML operation carrying the hash value of the operation field and the second attribute value to the target database storage node, so that the target database storage node operates the first target charging pile data corresponding to the operation field in the target storage area corresponding to the second attribute value;
the receiving module is further configured to receive an operation result returned by the target database storage node;
the sending module is further configured to forward the operation result to the user terminal.
6. The large-scale charging pile data processing device oriented to the electric vehicle charging network of claim 5, wherein each piece of charging pile data stored in the target storage area of the database storage node corresponds to an aging time; the device further comprises:
the monitoring module is used for monitoring the aging time of the charging pile data stored in the target storage area of the database storage node;
and the deleting module is used for deleting the second target charging pile data when the second target charging pile data with the aging time reaching the preset threshold value are detected.
7. The electric vehicle charging network-oriented large-scale charging pile data processing device according to claim 5, further comprising:
the judging module is used for judging whether the received charging pile data meet the current service identification;
the response module is used for responding to the received charging pile data when the received charging pile data meets the current service identification;
and the discarding module is used for discarding the charging pile data when the received charging pile data does not meet the current service identifier.
8. The data processing device of the large-scale charging pile oriented to the electric vehicle charging network is characterized in that,
the judging module is further used for judging whether the charging pile data is stored in the database storage node;
the response module is further configured to respond to the received charging pile data when the charging pile data does not exist in the database storage node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811284988.3A CN109471860B (en) | 2018-10-31 | 2018-10-31 | Large-scale charging pile data processing method and device for electric vehicle charging network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811284988.3A CN109471860B (en) | 2018-10-31 | 2018-10-31 | Large-scale charging pile data processing method and device for electric vehicle charging network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109471860A CN109471860A (en) | 2019-03-15 |
| CN109471860B true CN109471860B (en) | 2020-10-23 |
Family
ID=65666778
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811284988.3A Expired - Fee Related CN109471860B (en) | 2018-10-31 | 2018-10-31 | Large-scale charging pile data processing method and device for electric vehicle charging network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109471860B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110601240B (en) * | 2019-09-24 | 2021-08-03 | 广东电网有限责任公司 | Intelligent management and control method for power grid charging pile access |
| CN111488371A (en) * | 2020-04-07 | 2020-08-04 | 中国人民财产保险股份有限公司 | Data query method and device |
| CN113467917B (en) * | 2021-07-16 | 2024-03-29 | 中国电力科学研究院有限公司 | High-speed cache distributed storage method of power grid dispatching control system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391863A (en) * | 2014-10-23 | 2015-03-04 | 中国建设银行股份有限公司 | Data storage method and device |
-
2018
- 2018-10-31 CN CN201811284988.3A patent/CN109471860B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391863A (en) * | 2014-10-23 | 2015-03-04 | 中国建设银行股份有限公司 | Data storage method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109471860A (en) | 2019-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11645183B1 (en) | User interface for correlation of virtual machine information and storage information | |
| US11068441B2 (en) | Caseless file lookup in a distributed file system | |
| CN109471860B (en) | Large-scale charging pile data processing method and device for electric vehicle charging network | |
| WO2021091489A1 (en) | Method and apparatus for storing time series data, and server and storage medium thereof | |
| CN110347651A (en) | Method of data synchronization, device, equipment and storage medium based on cloud storage | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN103078945B (en) | To the method and system that browser crash data processes | |
| CN103501319A (en) | Low-delay distributed storage system for small files | |
| US20160140140A1 (en) | File classification in a distributed file system | |
| CN111061802A (en) | Power data management processing method and device and storage medium | |
| CN113515317A (en) | Data recovery method and device | |
| CN109189724B (en) | Method and device for improving audio and video data storage efficiency of video monitoring system | |
| CN110352410A (en) | Track the access module and preextraction index node of index node | |
| CN110609967B (en) | List generation method and device and storage medium | |
| CN104391947B (en) | Magnanimity GIS data real-time processing method and system | |
| CN101783814A (en) | Metadata storing method for mass storage system | |
| CN110704468A (en) | Data updating method and device and controller | |
| CN115470189A (en) | Data query method and device and readable storage medium | |
| CN111143294B (en) | Data retrieval method and device for power acquisition terminal and electronic equipment | |
| CN116628042A (en) | Data processing method, device, equipment and medium | |
| CN114138187A (en) | Method and device for managing timed snapshot volume of volume mapping copy service plug-in | |
| CN108021562A (en) | Deposit method, apparatus and distributed file system applied to distributed file system | |
| WO2019218677A1 (en) | Data storage method for power grid simulation analysis, device, and electronic apparatus | |
| CN103164431B (en) | The date storage method of relevant database and storage system | |
| CN105354310A (en) | MapReduce based map tile storage layout optimization method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20201023 Termination date: 20211031 |