CN111338696A - FPGA-based data stream field processing method and device - Google Patents

FPGA-based data stream field processing method and device Download PDF

Info

Publication number
CN111338696A
CN111338696A CN202010132645.6A CN202010132645A CN111338696A CN 111338696 A CN111338696 A CN 111338696A CN 202010132645 A CN202010132645 A CN 202010132645A CN 111338696 A CN111338696 A CN 111338696A
Authority
CN
China
Prior art keywords
field
data
bit
characteristic value
data stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010132645.6A
Other languages
Chinese (zh)
Other versions
CN111338696B (en
Inventor
王媛丽
刘钧锴
郝锐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202010132645.6A priority Critical patent/CN111338696B/en
Publication of CN111338696A publication Critical patent/CN111338696A/en
Application granted granted Critical
Publication of CN111338696B publication Critical patent/CN111338696B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/38Concurrent instruction execution, e.g. pipeline, look ahead
    • G06F9/3818Decoding for concurrent execution
    • G06F9/382Pipelined decoding, e.g. using predecoding
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00Digital computers in general; Data processing equipment in general
    • G06F15/76Architectures of general purpose stored program computers
    • G06F15/78Architectures of general purpose stored program computers comprising a single central processing unit
    • G06F15/7807System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/38Concurrent instruction execution, e.g. pipeline, look ahead
    • G06F9/3867Concurrent instruction execution, e.g. pipeline, look ahead using instruction pipelines

Abstract

The invention discloses a field processing method and a field processing device of a data stream based on an FPGA (field programmable gate array). A data characteristic value carrying field related information in the data stream is obtained; extracting field data in the data stream; judging the field type of the extracted field data according to the obtained data characteristic value; the field data is parsed based on the determined field type. The invention is based on the FPGA platform, positions and extracts the data field in the data stream through the data characteristic value, realizes the processing in a pipeline parallel mode as a whole, increases the throughput, reduces the processing delay, has high reusability, can flexibly change aiming at different data bus bit widths, field lengths and the like, and obtains higher efficiency in the application requiring low delay and high real-time property.

Description

FPGA-based data stream field processing method and device
Technical Field
The invention relates to the field of field processing, in particular to a field processing method and device based on an FPGA.
Background
With the rapid development of technologies such as big data, internet of things, mobile internet, cloud computing and the like, a large amount of data needs to be efficiently processed and analyzed in real time. Because analysis, processing and the like of data occupy a large amount of server resources, and meanwhile, with the failure of moore's law, the increase of the computing power of the CPU cannot meet the requirements of the services.
When the CPU performs parallel data processing, each computing unit processes different data packets, and all the computing units must perform the same processing at a uniform pace, that is, the parallel data packets must be simultaneously input and output. After the instruction is decoded, a plurality of execution units simultaneously access data in the memory by the CPU, a data packet required to be operated is obtained at one time, reading is carried out according to the characteristics of the fields, and the single-instruction data processing efficiency of the CPU is improved. The processing mode is to use software for processing, although single instruction multiple data streams are used for processing, the processing time delay in the effective data analysis and acquisition process is large, and the delay can reach millisecond level or even larger, which cannot be tolerated in the industry requiring low-delay measurement and monitoring, so that a method with high real-time performance and low-delay output is required for processing and analyzing data.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method and an apparatus for processing a data stream field based on an FPGA, which utilize an FPGA architecture to implement low-latency and high-throughput processing, and can flexibly change for different data bus bit widths and field lengths.
The technical scheme of the invention is as follows: a data stream field processing method based on FPGA includes the following steps:
acquiring a data characteristic value carrying field related information in a data stream;
extracting field data in the data stream;
judging the field type of the extracted field data according to the obtained data characteristic value;
the field data is parsed based on the determined field type.
Further, field data in the data stream is extracted based on the byte stop bit.
Further, the extracting field data in the data stream according to the byte stop bit specifically includes:
judging whether the highest bit of the byte has a set stop bit;
if the field is set, the byte is the last byte of the field;
if not, the next byte belongs to the field, and whether the highest bit of the next byte is set is continuously judged, and by analogy, the field data of all the fields in the data stream are sequentially extracted.
Furthermore, the data characteristic value comprises n bits, wherein the bit is 1 to indicate validity, and 0 to indicate invalidity;
judging the field type of the extracted field data according to the obtained data characteristic value, specifically:
and judging which bit of the data characteristic value corresponds to the field data, and determining the field type of the field data according to the corresponding bit.
Further, the determination of which bit of the field data corresponds to the data feature value specifically includes:
obtaining a field label and a field significant quantity value based on bit of the data characteristic value; the field label is used for labeling the bit of the data characteristic value from 0 to n-1 from high bit to low bit in sequence; the field effective number value is the sum of the number of effective fields contained in the bit corresponding to each bit of the data characteristic value and all the previous bits;
and calculating the sum of the stop bit numbers contained in the current byte and all the previous bytes received in the data stream, searching the field number value equal to the value, corresponding to the field label corresponding to the effective data characteristic value bit, and judging the corresponding data characteristic value bit based on the field label.
Further, the field type includes an integer type and a character type.
The technical scheme of the invention also comprises a data stream field processing device based on the FPGA, which comprises,
a data characteristic value acquisition module: acquiring a data characteristic value carrying field related information in a data stream;
a field data extraction module: extracting field data in the data stream;
a field type judging module: judging the field type of the extracted field data according to the obtained data characteristic value;
an analysis module: the field data is parsed based on the determined field type.
Further, the field data extraction module extracts field data in the data stream according to the byte stop bit.
Furthermore, the data characteristic value comprises n bits, wherein the bit is 1 to indicate validity, and 0 to indicate invalidity;
the field type judging module comprises a field type judging module,
a field label setting unit: setting field labels based on bit bits of the data characteristic values; the field labels are labels for bit bits of the data characteristic value from 0 to n-1 from high bit to low bit in sequence;
a field significant quantity value acquisition unit: obtaining a field significant quantity value based on bit bits of the data characteristic value; the field effective number value is the sum of the number of effective fields contained in the bit corresponding to each bit of the data characteristic value and all the previous bits;
a character type determination unit: calculating the sum of the stop bit numbers contained in the current byte and all the previous bytes received in the data stream, searching a field number value equal to the value, corresponding to a field label corresponding to the valid data characteristic value bit, and judging the corresponding data characteristic value bit based on the field label; the field type of the field data can be determined by obtaining the corresponding bit.
The FPGA-based data stream field processing method and device provided by the invention are based on the FPGA platform, and the data fields in the data stream are positioned and extracted through the data characteristic values, so that the processing in a pipeline parallel mode is integrally realized, the throughput is increased, and the processing delay is also reduced. Compared with software processing, the method provided by the invention has high reusability, can flexibly change bit width, field length and the like of different data buses, and has higher efficiency in the application requiring low delay and high real-time property.
Drawings
FIG. 1 is a schematic flow chart of a method according to an embodiment of the present invention.
Fig. 2 is a diagram illustrating field extraction in a data stream according to an embodiment of the invention.
FIG. 3 is a table illustrating field labels and field significand values according to an embodiment of the present invention.
Fig. 4 is a diagram illustrating the calculation of the number of stop bits for one message body according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings by way of specific examples, which are illustrative of the present invention and are not limited to the following embodiments.
Example one
The embodiment provides a data stream field processing method based on an FPGA, which processes an instruction data stream field based on an FPGA architecture, exerts high pipeline parallel processing capability of the FPGA, and can process data with low delay and high throughput. In the specific processing process, the data fields in the data stream are positioned and extracted through the data characteristic values, the field types are determined, the field data are analyzed and processed according to the field types, the reusability is high, and the data processing method can flexibly change aiming at different data bus bit widths, field lengths and the like.
As shown in fig. 1, the method comprises the following steps:
s1, acquiring a data characteristic value carrying field related information in the data stream;
s2, extracting field data in the data stream;
s3, judging the field type of the extracted field data according to the obtained data characteristic value;
s4, the field data is parsed based on the determined field type.
(1) Data stream field data extraction
Since the data is encoded in stop bits, the most significant bit of each byte indicates whether the next byte is part of a valid field. If the most significant bit of the byte is not set, the next byte data belongs to the field data, otherwise, the last byte data of the field.
Therefore, as shown in fig. 2, the stop bit (highest bit) of each byte in the data stream is used as a flag signal for data position extraction, i.e., the above step S2 extracts field data in the data stream according to the byte stop bit.
The specific implementation steps are as follows:
judging whether the highest bit of the byte has a set stop bit;
if the field is set, the byte is the last byte of the field;
if not, the next byte belongs to the field, and whether the highest bit of the next byte is set is continuously judged, and by analogy, the field data of all the fields in the data stream are sequentially extracted.
(2) Field type determination
It should be noted that a data characteristic value is set at the front end of byte data of a data stream, where the data characteristic value is an identifier including n bits and carries related information of field data, for example, a message body includes several fields, and the type of each field. A bit of 1 indicates valid, 0 indicates invalid, and a bit of 1 indicates a field data. As shown in the data characteristic value in fig. 3, there are 12 bits, including 7 bits of 1, which indicates that there are 7 fields in the message body, and the bit corresponding to each field indicates the type of the field, which can be determined according to the instruction protocol.
Therefore, in this embodiment, the step S3 determines the field type of the extracted field data according to the obtained data feature value, specifically, which bit of the data feature value the field data corresponds to, and determines the field type of the field data according to the corresponding bit.
Judging which bit of the data characteristic value corresponds to the field data, and the specific implementation process is as follows:
(a) the field index and the field significand value are obtained based on the bits of the data feature value.
As shown in fig. 3, the field labels are labels for the bits of the data characteristic value from 0 to n-1 from the high order to the low order; n is the total number of bits of the data characteristic value, n in the figure is 12, the label of the field corresponding to the highest bit is 0, and the label of the field corresponding to the lowest bit is 11;
the field effective number value is the sum of the number of effective fields contained in the bit corresponding to each bit of the data characteristic value and all the previous bits; if the first bit is 1, the corresponding field significant quantity value is 1, the second bit is 1, the corresponding field significant quantity value is the sum 2 of the first two bit significant quantities, the third bit is 0, the sum of the corresponding first three bit significant quantities is still 2, and so on.
(b) And calculating the sum of the stop bit numbers contained in the current byte and all the previous bytes received in the data stream, searching the field number value equal to the value, corresponding to the field label corresponding to the effective data characteristic value bit, and judging the corresponding data characteristic value bit based on the field label.
As shown in fig. 4, the sum of the data amount of the stop bit in each byte data of each message body is calculated, that is, the byte high bits of the data in one message body are accumulated from front to back, and the sum of the number of the stop bit in each current byte and the number of the stop bit in the previous data is calculated, so that when the bit of the field in the data characteristic value is valid and the value of the field valid number in the corresponding position is equal to the stop bit value calculated in one message body, which field is the currently extracted field can be determined.
For example, when the stop bit accumulation value in fig. 4 is 2, that is, the second field data is received, the stop bit accumulation value 2 corresponds to the field significant value 2 of the second column and the third column in fig. 3, but the bit of the data feature value of the second column is 1 significant, and the bit of the data feature value of the third column is 0 insignificant, the second field data corresponds to the field label with the label 1, which indicates the second bit of the data feature value corresponding to the field data, and the field type can be determined according to the command protocol by knowing that the second bit corresponds to the bit.
The field types of the data stream include integer and character types.
(3) Decoded output
And extracting field data, and determining the type of the field data, namely analyzing the field data according to the type of the field data, wherein different types correspond to different analysis modes.
Example two
The embodiment provides a data stream field processing apparatus based on FPGA, which includes:
(1) a data characteristic value acquisition module: acquiring a data characteristic value carrying field related information in a data stream;
a data characteristic value is set at the front end of byte data of a data stream, the data characteristic value is an identifier containing n bits and carries relevant information of field data, for example, a message body contains several fields and the type of each field. A bit of 1 indicates valid, 0 indicates invalid, and a bit of 1 indicates a field data. As shown in the data characteristic value in fig. 3, there are 12 bits, including 7 bits of 1, which indicates that there are 7 fields in the message body, and the bit corresponding to each field indicates the type of the field, which can be determined according to the instruction protocol.
(2) A field data extraction module: extracting field data in the data stream;
since the data is encoded in stop bits, the most significant bit of each byte indicates whether the next byte is part of a valid field. If the most significant bit of the byte is not set, the next byte data belongs to the field data, otherwise, the last byte data of the field.
Therefore, the stop bit (most significant bit) of each byte in the data stream is used as a flag signal for data position extraction, i.e., the module extracts field data in the data stream according to the byte stop bit.
(3) A field type judging module: judging the field type of the extracted field data according to the obtained data characteristic value;
the module can judge which bit of the data characteristic value corresponds to the field data, and the field type of the field data is determined according to the corresponding bit.
Calculating according to the data characteristic value to obtain a field label and a field effective numerical quantity in the data; calculating the sum of the data quantity of stop bits in each byte of data in each message body; when the bit of the field bit in the data characteristic value is valid and the value of the field significant number at the corresponding position is equal to the stop bit value calculated in a message body, the type of the currently extracted valid field is determined.
Specifically, the field type judging module includes,
a field label setting unit: setting field labels based on bit bits of the data characteristic values; the field labels are labels for bit bits of the data characteristic value from 0 to n-1 from high bit to low bit in sequence; n refers to the total number of bits of the data characteristic value;
a field significant quantity value acquisition unit: obtaining a field significant quantity value based on bit bits of the data characteristic value; the field effective number value is the sum of the number of effective fields contained in the bit corresponding to each bit of the data characteristic value and all the previous bits;
a character type determination unit: calculating the sum of the stop bit numbers contained in the current byte and all the previous bytes received in the data stream, searching a field number value equal to the value, corresponding to a field label corresponding to the valid data characteristic value bit, and judging the corresponding data characteristic value bit based on the field label; the field type of the field data can be determined by obtaining the corresponding bit.
(4) An analysis module: analyzing the field data based on the judged field type;
and extracting field data, and determining the type of the field data, namely analyzing the field data according to the type of the field data, wherein different types correspond to different analysis modes.
It should be noted that, when the defined data bus width is 64 bits, that is, the port data is 64 bits of data bit width, the maximum support can be made for 8 groups of shortest field extraction and positioning processing. So that 8 sets of decoding modules are used internally for processing data containing a maximum of 8 stop bits in one cycle.
The above disclosure is only for the preferred embodiments of the present invention, but the present invention is not limited thereto, and any non-inventive changes that can be made by those skilled in the art and several modifications and amendments made without departing from the principle of the present invention shall fall within the protection scope of the present invention.

Claims (9)

1. A data stream field processing method based on FPGA is characterized by comprising the following steps:
acquiring a data characteristic value carrying field related information in a data stream;
extracting field data in the data stream;
judging the field type of the extracted field data according to the obtained data characteristic value;
the field data is parsed based on the determined field type.
2. The FPGA-based data stream field processing method of claim 1, wherein field data in the data stream is extracted based on byte stop bits.
3. The FPGA-based data stream field processing method of claim 2, wherein the extracting of field data in the data stream according to the byte stop bits specifically comprises:
judging whether the highest bit of the byte has a set stop bit;
if the field is set, the byte is the last byte of the field;
if not, the next byte belongs to the field, and whether the highest bit of the next byte is set is continuously judged, and by analogy, the field data of all the fields in the data stream are sequentially extracted.
4. The FPGA-based data stream field processing method of claim 1, 2 or 3, wherein the data characteristic value comprises n bits, a bit of 1 indicates valid, and a 0 indicates invalid;
judging the field type of the extracted field data according to the obtained data characteristic value, specifically:
and judging which bit of the data characteristic value corresponds to the field data, and determining the field type of the field data according to the corresponding bit.
5. The FPGA-based data stream field processing method of claim 4, wherein the determining which bit of the field data corresponds to the data feature value specifically comprises:
obtaining a field label and a field significant quantity value based on bit of the data characteristic value; the field label is used for labeling the bit of the data characteristic value from 0 to n-1 from high bit to low bit in sequence; the field effective number value is the sum of the number of effective fields contained in the bit corresponding to each bit of the data characteristic value and all the previous bits;
and calculating the sum of the stop bit numbers contained in the current byte and all the previous bytes received in the data stream, searching the field number value equal to the value, corresponding to the field label corresponding to the effective data characteristic value bit, and judging the corresponding data characteristic value bit based on the field label.
6. The FPGA-based data stream field processing method of claim 1, 2, 3 or 5, wherein the field types include integer and character types.
7. A data stream field processing device based on FPGA is characterized in that the device comprises,
a data characteristic value acquisition module: acquiring a data characteristic value carrying field related information in a data stream;
a field data extraction module: extracting field data in the data stream;
a field type judging module: judging the field type of the extracted field data according to the obtained data characteristic value;
an analysis module: the field data is parsed based on the determined field type.
8. The field processing apparatus of claim 7, wherein the field data extraction module extracts the field data in the data stream according to the byte stop bit.
9. The FPGA-based data stream field processing apparatus of claim 7 or 8 wherein the data characteristic values comprise n bits, a bit of 1 indicating valid and a bit of 0 indicating invalid;
the field type judging module comprises a field type judging module,
a field label setting unit: setting field labels based on bit bits of the data characteristic values; the field labels are labels for bit bits of the data characteristic value from 0 to n-1 from high bit to low bit in sequence;
a field significant quantity value acquisition unit: obtaining a field significant quantity value based on bit bits of the data characteristic value;
the field effective number value is the sum of the number of effective fields contained in the bit corresponding to each bit of the data characteristic value and all the previous bits;
a character type determination unit: calculating the sum of the stop bit numbers contained in the current byte and all the previous bytes received in the data stream, searching a field number value equal to the value, corresponding to a field label corresponding to the valid data characteristic value bit, and judging the corresponding data characteristic value bit based on the field label; the field type of the field data can be determined by obtaining the corresponding bit.
CN202010132645.6A 2020-02-29 2020-02-29 FPGA-based data stream field processing method and device Active CN111338696B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010132645.6A CN111338696B (en) 2020-02-29 2020-02-29 FPGA-based data stream field processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010132645.6A CN111338696B (en) 2020-02-29 2020-02-29 FPGA-based data stream field processing method and device

Publications (2)

Publication Number Publication Date
CN111338696A true CN111338696A (en) 2020-06-26
CN111338696B CN111338696B (en) 2022-11-25

Family

ID=71183910

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010132645.6A Active CN111338696B (en) 2020-02-29 2020-02-29 FPGA-based data stream field processing method and device

Country Status (1)

Country Link
CN (1) CN111338696B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112073365A (en) * 2020-07-22 2020-12-11 深圳华云信息系统有限公司 Response message processing method and device based on FPGA

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070101241A1 (en) * 2005-10-17 2007-05-03 Enigma Semiconductor, Inc. 64b/66b Coding apparatus and method
CN105323192A (en) * 2014-06-19 2016-02-10 凯为公司 Method of modifying packets to a generic format for enabling programmable modifications and apparatus thereof

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070101241A1 (en) * 2005-10-17 2007-05-03 Enigma Semiconductor, Inc. 64b/66b Coding apparatus and method
CN105323192A (en) * 2014-06-19 2016-02-10 凯为公司 Method of modifying packets to a generic format for enabling programmable modifications and apparatus thereof

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
郭梓?等: "船舶通信报文实时动态按位解析方法", 《高技术通讯》 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112073365A (en) * 2020-07-22 2020-12-11 深圳华云信息系统有限公司 Response message processing method and device based on FPGA
CN112073365B (en) * 2020-07-22 2023-02-03 深圳华云信息系统有限公司 Response message processing method and device based on FPGA

Also Published As

Publication number Publication date
CN111338696B (en) 2022-11-25

Similar Documents

Publication Publication Date Title
US6742107B2 (en) Dynamically configured processing of composite stream input data using next conversion determining state transition table searched by converted input data
CN112104664B (en) Protocol conversion method, device and equipment for data of Internet of things
CN111338696B (en) FPGA-based data stream field processing method and device
CN106407201A (en) Data processing method and apparatus
CN104978526A (en) Virus signature extraction method and apparatus
CN112347020A (en) FAST market analysis system and method based on CGRA
CN113742298A (en) Airborne binary file general parallel analysis method and device and electronic equipment
CN110769002A (en) LabVIEW-based message analysis method, system, electronic device and medium
CN115269006A (en) Machine code instruction conversion method and device, electronic equipment and readable storage medium
CN112506910A (en) Multi-source data acquisition method and device, electronic equipment and storage medium
CN102567254B (en) The method that adopts dma controller to carry out data normalization processing
CN102541738B (en) Method for accelerating soft error resistance test of multi-core CPUs (central processing units)
CN110737678B (en) Data searching method, device, equipment and storage medium
CN116634042A (en) Big data message analysis method, device and storage medium
EP2654261B1 (en) Circuit for the fast analysis of packet headers transferred via a data bus
CN111782479A (en) Log processing method and device, electronic equipment and computer readable storage medium
CN112187935A (en) Information identification method and read-only memory
CN107994970B (en) General data decoding method based on ARINC429 bus communication
CN115834738B (en) Industrial control business behavior recognition method and device, electronic equipment and readable medium
CN1127018C (en) Method and apparatus for distributing commands to a plurality of circuit blocks
US11909801B1 (en) Ultra-low latency parsing system and method
CN112866180B (en) Data processing circuit, apparatus and method
US20080147888A1 (en) Address handling
CN111526120B (en) Method for generating TCP checksum by multistage pipeline circuit
CN113076178B (en) Message storage method, device and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant