CN116257771A

CN116257771A - Data acquisition equipment, data processing device and related method

Info

Publication number: CN116257771A
Application number: CN202111487751.7A
Authority: CN
Inventors: 胡新宇; 彭斌
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-12-07
Filing date: 2021-12-07
Publication date: 2023-06-13

Abstract

The application discloses data acquisition equipment is applied to the technical field of artificial intelligence. The data acquisition device includes: a feature extraction circuit and a data transmission circuit; the feature extraction circuit is used for acquiring environment data and carrying out feature extraction processing on the environment data based on the neural network model so as to obtain feature data, and is an analog circuit; the data transmission circuit is used for converting the characteristic data into a digital signal and transmitting the digital signal to the data processing device. The data processing device is arranged outside the data acquisition equipment and is used for processing the digital signals to obtain data processing results. In the scheme, the characteristic extraction of the environment data is realized by adopting the analog circuit with lower power consumption, so that the power consumption in the data processing process can be reduced; and only the feature extraction circuit is integrated in the data acquisition equipment, so that the data acquisition equipment is suitable for most algorithms, and the applicability of the data acquisition equipment is improved.

Description

Data acquisition equipment, data processing device and related method

Technical Field

The present disclosure relates to the field of artificial intelligence technologies, and in particular, to a data acquisition device, a data processing apparatus, and a related method.

Background

Artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar manner to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

Autopilot is a mainstream application in the field of artificial intelligence, and autopilot technology relies on cooperation of computer vision, radar, a monitoring device, a global positioning system and the like, so that an automotive vehicle can realize autopilot without active operation of human beings. Autonomous vehicles use various computing systems to assist in transporting passengers from one location to another. Some autonomous vehicles may require some initial input or continuous input from an operator (such as a pilot, driver, or passenger). The autonomous vehicle permits the operator to switch from a manual mode of operation to an autonomous mode or a mode in between. Because the automatic driving technology does not need human to drive the motor vehicle, the automatic driving technology can effectively avoid driving errors of human in theory, reduce traffic accidents and improve the transportation efficiency of the highway. Thus, autopilot technology is becoming more and more important.

At present, an automatic driving vehicle mainly relies on environment acquisition equipment such as cameras, radars and the like which are arranged on the vehicle to acquire environment information, and an operation chip in the automatic driving vehicle analyzes the environment information to determine a driving strategy of the automatic driving vehicle. As the number of environmental collection devices used by the automatic driving vehicle increases, and the accuracy of the environmental collection devices increases, the power demand of the automatic driving vehicle increases, resulting in an increase in energy consumption of the operation chip in the automatic driving vehicle. The energy consumption of the operation chip in the automatic driving vehicle is increased, so that the problems of overhigh power consumption, overhigh heating value and the like of the operation chip can be caused, and the normal running of the automatic driving vehicle is easily influenced.

Therefore, how to reduce the power consumption of the operation chip in the automatic driving vehicle is a problem to be solved.

Disclosure of Invention

A first aspect of the present application provides a data acquisition device comprising: the feature extraction circuit and the data transmission circuit. The characteristic extraction circuit is used for acquiring environment data and carrying out characteristic extraction processing on the environment data based on the neural network model so as to obtain characteristic data, the characteristic extraction circuit is an analog circuit, and the characteristic data is an analog signal. The feature data may be, for example, image features, and may be used to characterize feature information in an environmental image; alternatively, the feature data may be, for example, a point cloud data feature, and can be used to characterize feature information in the point cloud data. The feature extraction circuit performs feature extraction processing on the environmental image or the point cloud data based on the neural network model, so that feature data containing key information can be extracted, and the feature data can be used for a data processing device to execute subsequent data processing.

The data transmission circuit is used for converting the characteristic data into a digital signal and transmitting the digital signal to the data processing device. In the case that the feature data extracted by the feature extraction circuit is an analog signal, the analog-to-digital converter in the data transmission circuit may convert the feature data in the form of the analog signal into a digital signal so as to transmit the feature data to the data processing apparatus.

The data processing device is arranged outside the data acquisition equipment, namely the data processing device is separated from the data acquisition equipment. And the data processing device is used for processing the digital signals to obtain data processing results.

In the scheme, the characteristic extraction of the environment data is realized by adopting the analog circuit with lower power consumption, so that the power consumption in the data processing process can be reduced. And, through carrying out characteristic extraction to the environment data by the data acquisition equipment and then sending the characteristic data to the data processing device, the data volume of the transmission data can be reduced, thereby reducing the energy consumption in the data processing process and improving the data processing efficiency.

In addition, the characteristic extraction circuit in the data acquisition equipment is used for carrying out characteristic extraction processing on the acquired raw data, and the extracted characteristic data is sent to the data processing device, so that the whole data processing process is carried out on the basis of the acquired raw data. Compared with the prior art that the collected original data is compressed and then transmitted to the central side for processing, the scheme does not lose detail information, the processing capacity of a tiny target is enhanced, and the accuracy of data processing is higher.

Since the feature extraction process in the algorithm for processing the environmental data is generally unchanged or less changed, the process of processing the feature data is generally more changed. Therefore, the characteristic extraction circuit responsible for the characteristic extraction processing process is integrated in the data acquisition equipment in the scheme, so that the data acquisition equipment is suitable for most algorithms and is not influenced by algorithm updating, the applicability of the data acquisition equipment is improved, and frequent updating of software or hardware of the data acquisition equipment is avoided.

In a possible implementation, the data processing device is specifically configured to process the feature data based on a digital circuit. Digital circuits are capable of performing a wider variety of types and more complex arithmetic operations, such as exponentiations, than analog circuits; moreover, the applicability of the digital circuit is strong, and the same digital circuit can be used for executing different algorithms.

In the scheme, the digital circuit can support more kinds and more complex arithmetic operation, and the applicability of the digital circuit is strong, so that the processing of the characteristic data is realized based on the digital circuit, and the characteristic data processing requirements of different algorithms can be met, so that the data acquisition equipment and the data processing device can support different algorithms, and the flexibility of supporting different algorithms is ensured.

In one possible implementation, the data acquisition device and the data processing apparatus are deployed at different locations in the vehicle, respectively. For example, the data acquisition device is disposed outside the vehicle, and the data processing apparatus is disposed inside the vehicle.

In one possible implementation, the data acquisition device is a camera and the environmental data is image data; or, the data acquisition equipment is a laser radar or millimeter wave radar, and the environmental data is point cloud data. The point cloud data refers to a set of vectors in a three-dimensional coordinate system, that is, a set of sampling points with space coordinates acquired by a radar device, and can represent an obstacle in a three-dimensional space.

In one possible implementation, the characteristic data is used for the data processing apparatus to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

In one possible implementation manner, the feature extraction circuit comprises a convolution operation circuit and a residual error structure circuit; that is, the feature extraction circuit is used to implement a feature extraction network that includes convolution operations and residual structures.

The convolution operation circuit is used for executing multi-layer convolution operation on the environment data to obtain the characteristic data; the residual structure circuit is connected with the convolution operation circuit and is used for storing output data of any layer of convolution operation executed by the convolution operation circuit so that the output data is added with output data of a target layer of convolution operation to obtain input data of a next layer of convolution operation of the target layer of convolution operation. The convolution operation layer corresponding to the output data stored by the residual structure circuit is positioned before the convolution operation of the target layer. For example, the residual structure circuit stores the output data of the layer 1 convolution operation such that the output data of the layer 1 convolution operation is added to the output data of the layer 20 convolution operation (i.e., the target layer convolution operation) to obtain the input data of the layer 21 convolution operation.

In the scheme, the residual structure is introduced into the feature extraction network realized by the feature extraction circuit, and the residual structure in the feature extraction network is realized by the residual structure circuit, so that the phenomenon that the feature extraction network is degraded when the convolution operation with more layers is performed on the feature extraction network can be avoided, and the capability of the feature extraction network for extracting features is improved.

In one possible implementation, the residual structure circuit includes a capacitor, an input switch, and an output switch; the capacitor is used for storing an input signal when the input switch is closed and the output switch is opened; alternatively, the capacitor is configured to output the stored signal when the input switch is turned off and the output switch is turned on.

In the scheme, the residual structure circuit is realized by adopting the analog circuit, so that the complex process of storing data into a memory after analog-to-digital conversion is performed and outputting the data in a digital-to-analog conversion mode is avoided, and the power consumption required by the analog-to-digital conversion and the digital-to-analog conversion is saved. That is, the residual structure circuit implemented based on the analog circuit can effectively save power consumption as compared with the residual structure circuit implemented using the digital circuit.

In one possible implementation, the feature extraction circuit further includes an activation function circuit; the activation function circuit is connected with the convolution operation circuit and is used for carrying out activation function processing on output data of one or more layers of convolution operations executed by the convolution operation circuit.

In the scheme, the activation function circuit is used for carrying out activation function processing on the output data of convolution operation, and nonlinear factors can be introduced into the characteristic extraction processing process, so that the expression capacity of the extracted characteristic data is improved.

In one possible implementation, the activation function circuit includes a gate and a comparator; the comparator is used for comparing an input signal of the activation function circuit with a zero value signal and sending a first signal to the gating device when the input signal is greater than or equal to the zero value signal or sending a second signal to the gating device when the input signal is less than the zero value signal; the gate is configured to output the input signal when the first signal is received or output the zero value signal when the second signal is received.

In one possible implementation, the activation function circuit includes a gate, a comparator, and an attenuation circuit connected to the gate; the comparator is used for comparing an input signal of the activation function circuit with a zero value signal and sending a first signal to the gating device when the input signal is greater than or equal to the zero value signal or sending a second signal to the gating device when the input signal is less than the zero value signal; the attenuation circuit is used for carrying out attenuation treatment on the input circuit to obtain an attenuated input signal; the gate is configured to output the input signal when the first signal is received or output the attenuated input signal when the second signal is received.

In one possible implementation, the data acquisition device further comprises a sensor for acquiring the environmental data.

A second aspect of the present application provides a data processing apparatus comprising: the data receiving circuit is used for receiving characteristic data sent by the data acquisition equipment, the characteristic data are obtained by extracting characteristics of environment data through a neural network model by an analog circuit in the data acquisition equipment, and the data processing device is deployed outside the data acquisition equipment; and the data processing circuit is used for processing the characteristic data to obtain a data processing result.

In the scheme, the characteristic extraction of the environment data is realized by adopting the analog circuit with lower power consumption, so that the power consumption in the data processing process can be reduced. And the data acquisition equipment performs characteristic extraction on the environment data and then sends the characteristic data to the data processing device, and the data processing device realizes the subsequent processing of the characteristic data, so that the data quantity of the transmission data can be reduced, the energy consumption in the data processing process is reduced, and the data processing efficiency is improved.

In a possible implementation, the data processing circuit is configured to perform one or more of the following tasks based on the feature data: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

In one possible implementation, the data processing circuit is a digital circuit.

A third aspect of the present application provides a data processing method, where the method is applied to a data acquisition device, and the method includes: performing feature extraction processing on the environmental data through a neural network model to obtain feature data, wherein the feature extraction processing on the environmental data is realized on the basis of an analog circuit, and the feature data is an analog signal; converting the characteristic data into a digital signal and transmitting the digital signal to a data processing device; the data processing device is deployed outside the data acquisition equipment and is used for processing the digital signals to obtain data processing results.

In a possible implementation, the data processing device is specifically configured to process the feature data based on a digital circuit.

In one possible implementation, the data acquisition device and the data processing apparatus are deployed at different locations in the vehicle, respectively.

In one possible implementation, the data acquisition device is a camera and the environmental data is image data; or, the data acquisition equipment is a laser radar or millimeter wave radar, and the environmental data is point cloud data.

In one possible implementation manner, the feature extraction processing on the environmental data includes: performing a multi-layer convolution operation on the environmental data to obtain the feature data; and in the process of executing the multi-layer convolution operation on the environment data, storing the output data of any one layer of convolution operation so as to enable the output data to be added with the output data of the target layer of convolution operation, and obtaining the input data of the next layer of convolution operation of the target layer of convolution operation.

In one possible implementation manner, the feature extraction processing is performed on the environmental data, and the method further includes: and in the process of performing multi-layer convolution operation on the environment data, performing activation function processing on the output data of one or more layers of convolution operation.

A fourth aspect of the present application provides a data processing method, the method being applied to a data processing apparatus, the method comprising: the method comprises the steps of receiving characteristic data sent by data acquisition equipment, wherein the characteristic data are obtained by extracting characteristics of environment data through a neural network model by an analog circuit in the data acquisition equipment, and the data processing device is deployed outside the data acquisition equipment; and processing the characteristic data to obtain a data processing result.

In one possible implementation, the method is applied to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

In one possible implementation manner, the processing the feature data to obtain a data processing result includes: and processing the characteristic data based on the digital circuit to obtain a data processing result.

A fifth aspect of the present application provides a data processing apparatus comprising: a memory and a processor; the memory stores code, the processor being configured to execute the code, the data processing apparatus performing the method as in any one of the implementations of the third aspect when the code is executed.

A sixth aspect of the present application provides a data processing apparatus comprising: a memory and a processor; the memory stores code, the processor being configured to execute the code, the data processing apparatus performing the method as in any one of the implementations of the fourth aspect when the code is executed.

A seventh aspect of the present application provides a vehicle comprising a data acquisition device as in any one of the implementations of the first aspect.

In a possible implementation manner, the vehicle further includes a data processing device as in any one implementation manner of the second aspect.

An eighth aspect of the present application provides a video surveillance system comprising a data acquisition device as in any one of the implementations of the first aspect and a data processing apparatus as in any one of the implementations of the second aspect.

In one possible implementation, the data processing device is deployed in a server in the cloud.

A ninth aspect of the present application provides a computer readable storage medium having a computer program stored therein, which when run on a computer causes the computer to perform the method as any one of the implementations of the third to fourth aspects.

A tenth aspect of the present application provides a computer program product which, when run on a computer, causes the computer to perform the method as in any one of the implementations of the third to fourth aspects.

The solutions provided in the second aspect to the tenth aspect are used to implement or cooperate with implementing the data processing apparatus provided in the first aspect, so that the same or corresponding beneficial effects as those in the first aspect can be achieved, and no further description is given here.

Drawings

Fig. 1 is a schematic diagram of an architecture of an autopilot scenario provided in an embodiment of the present application;

fig. 2 is a schematic architecture diagram of a video monitoring scenario provided in an embodiment of the present application;

FIG. 3 is a functional block diagram of a vehicle 100 provided in an embodiment of the present application;

fig. 4 is a schematic structural diagram of a computer system 101 in a vehicle according to an embodiment of the present application;

fig. 5 is a schematic diagram of a convolutional neural network according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of another convolutional neural network provided in an embodiment of the present application;

fig. 7 is a schematic structural diagram of a data acquisition device and a data processing apparatus according to an embodiment of the present application;

fig. 8 is a schematic diagram of a residual structure circuit according to an embodiment of the present application;

Fig. 9 is a schematic diagram of a residual structure circuit according to an embodiment of the present application;

FIG. 10 is a schematic diagram of another residual structure circuit according to an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of a ReLU activation function according to an embodiment of the application;

FIG. 12 is a schematic diagram of an activation function circuit according to an embodiment of the present disclosure;

FIG. 13 is a schematic diagram of a Leaky ReLU activation function according to an embodiment of the disclosure;

FIG. 14 is a schematic diagram of an activation function circuit according to an embodiment of the present application;

fig. 15 is a schematic diagram of a dash activation function according to an embodiment of the present application;

fig. 16 is a schematic flow chart of a target detection algorithm according to an embodiment of the present application;

FIG. 17 is a flowchart of a binocular depth estimation algorithm according to an embodiment of the present disclosure;

fig. 18 is a schematic diagram of a data acquisition device and a data processing apparatus according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the accompanying drawings. As one of ordinary skill in the art can appreciate, with the development of technology and the appearance of new scenes, the technical solutions provided in the embodiments of the present application are applicable to similar technical problems.

The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which the embodiments of the application described herein have been described for objects of the same nature. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

In recent years, with rapid development of the autopilot field, more and more advanced deep learning algorithms have been developed to be applied to target recognition in the autopilot field. These deep learning algorithms typically increase the performance of the neural network by increasing the depth and the amount of parameters of the neural network to increase the accuracy of target recognition during autopilot. With the improvement of the depth and parameter quantity of the neural network in the deep learning algorithm, the calculation force of the automatic driving vehicle is more required.

In addition, as the number of environmental collection devices used by the automatic driving vehicle increases, and the accuracy of the environmental collection devices increases, the power demand of the automatic driving vehicle increases, resulting in an increase in energy consumption of the operation chip in the automatic driving vehicle. The energy consumption of the operation chip in the automatic driving vehicle is increased, so that the problems of overhigh power consumption, overhigh heating value and the like of the operation chip can be caused, and the normal running of the automatic driving vehicle is easily influenced.

In general, an autonomous vehicle mainly relies on environmental collection devices such as cameras and radars disposed on the vehicle to collect environmental information, and a controller in the autonomous vehicle analyzes the environmental information to determine a driving strategy of the autonomous vehicle. Taking a camera arranged on an automatic driving vehicle as an example, in the running process of the automatic driving vehicle, the camera acquires an environment image in real time, and after compression encoding processing is carried out on the environment image, the compressed environment image is sent to a computing chip of the automatic driving vehicle. After decoding the environment image, the operation chip in the automatic driving vehicle analyzes and processes the decoded environment image based on the neural network to obtain a corresponding data processing result, and the data processing result is used for determining a driving strategy. For example, after the operation chip analyzes and processes the environmental image, an obstacle recognition result in the environmental image is obtained, and the obstacle recognition result is used for determining that the driving strategy is deceleration driving or detouring.

Under the conditions that more environment acquisition devices are deployed in an automatic driving vehicle and the precision of the environment acquisition devices is high, the operation chip generally needs to consume a great deal of calculation power to decode the environment information transmitted by the environment acquisition devices, so that the energy consumption of the operation chip is improved.

In view of this, the embodiment of the application provides a data acquisition device and a data processing apparatus, in which a feature extraction circuit is integrated in the data acquisition device, so that feature extraction can be performed on environmental data acquired by the data acquisition device, and feature data with smaller data volume can be obtained. Then, the data acquisition device sends the characteristic data to the data processing device, and the data processing device performs further processing on the characteristic data to obtain a final data processing result. The data acquisition equipment is used for extracting the characteristics of the acquired environmental data and then sending the characteristic data to the data processing device, so that the data quantity of the transmitted data can be reduced, the energy consumption in the data processing process is reduced, and the data processing efficiency is improved.

The data acquisition device and the data processing apparatus provided in the embodiments of the present application may be, for example, an electronic device in autopilot (self driving), a wireless electronic device in smart grid (smart grid), a wireless electronic device in transportation safety (transportation safety), a wireless electronic device in smart city (smart city), a wireless electronic device in smart home (smart home), an electronic device in deep sea exploration, an electronic device in telemetry and remote sensing, a smart phone (mobile phone), a personal computer (personal computer, PC), a notebook computer, a tablet computer, a smart television, a server, a mobile internet device (mobile internet device, MID), a wearable device (such as a smart watch, smart glasses or smart helmet), a wireless electronic device in Virtual Reality (VR) device, an augmented reality (augmented reality, AR) device, a wireless electronic device in industrial control (industrial control), a wireless electronic device in remote operation (remote medical surgery), or the like. And the data acquisition equipment and the data processing device are in communication connection in a wired or wireless mode, so that the data acquisition equipment can transmit characteristic data to the data processing device. The following embodiments are not particularly limited to the specific form of the data collection device and the data processing apparatus.

For example, the data acquisition device and the data processing apparatus provided in the embodiments of the present application may be applied in the field of automatic driving, video monitoring, deep sea exploration, and remote sensing and telemetry.

Referring to fig. 1, fig. 1 is a schematic diagram of an architecture of an autopilot scenario according to an embodiment of the present application. As shown in fig. 1, in the field of autopilot, a data acquisition device, which may be, for example, a camera or a radar, and a data processing device, which may be, for example, a central controller or a device in a central controller, are deployed in an autopilot vehicle. Specifically, after the data acquisition device acquires environmental data such as an environmental image or point cloud data, the data acquisition device performs feature extraction processing on the environmental data to obtain feature data. The data acquisition device then transmits the characteristic data to the data processing means. After the data processing device receives the characteristic data, the characteristic data is processed to obtain a corresponding data processing result, and the data processing result is used for determining the running strategy of the automatic driving vehicle.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a video monitoring scene according to an embodiment of the present application. As shown in fig. 2, in the field of video monitoring, a video monitoring system is deployed with a data acquisition device and a data processing device, where the data acquisition device may be, for example, a camera deployed outdoors or indoors, and the data processing device may be, for example, deployed in a local computer room or cloud server. Specifically, after the data acquisition equipment acquires the environmental image, the environmental image is subjected to feature extraction processing to obtain feature data, and the feature data is sent to the data processing device. After the data processing device receives the feature data, the feature data is processed to obtain a corresponding data processing result, and the data processing result can be an image classification result or a target detection result, for example.

In the field of deep sea exploration, a detector is connected to a computing center on the ground. The detector is provided with data acquisition equipment, and the computing center on the ground is provided with a data processing device. After the deep sea image is acquired, the data acquisition equipment on the detector extracts the characteristic data of the deep sea image, and the characteristic data is transmitted back to the data processing device of the computing center. And the data processing device sends an instruction to the data acquisition equipment when recognizing that the data acquisition equipment has shot a specific target according to the characteristic data so as to acquire a high-pixel target image.

In the field of telemetry and remote sensing, a drone for implementing telemetry and remote sensing is connected to a computing center on the ground. The unmanned aerial vehicle is provided with data acquisition equipment, and a computing center on the ground is provided with a data processing device. After the ground image is acquired, the data acquisition equipment on the unmanned aerial vehicle extracts the characteristic data of the ground image, and transmits the characteristic data back to the data processing device of the computing center. And the data processing device sends an instruction to the data acquisition equipment when recognizing that the data acquisition equipment has shot a specific target according to the characteristic data so as to acquire a high-pixel target image.

The scenario to which the embodiment of the present application is applied is described above, and for ease of understanding, the apparatus to which the embodiment of the present application is applied in the autopilot scenario will be described below.

Fig. 3 is a functional block diagram of a vehicle 100 according to an embodiment of the present application. In one embodiment, the vehicle 100 is configured in a fully or partially autonomous mode. For example, the vehicle 100 may control itself while in the automatic driving mode, and the current state of the vehicle and its surrounding environment may be determined by a human operation, the possible behavior of at least one other vehicle in the surrounding environment may be determined, and the confidence level corresponding to the possibility of the other vehicle performing the possible behavior may be determined, and the vehicle 100 may be controlled based on the determined information. While the vehicle 100 is in the autonomous mode, the vehicle 100 may be placed into operation without interaction with a person.

The vehicle 100 may include various subsystems, such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, as well as a power source 110, a computer system 112, and a user interface 116. Alternatively, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. In addition, each of the subsystems and elements of the vehicle 100 may be interconnected by wires or wirelessly.

The travel system 102 may include components that provide powered movement of the vehicle 100. In one embodiment, propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels/tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or other type of engine combination, such as a hybrid engine of a gasoline engine and an electric motor, or a hybrid engine of an internal combustion engine and an air compression engine. Engine 118 converts energy source 119 into mechanical energy.

Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source 119 may also provide energy to other systems of the vehicle 100.

The transmission 120 may transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other devices, such as a clutch. Wherein the drive shaft may comprise one or more axles that may be coupled to one or more wheels 121.

The sensor system 104 may include several sensors that sense information about the environment surrounding the vehicle 100. For example, the sensor system 104 may include a positioning system 122 (which may be a GPS system, or a Beidou system, or other positioning system), an inertial measurement unit (inertial measurement unit, IMU) 124, a radar system 126, a laser rangefinder 128, and a camera 130. The sensor system 104 may also include sensors (e.g., in-vehicle air quality monitors, fuel gauges, oil temperature gauges, etc.) of the internal systems of the monitored vehicle 100. Sensor data from one or more of these sensors may be used to detect objects and their corresponding characteristics (location, shape, direction, speed, etc.). Such detection and identification is a critical function of the safe operation of autonomous vehicle 100.

Among them, with the development of Advanced Driving Assistance Systems (ADAS) and unmanned technologies, they put higher demands on the performance of the radar system 126, such as distance, angular resolution, and the like. The improvement of the distance and the angular resolution of the vehicle-mounted radar system 126 makes the vehicle-mounted radar system 126 detect a plurality of measurement points when imaging a target, so as to form high-resolution point cloud data, and the radar system 126 in the application may also be referred to as a point cloud imaging radar.

The positioning system 122 may be used to estimate the geographic location of the vehicle 100. The IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 may be a combination of an accelerometer and a gyroscope.

The radar system 126 may utilize radio signals to sense objects within the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing an object, the radar system 126 may be used to sense the speed and/or heading of the object.

The laser rangefinder 128 may utilize a laser to sense objects in the environment in which the vehicle 100 is located. In some embodiments, laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

The camera 130 may be used to capture a plurality of images of the surrounding environment of the vehicle 100. The camera 130 may be a still camera or a video camera. In the present embodiment, the camera 130 may also be referred to as an image capturing apparatus.

The control system 106 is configured to control the operation of the vehicle 100 and its components. The control system 106 may include various elements including a steering system 132, a throttle 134, a brake unit 136, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

The steering system 132 is operable to adjust the direction of travel of the vehicle 100. For example, in one embodiment may be a steering wheel system.

The throttle 134 is used to control the operating speed of the engine 118 and thus the speed of the vehicle 100.

The brake unit 136 is used to control the vehicle 100 to decelerate. The brake unit 136 may use friction to slow the wheel 121. In other embodiments, the braking unit 136 may convert the kinetic energy of the wheels 121 into electric current. The brake unit 136 may take other forms to slow the rotational speed of the wheels 121 to control the speed of the vehicle 100.

The computer vision system 140 may process and analyze the images captured by the camera 130 to identify objects and/or features in the environment surrounding the vehicle 100. The objects and/or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, in-motion restoration structure (Structure from Motion, SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map an environment, track objects, estimate the speed of objects, and so forth.

The route control system 142 is used to determine a travel route of the vehicle 100. In some embodiments, route control system 142 may incorporate data from sensor 138, GPS 122, and one or more predetermined maps to determine a travel route for vehicle 100.

The obstacle avoidance system 144 is operable to identify, evaluate, and avoid or otherwise overcome potential obstacles in the environment of the vehicle 100.

Of course, in one example, control system 106 may additionally or alternatively include components other than those shown and described. Or some of the components shown above may be eliminated.

The vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users through the peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, a vehicle computer 148, a microphone 150, and/or a speaker 152.

In some embodiments, the peripheral device 108 provides a means for a user of the vehicle 100 to interact with the user interface 116. For example, the vehicle computer 148 may provide information to a user of the vehicle 100. The user interface 116 is also operable with the vehicle computer 148 to receive user input. The vehicle computer 148 may be operated by a touch screen. In other cases, the peripheral device 108 may provide a means for the vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 100. Similarly, speaker 152 may output audio to a user of vehicle 100.

The wireless communication system 146 may communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communications such as code division multiple access (code division multiple access, CDMA), enhanced multimedia disc system (enhanced versatile disk, EVD), global system for mobile communications (global system for mobile communications, GSM)/general packet radio service (general packet radio service, GPRS), or 4G cellular communications such as LTE. Or 5G cellular communication. The wireless communication system 146 may communicate with a wireless local area network (wireless local area network, WLAN) using WiFi. In some embodiments, the wireless communication system 146 may utilize an infrared link, bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 146 may include one or more dedicated short-range communication (dedicated short range communications, DSRC) devices, which may include public and/or private data communications between vehicles and/or roadside stations.

The power source 110 may provide power to various components of the vehicle 100. In one embodiment, the power source 110 may be a rechargeable lithium ion or lead acid battery. One or more battery packs of such batteries may be configured as a power source to provide power to various components of the vehicle 100. In some embodiments, the power source 110 and the energy source 119 may be implemented together, such as in some all-electric vehicles.

Some or all of the functions of the vehicle 100 are controlled by a computer system 112. The computer system 112 may include at least one processor 113, the processor 113 executing instructions 115 stored in a non-transitory computer-readable medium, such as a data storage 114. The computer system 112 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner.

The processor 113 may be any conventional processor, such as a commercially available central processing unit (central processing unit, CPU). Alternatively, the processor may be a special purpose device such as an application specific integrated circuit (application specific integrated circuit, ASIC) or other hardware-based processor. Although FIG. 1 functionally illustrates a processor, memory, and other elements of computer 110 in the same block, it will be understood by those of ordinary skill in the art that the processor, computer, or memory may in fact comprise a plurality of processors, computers, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard disk drive or other storage medium located in a different housing than computer 110. Thus, references to a processor or computer will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering component and the retarding component, may each have their own processor that performs only calculations related to the component-specific functions.

In the embodiment of the present application, the processor 113 may acquire the data of the camera 130 and other sensor devices, and perform positioning of the vehicle based on the acquired data.

In various aspects described herein, the processor may be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle and others are performed by a remote processor, including taking the necessary steps to perform a single maneuver.

In some embodiments, the data storage 114 may contain instructions 115 (e.g., program logic) that the instructions 115 may be executed by the processor 113 to perform various functions of the vehicle 100, including those described above. The data storage 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and/or control one or more of the propulsion system 102, the sensor system 104, the control system 106, and the peripherals 108.

In addition to instructions 115, data storage 114 may store data such as road maps, route information, vehicle location, direction, speed, and other vehicle data, as well as other information. Such information may be used by the vehicle 100 and the computer system 112 when the vehicle 100 is in autonomous, semi-autonomous, and/or manual modes.

A user interface 116 for providing information to a user of the vehicle 100 or receiving information from the vehicle 100. Optionally, the user interface 116 may include one or more input/output devices within the set of peripheral devices 108, such as a wireless communication system 146, a car-in-computer 148, a microphone 150, and a speaker 152.

The computer system 112 may control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the travel system 102, the sensor system 104, and the control system 106) as well as from the user interface 116. For example, the computer system 112 may utilize inputs from the control system 106 to control the steering unit 132 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of vehicle 100 and its subsystems.

Alternatively, one or more of these components may be mounted separately from or associated with vehicle 100. For example, the data storage 114 may exist partially or completely separate from the vehicle 100. The above components may be communicatively coupled together in a wired and/or wireless manner.

Alternatively, the above components are only an example, and in practical applications, components in the above modules may be added or deleted according to actual needs, and fig. 1 should not be construed as limiting the embodiments of the present invention.

An autonomous car traveling on a road, such as the vehicle 100 above, may identify objects within its surrounding environment to determine adjustments to the current speed. The object may be another vehicle, a traffic control device, or another type of object. In some examples, each identified object may be considered independently and based on its respective characteristics, such as its current speed, acceleration, spacing from the vehicle, etc., may be used to determine the speed at which the autonomous car is to adjust.

Alternatively, the vehicle 100 or a computing device associated with the vehicle 100 (e.g., the computer system 112, the computer vision system 140, the data storage 114 of fig. 3) may predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on a road, etc.). Alternatively, each identified object depends on each other's behavior, so all of the identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 100 is able to adjust its speed based on the predicted behavior of the identified object. In other words, an autonomous car is able to determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, the speed of the vehicle 100 may also be determined in consideration of other factors, such as the lateral position of the vehicle 100 in the road on which it is traveling, the curvature of the road, the proximity of static and dynamic objects, and so forth.

In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of the vehicle 100 so that the autonomous vehicle follows a given trajectory and/or maintains safe lateral and longitudinal distances from objects in the vicinity of the autonomous vehicle (e.g., cars in adjacent lanes on the roadway).

The vehicle 100 may be a car, a truck, a motorcycle, a bus, a ship, an airplane, a helicopter, a mower, an amusement ride, a casino vehicle, construction equipment, an electric car, a golf car, a train, or the like, and the embodiment of the present application is not particularly limited.

Referring to fig. 4, fig. 4 is a schematic structural diagram of a computer system 101 in a vehicle according to an embodiment of the present application. As shown in fig. 4, computer system 101 includes a processor 103, with processor 103 coupled to a system bus 105. Processor 103 may be one or more processors, each of which may include one or more processor cores. A display adapter 107, which may drive a display 109, the display 109 being coupled to the system bus 105. The system bus 105 is coupled to an input output (I/O) bus via a bus bridge 111. I/O interface 115 is coupled to an I/O bus. The I/O interface 115 communicates with various I/O devices such as an input device 117 (e.g., keyboard, mouse, touch screen, etc.), a multimedia disk (media track) 121, (e.g., CD-ROM (compact disc read-only memory), multimedia interface, etc.). Transceiver 123 (which may transmit and/or receive radio communication signals), camera 155 (which may capture still and moving digital video images), and external USB port 125. Wherein the interface to which I/O interface 115 is optionally connected may be a USB interface.

The processor 103 may be any conventional processor including a reduced instruction set computing (reduced instruction set Computing, RISC) processor, a complex instruction set computing (complexinstruction set computing, CISC) processor, or a combination thereof. In the alternative, the processor may be a dedicated device such as an ASIC. Alternatively, the processor 103 may be a neural network processor or a combination of a neural network processor and the conventional processors described above.

Alternatively, in various embodiments herein, computer system 101 may be located remotely from the autonomous vehicle and may be in wireless communication with the autonomous vehicle. In other aspects, some of the processes herein are performed on a processor disposed within the autonomous vehicle, others are performed by a remote processor, including taking the actions required to perform a single maneuver.

Computer 101 may communicate with software deploying server 149 through network interface 129. The network interface 129 is illustratively a hardware network interface, such as a network card. The network 127 may be an external network, such as the Internet, or an internal network, such as an Ethernet or virtual private network (virtual private network, VPN). Optionally, the network 127 may also be a wireless network, such as a WiFi network, cellular network, or the like.

The hard drive interface is coupled to the system bus 105. The hardware drive interface is coupled to the hard disk drive. System memory 135 is coupled to system bus 105. The data running in system memory 135 may include an operating system 137 and application programs 143 for computer 101.

The operating system includes Shell 139 and kernel 141.Shell 139 is an interface between the user and the kernel of the operating system. A shell is the outermost layer of the operating system. Shell manages interactions between users and the operating system: waiting for user input, interpreting the user input to the operating system, and processing output results of a variety of operating systems.

Kernel 141 is made up of those parts of the operating system that are used to manage memory, files, peripherals, and system resources. Kernel 141 interacts directly with the hardware, the operating system kernel typically runs processes and provides inter-process communication, CPU time slice management, interrupts, memory management, and IO management, among others.

The application programs 143 include programs that control the automated driving of the vehicle, such as programs that manage the interaction of the automated driving vehicle with obstacles on the road, programs that control the route or speed of the automated driving vehicle, programs that control the interaction of the automated driving vehicle with other automated driving vehicles on the road. Application 143 also resides on the system of the copying server 149. In one embodiment, computer system 101 may download application 143 from software deployment server 149 when execution of application 143 is desired.

A sensor 153 is associated with computer system 101. Sensor 153 is used to detect the environment surrounding computer system 101. For example, the sensor 153 may detect animals, automobiles, obstacles, crosswalks, etc., and further the sensor 153 may detect the environment surrounding such animals, automobiles, obstacles, crosswalks, etc., such as: the environment surrounding the animal, e.g., other animals present around the animal, weather conditions, the brightness of the surrounding environment, etc. Alternatively, if the computer 101 is located on an autonomous car, the sensor may be a radar system or the like.

In the embodiment of the present application, the process of performing feature extraction processing on environmental data by the data acquisition device and the process of processing feature data by the data processing device relate to a neural network model, and for convenience of understanding, related terms, neural network models and other related concepts related to the embodiment of the present application are described below.

(1) A neural network.

The neural network may be composed of neural units, which may refer to an arithmetic unit with xs (i.e., input data) and intercept 1 as inputs, and the output of the arithmetic unit may be:

where s=1, 2, … … n, n is a natural number greater than 1, ws is the weight of xs, and b is the bias of the neural unit. f is an activation function (activation functions) of the neural unit for introducing a nonlinear characteristic into the neural network to convert an input signal in the neural unit to an output signal. The output signal of the activation function may be used as an input to a next convolutional layer, and the activation function may be a sigmoid function. A neural network is a network formed by joining together a plurality of the above-described single neural units, i.e., the output of one neural unit may be the input of another neural unit. The input of each neural unit may be connected to a local receptive field of a previous layer to extract features of the local receptive field, which may be an area composed of several neural units.

(2) The convolutional neural network (Convosutionas Neuras Network, CNN) is a deep neural network with a convolutional structure. The convolutional neural network comprises a feature extractor consisting of a convolutional layer and a sub-sampling layer. The feature extractor can be seen as a filter and the convolution process can be seen as a convolution with an input image or convolution feature plane (feature map) using a trainable filter. The convolutional layers refer to neuron layers (e.g., a first convolutional layer, a second convolutional layer in this embodiment) in the convolutional neural network that perform convolutional processing on an input signal. In the convolutional layer of the convolutional neural network, one neuron may be connected with only a part of adjacent layer neurons. A convolutional layer typically contains a number of feature planes, each of which may be composed of a number of neural elements arranged in a rectangular pattern. Neural elements of the same feature plane share weights, where the shared weights are convolution kernels. Sharing weights can be understood as the way image information is extracted is independent of location. The underlying principle in this is: the statistics of a certain part of the image are the same as other parts. I.e. meaning that the image information learned in one part can also be used in another part. So we can use the same learned image information for all locations on the image. In the same convolution layer, a plurality of convolution kernels may be used to extract different image information, and in general, the greater the number of convolution kernels, the more abundant the image information reflected by the convolution operation.

The convolution kernel can be initialized in the form of a matrix with random size, and reasonable weight can be obtained through learning in the training process of the convolution neural network. In addition, the direct benefit of sharing weights is to reduce the connections between layers of the convolutional neural network, while reducing the risk of overfitting.

Specifically, as shown in fig. 5, fig. 5 is a schematic diagram of a convolutional neural network according to an embodiment of the present application. Convolutional Neural Network (CNN) 100 may include an input layer 110, a convolutional/pooling layer 120, where the pooling layer is optional, and a neural network layer 130.

The structure formed by the convolution layer/pooling layer 120 and the neural network layer 130 may be a first convolution layer and a second convolution layer described in the application, where the input layer 110 is connected to the convolution layer/pooling layer 120, the convolution layer/pooling layer 120 is connected to the neural network layer 130, an output of the neural network layer 130 may be input to an activation layer, and the activation layer may perform a nonlinear processing on the output of the neural network layer 130.

Convolution layer/pooling layer 120. Convolution layer: the convolutional/pooling layer 120 as shown in fig. 5 may include layers as examples 121-126, in one implementation, 121 being a convolutional layer, 122 being a pooling layer, 123 being a convolutional layer, 124 being a pooling layer, 125 being a convolutional layer, 126 being a pooling layer; in another implementation, 121, 122 are convolutional layers, 123 are pooling layers, 124, 125 are convolutional layers, and 126 are pooling layers. I.e. the output of the convolution layer may be used as input to a subsequent pooling layer or as input to another convolution layer to continue the convolution operation.

Taking the example of the convolution layer 121, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels, and function in image processing as a filter that extracts specific information from an input image matrix, where the convolution operators may be essentially a weight matrix, which is usually predefined, and where the weight matrix is usually processed on the input image in a horizontal direction (or two pixels followed by two pixels … … depending on the value of the step size stride) to perform the task of extracting specific features from the image. The size of the weight matrix should be related to the size of the image, and it should be noted that the depth dimension (depth dimension) of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Thus, convolving with a single weight matrix produces a convolved output of a single depth dimension, but in most cases does not use a single weight matrix, but instead applies multiple weight matrices of the same dimension. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific color of the image, another weight matrix is used to blur … … unnecessary noise points in the image, the dimensions of the weight matrices are the same, the dimensions of feature images extracted by the weight matrices with the same dimensions are the same, and the extracted feature images with the same dimensions are combined to form the output of convolution operation.

The weight values in the weight matrices are required to be obtained through a large amount of training in practical application, and each weight matrix formed by the weight values obtained through training can extract information from the input image, so that the convolutional neural network 100 is helped to perform correct prediction.

When convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (e.g., 121) tends to extract more general features, which may also be referred to as low-level features; as the depth of the convolutional neural network 100 increases, features extracted by the later convolutional layers (e.g., 126) become more complex, such as features of high level semantics, which are more suitable for the problem to be solved.

Pooling layer: since it is often desirable to reduce the number of training parameters, the convolutional layers often require periodic introduction of pooling layers, i.e., layers 121-126 as illustrated at 120 in FIG. 2, which may be one convolutional layer followed by one pooling layer, or multiple convolutional layers followed by one or more pooling layers.

Neural network layer 130: after processing by the convolutional layer/pooling layer 120, the convolutional neural network 100 is not yet sufficient to output the required output information. Because, as previously described, the convolution/pooling layer 120 will only extract features and reduce the parameters imposed by the input image. However, in order to generate the final output information (the required class information or other relevant information), convolutional neural network 100 needs to utilize neural network layer 130 to generate the output of the number of classes required for one or a group. Thus, multiple hidden layers (131, 132 to 13n as shown in fig. 2) may be included in the neural network layer 130, and the output layer 140, where parameters included in the multiple hidden layers may be pre-trained according to relevant training data of a specific task type, for example, the task type may include image recognition, image classification, object detection, and so on.

After the underlying layers of the neural network layer 130, i.e., the final layer of the overall convolutional neural network 100 is the output layer 140, the output layer 140 has a class-cross entropy-like loss function, specifically for calculating the prediction error, once the forward propagation of the overall convolutional neural network 100 (e.g., propagation from 110 to 140 in fig. 2) is completed (e.g., propagation from 140 to 110 in fig. 2) and the backward propagation (e.g., propagation from 140 to 110 in fig. 2) begins to update the weights and deviations of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the desired result.

It should be noted that the convolutional neural network 100 shown in fig. 5 is only an example of a convolutional neural network, and the convolutional neural network may also exist in the form of other network models in a specific application. For example, referring to fig. 6, fig. 6 is a schematic diagram of another convolutional neural network according to an embodiment of the present application. The multiple convolutional layers/pooling layers shown in fig. 6 are parallel and the separately extracted features are all input to the full neural network layer 130 for processing.

The scenario applied by the embodiment of the present application and technical concepts such as related terms are introduced above, and the data acquisition device and the data processing apparatus provided by the embodiment of the present application will be described in detail below.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a data acquisition device and a data processing apparatus according to an embodiment of the present application.

As shown in fig. 7, the data acquisition device includes a sensor, a feature extraction circuit, and a data transmission circuit; the data processing apparatus includes a data receiving circuit and a data processing circuit.

In the data acquisition device, sensors are used to acquire environmental data. For example, in the field of autopilot, the data acquisition device may be a camera, for example, the environmental data acquired by a sensor in the camera being an environmental image. The data acquisition equipment can also be radar devices such as a laser radar or a millimeter wave radar, and the environmental data acquired by the laser radar and the millimeter wave radar can be point cloud data. The point cloud data refers to a set of vectors in a three-dimensional coordinate system, that is, a set of sampling points with space coordinates acquired by a radar device, and can represent an obstacle in a three-dimensional space.

In addition, in the field of video monitoring, the data acquisition device may also be a camera. The camera may be, for example, a monitoring camera deployed outdoors or indoors for capturing an image of the environment outdoors or indoors.

The characteristic extraction circuit in the data acquisition equipment is connected with the sensor, and can acquire the environmental data acquired by the sensor. The feature extraction circuit is used for carrying out feature extraction processing on the environmental data based on the neural network model to obtain feature data. Wherein the data volume of the feature data is smaller than the data volume of the environment data, and the feature data can be, for example, image features and can be used for representing feature information in the environment image; alternatively, the feature data may be, for example, a point cloud data feature, and can be used to characterize feature information in the point cloud data. In general, the environmental image and the point cloud data generally contain more redundant information, and feature extraction processing is performed on the environmental image or the point cloud data through the neural network model, so that feature data containing key information can be extracted, and the feature data can be used for a data processing device to execute subsequent data processing. The neural network model used for performing feature extraction processing on the environmental data in the feature extraction circuit may be referred to as a feature extraction network, for example.

For example, assume that the data acquisition device is a camera that is used to acquire high definition video at 1920×1080 resolution, and the acquisition frame rate of the video is 30 frames/second. Then, the number of pixels collected by the camera per second is specifically: 1920×1080×3 channels×30 frames/second=180 mega (M) points/second, i.e. the amount of data transmitted by the camera per second includes a value of 180M pixels. In the case that the camera performs feature extraction based on the feature extraction network and then transmits the feature extraction, the number of pixels required to be transmitted by the camera per second is specifically 13×13×1024×30 frames/second=5M dots/second, which is calculated by the feature extraction network in the target detection algorithm. As a comparison shows, the data amount of the feature data is 2.7% of the original video data. Obviously, the data size of the feature data obtained after the feature extraction processing is far smaller than that of the environment data.

Illustratively, the feature data extracted by the feature extraction circuit is used by the data processing apparatus to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

The data transmission circuit in the data acquisition equipment is connected with the feature extraction circuit, and feature data extracted by the feature extraction circuit can be acquired. The data transmission circuit is configured to transmit the feature data acquired from the feature extraction circuit to the data processing apparatus.

In the data processing device, the data receiving circuit is used for receiving characteristic data sent by the data acquisition equipment, wherein the characteristic data are obtained by the data acquisition equipment after characteristic extraction of acquired environment data based on a neural network model.

The data processing circuit in the data processing device is connected to the data receiving circuit, and can acquire the feature data from the data receiving circuit. The data processing circuit is used for processing the characteristic data to obtain a data processing result. The data processing circuit may, for example, process the feature data based on a neural network model to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation. That is, the data processing result obtained by the data processing circuit after processing the feature data may be an image classification result, a target detection result, a target recognition result, a semantic segmentation result, and/or a depth estimation result.

In some embodiments, the data processing apparatus may be a central processing unit (central processing unit, CPU) or a configurable dedicated circuit. In the field of autopilot, the data processing device may be deployed in a centralized controller in a vehicle; in the field of video monitoring, the data processing device can be deployed in a local machine room or a cloud server.

In short, in this embodiment, the autopilot algorithm or the video surveillance algorithm is divided into two parts, one part is implemented in the data acquisition device, and the other part is implemented in the data processing device. Taking an autopilot algorithm as an example, the autopilot algorithm is divided into a feature extraction algorithm and a decision processing algorithm, wherein the feature extraction algorithm is executed on a data acquisition device such as a camera or a radar device and is used for executing feature extraction processing on environmental data; the decision processing algorithm is executed on a data processing device such as a centralized controller, and is used for analyzing and processing the feature data to obtain a decision result, for example, an image classification result, a target detection result, a target recognition result, a semantic segmentation result or a depth estimation result.

The automatic driving algorithm or the video monitoring algorithm can be implemented based on a neural network model, so that the neural network model for implementing the automatic driving algorithm or the video monitoring algorithm can be divided into two parts, wherein one part is a feature extraction network and the other part is a decision network. The feature extraction network is deployed in a feature extraction circuit in the data acquisition equipment and is used for realizing feature extraction processing of the environmental data; the decision network is deployed in the data processing device and is used for obtaining a data processing result based on the characteristic data.

In this embodiment, the feature extraction circuit is integrated in the data acquisition device, so that feature extraction can be performed on the environmental data acquired by the data acquisition device, and feature data with smaller data size can be obtained. Then, the data acquisition device sends the characteristic data to the data processing device, and the data processing device performs further processing on the characteristic data to obtain a final data processing result. The data acquisition equipment is used for extracting the characteristics of the acquired environmental data and then sending the characteristic data to the data processing device, so that the data quantity of the transmission data can be reduced, the energy consumption of the data processing process is reduced, and the data processing efficiency is improved.

In addition, in the embodiment, the feature extraction circuit in the data acquisition equipment is used for carrying out feature extraction processing on the acquired raw data, and the extracted feature data is sent to the data processing device, so that the whole data processing process is carried out on the basis of the acquired raw data. Compared with the prior art that the collected original data is compressed and then transmitted to the central side for processing, the scheme of the embodiment does not lose detail information, enhances the processing capability of the micro target, and can discover more targets earlier.

Moreover, applicants have found that algorithms that process environmental data (e.g., autopilot algorithms and video surveillance algorithms) typically feature frequent update iterations, due to the rapid development of neural network models. However, in the case of updating the autopilot algorithm and the video surveillance algorithm, the feature extraction process in both algorithms is generally unchanged or less changed, and the process of feature data is generally more changed. Therefore, the feature extraction circuit responsible for the feature extraction processing process is integrated in the data acquisition equipment, so that the data acquisition equipment is suitable for most algorithms and is not influenced by algorithm updating, the applicability of the data acquisition equipment is improved, and frequent updating of software or hardware of the data acquisition equipment is avoided.

In some possible embodiments, the feature extraction circuit in the data acquisition device is an analog circuit and the data processing circuit in the data processing apparatus is a digital circuit. The analog circuit and the digital circuit are two circuits commonly used in the data processing process. Analog circuits refer to circuits that are used to perform transmission, conversion, processing, amplification, measurement, display, and the like of analog signals. Analog signals refer to continuously varying electrical signals. The digital circuit is a circuit that performs arithmetic operation and logical operation on digital quantities using digital signals. A digital signal is a signal in which an independent variable is discrete and a dependent variable is also discrete, the independent variable of such a signal being represented by an integer, and the dependent variable being represented by one of finite numbers. In a computer, the magnitude of a digital signal is often represented by a binary number with a limit.

Digital circuits are capable of performing a wider variety of types and more complex arithmetic operations, such as exponentiations, than analog circuits; moreover, the applicability of the digital circuit is strong, and the same digital circuit can be used for executing different algorithms. Thus, data processing based on neural network models is currently usually done by digital circuits, for example by a CPU or graphics processor (graphics processing unit, GPU).

However, compared with a digital circuit, an analog circuit has a simple structure and small hardware consumption, and can achieve higher operation parallelism with lower consumption. That is, the power consumption of the analog circuit is smaller than that of the digital circuit while providing the same computing power.

The feature processing procedures of these algorithms are generally the same or similar for different autopilot algorithms or video surveillance algorithms, except for the procedure of processing the feature data. That is, the feature extraction networks employed by the different autopilot or video surveillance algorithms are typically the same or similar, and the algorithms differ primarily in the decision network employed.

Based on this, the feature extraction processing is realized with an analog circuit and the processing of feature data is realized with a digital circuit in the present embodiment. Because the characteristic extraction processing processes of most algorithms are the same or similar, the characteristic extraction processing is realized based on the analog circuit, and the characteristic extraction processing requirements of most algorithms can be realized under the condition of not changing the structure of the analog circuit; the processing of the characteristic data is realized based on the digital circuit, so that the characteristic data processing requirements of different algorithms can be met, the data acquisition equipment and the data processing device can support different algorithms, and the flexibility of supporting different algorithms is ensured.

In addition, in the automatic driving algorithm or the video monitoring algorithm, the calculation amount of the feature extraction network is generally higher than that of the decision network, that is, most of calculation amount in the automatic driving algorithm or the video monitoring algorithm is concentrated in the feature extraction processing process. Therefore, the analog circuit replaces the digital circuit to realize the characteristic extraction processing process with high calculation amount, so that the power consumption in the data processing process can be effectively reduced, and the energy consumption of an operation chip or a video monitoring server in the automatic driving vehicle can be reduced.

Specifically, in the field of automatic driving, an analog circuit is used to implement feature extraction processing, and a digital circuit is used to implement processing of feature data, so that the energy efficiency ratio of the entire computing architecture in a vehicle can be improved, for example, the energy efficiency ratio of the current computing architecture is improved from 1to 2to 20to more than ps/W. Wherein, tips is the arithmetic capability unit of the processor, and 1TOPS represents that the processor can perform one trillion times per second; tops/W is used to measure how many trillion operations a processor can do with 1W power consumption.

It is to be understood that, in the case where the feature extraction circuit is an analog circuit, feature data obtained by performing feature extraction processing on environmental data by the feature extraction circuit is an analog signal. Because a certain physical distance exists between the data acquisition device and the data processing device in the automatic driving field or the video monitoring field, and the transmission efficiency and the anti-interference capability of the analog signal are lower than those of the digital signal in the scene of long-distance transmission, in order to ensure the efficiency and the accuracy of data transmission, the characteristic data can be converted into the digital signal and then transmitted in the embodiment.

In particular, the data transmission circuit may comprise an analog-to-digital converter for converting the characteristic data into a digital signal and transmitting the digital signal to the data processing device. That is, the feature extraction circuit performs feature extraction processing on the environmental data to obtain feature data represented by an analog signal; then, the analog-to-digital converter performs an analog-to-digital conversion process on the feature data to obtain feature data expressed as a digital signal, and transmits the feature data expressed as a digital signal to the data processing apparatus. In this way, the data acquisition device transmits the characteristic data to the data processing device in a digital signal mode, so that the efficiency and the accuracy of data transmission are ensured.

Optionally, in the case that the data acquisition device needs to frequently transmit a large amount of data, the data transmission circuit may further include an encoding circuit, where the encoding circuit is connected to the analog-to-digital converter, and is configured to obtain a digital signal from the analog-to-digital converter (i.e., the feature data converted by the analog-to-digital converter), and encode the digital signal to obtain the encoded feature data. Finally, the encoding circuit transmits the encoded characteristic data to a data processing device. The encoding circuit may encode the feature data by using an existing data encoding algorithm, which is not particularly limited in this embodiment.

In particular, the data receiving circuit may also be provided with a data decoding algorithm which is matched to the encoding algorithm in the data acquisition device. Based on the data decoding algorithm, the data receiving circuit can realize the decoding processing of the encoded characteristic data, so that the characteristic data is restored.

The data acquisition equipment is used for carrying out compression coding processing on the characteristic data and transmitting the coded characteristic data to the data processing device, so that the transmitted data volume can be reduced, the characteristic data has certain error correction capability in the transmission process, and the transmission reliability of the data is improved.

For ease of understanding, the implementation of the feature extraction circuit in the data acquisition device will be described in detail below.

Optionally, the feature extraction circuit includes a convolution operation circuit and a residual structure circuit. That is, the feature extraction circuit is used to implement a feature extraction network that includes convolution operations and residual structures.

The convolution operation circuit is used for executing multi-layer convolution operation on the environment data to obtain the characteristic data. Among the multiple layers of convolution operations executed in the convolution operation circuit, the input of the first layer of convolution operation is environment data, and the input of any layer of convolution operation except the first layer is the output of the previous layer of convolution operation. In brief, the feature extraction network for realizing the feature extraction circuit comprises a plurality of convolution layers connected in sequence, and the convolution operation circuit is used for realizing convolution operation in each convolution layer. Since the nature of convolution operation is composed of a large number of multiplication operations and addition operations, a convolution operation circuit may be composed of a combination of a large number of multiplication circuits and addition circuits. In practical applications, the structure of the convolution operation circuit may be determined according to the number of layers of the convolution layer in the feature extraction network and the operation manner of the convolution layer, which is not described herein.

The residual structure circuit is connected with the convolution operation circuit and is used for storing output data of any layer of convolution operation executed by the convolution operation circuit so that the output data is added with output data of a target layer of convolution operation to obtain input data of a next layer of convolution operation of the target layer of convolution operation. The convolution operation layer corresponding to the output data stored by the residual structure circuit is positioned before the convolution operation of the target layer. For example, the residual structure circuit stores the output data of the layer 1 convolution operation such that the output data of the layer 1 convolution operation is added to the output data of the layer 20 convolution operation (i.e., the target layer convolution operation) to obtain the input data of the layer 21 convolution operation.

Specifically, the feature extraction circuit may include one or more residual structure circuits, where different residual structure circuits are used to store output data of different layers of convolution operations. For any one of the residual structure circuits, the residual structure circuit may store output data of a specific layer convolution operation and output the output data of the specific layer convolution operation to one or more subsequent layers of other convolution operations as one of inputs to the one or more subsequent layers of other convolution operations.

For example, reference may be made to fig. 8, and fig. 8 is a schematic diagram of a residual structure circuit according to an embodiment of the present application. As shown in fig. 8, the residual structure circuit may acquire output data of the convolution operations of the layers of the layer 1 convolution operation, the layer 2 convolution operation, the layer 3 convolution operation, and the like, store the acquired output data, and then output the stored output data to one of the inputs as the other layer convolution operation, thereby obtaining actual input data of the other layer convolution operation. For example, the residual structure circuit may acquire and store output data of the layer 1 convolution operation and output the stored output data between the n-1 layer convolution operation and the n-layer convolution operation such that an actual input of the n-layer convolution operation is a sum of the output data of the n-1 layer convolution operation and the output data of the layer 1 convolution operation.

In this embodiment, by introducing a residual structure in the feature extraction network and implementing the residual structure in the feature extraction network by the residual structure circuit, the phenomenon that the feature extraction network is degraded when the convolution operation with more layers occurs can be avoided, and the capability of the feature extraction network to extract features is improved.

Optionally, the feature extraction circuit may further include an activation function circuit, where the activation function circuit is connected to the convolution operation circuit, and the activation function circuit is configured to perform activation function processing on output data of one or more layers of convolution operations performed by the convolution operation circuit. That is, the feature extraction circuit may include one or more activation function circuits, and any one of the activation function circuits may be disposed between two adjacent layers of convolution operations, and configured to perform activation function processing on output data of a previous layer of convolution operation, so that input data of a subsequent layer of convolution operation is output data after performing activation function processing in the previous layer of convolution operation. The activation function circuit may be a circuit for implementing a ReLU activation function, a leak ReLU activation function, and/or a mix activation function.

In this embodiment, the activation function circuit performs activation function processing on the output data of the convolution operation, so that a nonlinear factor can be introduced into the process of feature extraction processing, thereby improving the expression capability of the extracted feature data.

Optionally, the feature extraction circuit further comprises a batch normalization circuit and/or a pooling circuit.

The batch normalization circuit is connected with the convolution operation circuit and is used for carrying out batch normalization processing on output data of one or more layers of convolution operations executed by the convolution operation circuit. That is, the feature extraction circuit may include one or more batch normalization circuits, and any batch normalization circuit may be disposed between two adjacent layers of convolution operations, and configured to perform batch normalization processing on output data of a previous layer of convolution operation, so that input data of a subsequent layer of convolution operation is output data after batch normalization processing is performed in the previous layer of convolution operation.

The batch normalization processing refers to the preservation of the average value of a small batch of data of the neural network model in the training process, and in the reasoning stage of the neural network model, all output data of a certain layer of convolution layer are divided by the average value. Specifically, the batch normalization circuit may store a first threshold in advance, and divide output data of one or more layers of convolution operations output by the convolution operation circuit by the first threshold to obtain data after batch normalization processing; alternatively, the batch normalization circuit may store a second threshold in advance, and multiply the output data of one or more layers of convolution operations output by the convolution operation circuit by the second threshold to obtain data after batch normalization, where the second threshold may be the inverse of the first threshold. In addition, the batch normalization circuit may be configured to update the weight parameter in one or more layers of convolution operations, so that the output data obtained after the convolution operations are data after the batch normalization processing is performed. For example, the batch normalization circuit may divide the weight parameters in one or more layers of convolution operations by a threshold value to obtain updated weight parameters, and the data output by the convolution operation performed based on the updated weight parameters is the data after the batch normalization processing.

In this embodiment, by performing batch normalization processing on the output data of each layer of convolution operation, the output data of each layer of convolution operation can conform to the same distribution manner, so as to ensure stability of the output data of each layer of convolution operation, and further improve generalization capability of the feature extraction network.

In the feature extraction circuit, a pooling processing circuit is connected with the convolution operation circuit, and the pooling processing circuit is used for performing pooling processing on output data of one or more layers of convolution operations executed by the convolution operation circuit.

The pooling processing executed by the pooling processing circuit is used for reducing the data quantity of the features in the feature extraction processing, and the pooling processing can be one or more of mean pooling processing and maximum pooling processing.

For the mean value pooling processing, the characteristic data output in the convolution operation is divided into a plurality of parts, each part comprises a certain number of characteristic values, and then the mean value of the characteristic values in each part is calculated, so that the characteristic mean value of each part in the plurality of parts is obtained, and the characteristic data after pooling processing is obtained. For example, assuming that the feature data output by the convolution operation is a feature map of 100×100, the mean pooling process may divide the feature map into 2500 parts, where each part has a size of 2×2, and then calculate the mean value of the feature values in each part, so as to obtain the feature mean value of each part of 2500 parts, and further obtain the feature data after pooling, that is, a feature map of 50×50. In circuit implementation, the pooling processing circuit may perform a convolution multiplication and addition operation on the eigenvalues included in each part and the vector with the value of 1, to obtain the average value of each part, for example, perform the convolution multiplication and addition operation on the 2×2 matrix and [1,1 ].

For maximum value pooling processing, specifically, feature data output in convolution operation is uniformly divided into a plurality of parts, each part comprises a certain number of feature values, and then the maximum value of the feature values in each part is obtained, so that the feature average value of each part in the plurality of parts is obtained, and further the feature data after pooling processing is obtained. For example, assuming that the feature data output by the convolution operation is a feature map of 100×100, the mean pooling process may divide the feature map into 2500 parts, where the size of each part is 2×2, and then calculate the maximum value of the feature values in each part, so as to obtain the feature mean value of each part of 2500 parts, and further obtain the feature data after the pooling process, that is, a feature map of 50×50. In circuit implementation, the pooling circuit may convert the eigenvalues included in each part into a 1-dimensional vector, for example, convert a 2×2 matrix into a 1*4 vector, and then obtain and output the maximum value in the 1*4 vector through a comparator, so as to obtain the feature data after pooling.

Since the feature extraction circuit may be an analog circuit as described above, the detailed implementation of the residual structure circuit and the activation function circuit in the case where the feature extraction circuit is an analog circuit will be described below.

In one possible implementation, the residual configuration circuit includes a capacitance, an input switch disposed between the capacitance and an input of the residual configuration circuit, and an output switch disposed between the capacitance and an output of the residual configuration circuit. The capacitor is used for storing an input signal when the input switch is closed and the output switch is opened; the capacitor is configured to output the stored signal when the input switch is open and the output switch is closed.

Referring to fig. 9, fig. 9 is a schematic diagram of a residual structure circuit according to an embodiment of the present application. As shown in fig. 9, vin represents an input voltage, sin represents an input switch, sout represents an output switch, a capacitor is connected between Sin and Sout, and Vout represents an output voltage. When Sin is closed and Sout is open, vin is input to the capacitor to be charged, and the voltage Vc after the capacitor is charged is Vin; with Sin open and Sout closed, the capacitor discharges to output a voltage such that Vout is the voltage Vc of the capacitor, i.e., vout=vc=vin.

In another possible implementation, the residual structure circuit may be formed by a combined circuit including a capacitor and a transistor. Referring to fig. 10, fig. 10 is a schematic diagram of another residual structure circuit according to an embodiment of the present application. As shown in fig. 10, the residual structure circuit includes a capacitor C, a switch Sin, a switch Sout, a switch S1, a switch S2, a current transistor NM1, and a source follower transistor NM2.Vin represents an input voltage and Vout represents an output voltage.

In the ready mode, the switches S1 and S2 are closed, the switches Sin and Sout are opened, and the gate-source voltage Vgs2 on NM2 is sampled to the capacitor C, so the voltage vc=vgs 2 of the capacitor C.

In the input mode, switch Sin and switch Sout are closed, and switch S1 and switch S2 are open. Because the upper plate of the capacitor C is in a high-resistance state, the upper plate voltage vc_u of the capacitor C varies with the lower plate voltage of the capacitor C. Since the capacitor C is connected to the input voltage in the input mode, vc_u=vin+vc=vin+vgs 2.

In the output mode, switch Sout is closed, switch Sin, switch S1 and switch S2 are open, and since bias voltage Vbias is constant, NM1 and NM2 currents are constant, the gate-source voltage of NM2 remains unchanged, and thus Vgs 2=vc_u-Vout, i.e., vout=vc_u-Vgs 2=vin, can be obtained.

In this embodiment, the residual structure circuit is implemented by adopting the analog circuit, so that the complex process of storing data into the memory after performing analog-to-digital conversion and outputting the data in a digital-to-analog conversion mode is avoided, and the power consumption required by the analog-to-digital conversion and the digital-to-analog conversion is saved. That is, compared with the residual structure circuit implemented by the digital circuit, the residual structure circuit implemented by the analog circuit can effectively save power consumption.

In one possible implementation, the activation function circuit may be a circuit for implementing a ReLU activation function, and specifically includes a gate and a comparator; the comparator is used for comparing an input signal of the activation function circuit with a zero value signal, sending a first signal to the gating device when the input signal is greater than or equal to the zero value signal, and sending a second signal to the gating device when the input signal is less than the zero value signal; the gate is configured to output the input signal when the first signal is received and to output the zero value signal when the second signal is received. That is, when the input signal is greater than or equal to the zero value signal, the output of the activation function circuit is the input signal; when the input signal is less than the zero value signal, the output of the activation function circuit is the zero value signal.

Referring to fig. 11 and fig. 12, fig. 11 is a schematic diagram of a ReLU activation function according to an embodiment of the present application;

fig. 12 is a schematic diagram of an activation function circuit according to an embodiment of the present application. As shown in fig. 11, for the ReLU activation function, when the input data is greater than or equal to 0, the output of the ReLU activation function is the input data; when the input data is smaller than 0, the output of the ReLU activation function is 0. Based on this, it is possible to refer to fig. 12, in an activation function circuit for implementing a ReLU activation function, a voltage comparator is used to compare a relationship between input data and zero voltage, and transmit a first signal to a gate when the input data is greater than or equal to zero voltage, and transmit a second signal to the gate when the input data is less than zero voltage. In this way, the gating device outputs input data when acquiring the first signal; and inputting a zero value voltage when the second signal is acquired, thereby realizing the function activation in the ReLU.

In another possible implementation, the activation function circuit may be a circuit for implementing a leak ReLU activation function, and the activation function circuit includes a gate, a comparator, and an attenuation circuit connected to the gate. The comparator is used for comparing an input signal of the activation function circuit with a zero value signal, sending a first signal to the gating device when the input signal is greater than or equal to the zero value signal, and sending a second signal to the gating device when the input signal is less than the zero value signal; the attenuation circuit is used for carrying out attenuation treatment on the input circuit to obtain an attenuated input signal; the gate is configured to output the input signal when the first signal is received and to output the attenuated input signal when the second signal is received.

That is, the activation function circuit for implementing the leak ReLU activation function further includes an attenuation circuit for performing attenuation processing on the input data so that the output data is the attenuated input data, as compared with the activation function circuit for implementing the ReLU activation function.

Referring to fig. 13 and 14, fig. 13 is a schematic diagram of a leak ReLU activation function according to an embodiment of the present application; fig. 14 is a schematic diagram of an activation function circuit according to an embodiment of the present application. As shown in fig. 13, for the leak ReLU activation function, when the input data is greater than or equal to 0, the output of the ReLU activation function is the attenuated input data; when the input data is smaller than 0, the output of the ReLU activation function is 0. Based on this, it is possible to refer to fig. 14, in an activation function circuit for implementing a ReLU activation function, a voltage comparator is used to compare the relationship between input data and zero voltage, and send a first signal to the gate when the input data is greater than or equal to zero voltage, and send a second signal to the gate when the input data is less than zero voltage. In this way, the gating device outputs input data when acquiring the first signal; and inputting a zero value voltage when the second signal is acquired, thereby realizing the function activation in the ReLU.

Furthermore, the activation function circuit may also be used to implement a Mish activation function. Specifically, referring to fig. 15, fig. 15 is a schematic diagram of a dash activation function according to an embodiment of the present application. As shown in fig. 15, the expression of the dash activation function can be expressed as: f (x) =x tan h (ln (1+e x)); alternatively, the expression of the Mish activation function may be expressed as: f (x) =x sigmode (x), where sigmode () represents an S-type function.

For easy understanding, the data acquisition device and the data processing apparatus provided in the embodiments of the present application will be described below in conjunction with a part of algorithms in the autopilot field.

Referring to fig. 16, fig. 16 is a flowchart of an object detection algorithm according to an embodiment of the present application. As shown in fig. 16, the target detection algorithm includes two parts, namely: a feature extraction section and a feature data processing section. Wherein the feature extraction is partly done by a Backbone network (Backbone), which may also be referred to as feature extraction network. The feature data processing part is completed by a neck (heck) network and a Head (Head) network.

In the backbone network for performing feature extraction, convolution operations, batch normalization, residual operations, activation function processing, and pooling processing are included. The method comprises convolution operation, batch normalization, activation function processing, up-sampling and fusion processing in a neck network; in the head network, an average pooling process and a softmax process are included.

Referring to fig. 17, fig. 17 is a schematic flow chart of a binocular depth estimation algorithm according to an embodiment of the present application. As shown in fig. 17, the binocular depth estimation algorithm includes two parts, respectively: the feature extraction part and the visual estimation part may be implemented based on different network structures, respectively. The feature extraction part comprises pooling processing, convolution operation, residual operation, batch normalization, activation function processing and n times of convolution operation. The visual estimation part comprises convolution operation, batch normalization, cavity convolution processing, long context up-sampling, occlusion picture output and visual residual picture output.

Referring to fig. 18, fig. 18 is a schematic diagram of a data acquisition device and a data processing apparatus according to an embodiment of the present application. As shown in fig. 18, the data acquisition device includes a sensor, a feature extraction circuit and an analog-to-digital converter, wherein the feature extraction circuit includes a convolution operation circuit, a batch normalization circuit, a residual structure circuit, an activation function circuit and a pooling processing circuit. The data processing device comprises a convolution operation unit, an activation function unit, a Softmax unit, a prediction unit and a classification unit. Specifically, a sensor in the data acquisition equipment inputs acquired environmental data into a feature extraction circuit, the feature extraction circuit extracts feature data, and then an analog-to-digital converter converts the feature data into a digital signal and sends the digital signal to a data processing device. The data processing device processes the characteristic data through the convolution operation unit, the activation function unit, the Softmax unit, the prediction unit and the classification unit to finally obtain the classification result of the image.

The embodiment of the application also provides a data processing method, which is applied to the data acquisition equipment and comprises the following steps: performing feature extraction processing on the environmental data through a neural network model to obtain feature data, wherein the feature extraction processing on the environmental data is realized on the basis of an analog circuit, and the feature data is an analog signal; converting the characteristic data into a digital signal and transmitting the digital signal to a data processing device; the data processing device is deployed outside the data acquisition equipment and is used for processing the digital signals to obtain data processing results.

The embodiment of the application also provides a data processing method, which is applied to the data processing device and comprises the following steps: the method comprises the steps of receiving characteristic data sent by data acquisition equipment, wherein the characteristic data are obtained by extracting characteristics of environment data through a neural network model by an analog circuit in the data acquisition equipment, and the data processing device is deployed outside the data acquisition equipment; and processing the characteristic data to obtain a data processing result.

Further, embodiments of the present application also provide a computer program product, which when run on a computer, causes the computer to perform the steps as performed by the aforementioned data acquisition device or data processing apparatus, or causes the computer to perform the steps as performed by the aforementioned data acquisition device or data processing apparatus.

In an embodiment of the present application, a computer-readable storage medium is further provided, in which a program for performing signal processing is stored, which when executed on a computer causes the computer to perform steps performed by the aforementioned data acquisition device or data processing apparatus, or causes the computer to perform steps performed by the aforementioned data acquisition device or data processing apparatus.

The embodiment of the application also provides a chip system which comprises a processor and an interface circuit, wherein the interface circuit is used for receiving the instruction and transmitting the instruction to the processor. Wherein the processor is configured to implement the method in any of the method embodiments described above.

Optionally, the system on chip further includes a memory, and the processor in the system on chip may be one or more. The processor may be implemented in hardware or in software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general purpose processor, with the method of any of the method embodiments described above being implemented by reading software code stored in a memory.

Alternatively, the memory in the system-on-chip may be one or more. The memory may be integral with the processor or separate from the processor, and is not limited in this application. For example, the memory may be a non-transitory processor, such as a ROM, which may be integrated on the same chip as the processor, or may be separately provided on different chips, and the type of memory and the manner of providing the memory and the processor are not specifically limited in this application.

The above embodiments of the present application are described in detail, and steps in the method of the embodiments of the present application may be sequentially scheduled, combined or pruned according to actual needs; the modules in the device of the embodiment of the application can be divided, combined or deleted according to actual needs.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.

The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

It should be understood that in the embodiments of the present application, "B corresponding to a" means that B is associated with a, from which B may be determined. It should also be understood that determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

Claims

1. A data acquisition device, comprising: a feature extraction circuit and a data transmission circuit;

the characteristic extraction circuit is used for acquiring environment data and carrying out characteristic extraction processing on the environment data based on the neural network model so as to obtain characteristic data, the characteristic extraction circuit is an analog circuit, and the characteristic data is an analog signal;

the data transmission circuit is used for converting the characteristic data into a digital signal and sending the digital signal to the data processing device;

The data processing device is deployed outside the data acquisition equipment and is used for processing the digital signals to obtain data processing results.

2. The data acquisition device according to claim 1, characterized in that the data processing means are in particular adapted to process the characteristic data based on digital circuitry.

3. A data acquisition device according to claim 1 or 2, wherein the data acquisition device and the data processing means are deployed at different locations in the vehicle, respectively.

4. A data acquisition device according to claim 3, wherein the data acquisition device is a camera and the environmental data is image data;

or, the data acquisition equipment is a laser radar or millimeter wave radar, and the environmental data is point cloud data.

5. A data acquisition device according to claim 3 or 4, wherein the characteristic data is used for the data processing means to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

6. The data acquisition device of any one of claims 1-5, wherein the feature extraction circuit comprises a convolution operation circuit, a residual structure circuit;

The convolution operation circuit is used for executing multi-layer convolution operation on the environment data to obtain the characteristic data;

the residual structure circuit is connected with the convolution operation circuit and is used for storing output data of any layer of convolution operation executed by the convolution operation circuit so that the output data is added with output data of a target layer of convolution operation to obtain input data of a next layer of convolution operation of the target layer of convolution operation.

7. The data acquisition device of claim 6, wherein the residual configuration circuit comprises a capacitor, an input switch, and an output switch;

the capacitor is used for storing an input signal when the input switch is closed and the output switch is opened;

alternatively, the capacitor is configured to output the stored signal when the input switch is turned off and the output switch is turned on.

8. The data acquisition device of claim 6 or 7, wherein the feature extraction circuit further comprises an activation function circuit;

the activation function circuit is connected with the convolution operation circuit and is used for carrying out activation function processing on output data of one or more layers of convolution operations executed by the convolution operation circuit.

9. The data acquisition device of claim 8, wherein the activation function circuit comprises a gate and a comparator;

the comparator is used for comparing an input signal of the activation function circuit with a zero value signal and sending a first signal to the gating device when the input signal is greater than or equal to the zero value signal or sending a second signal to the gating device when the input signal is less than the zero value signal;

the gate is configured to output the input signal when the first signal is received or output the zero value signal when the second signal is received.

10. The data acquisition device of claim 8, wherein the activation function circuit comprises a gate, a comparator, and an attenuation circuit coupled to the gate;

the attenuation circuit is used for carrying out attenuation treatment on the input circuit to obtain an attenuated input signal;

The gate is configured to output the input signal when the first signal is received or output the attenuated input signal when the second signal is received.

11. The data acquisition device of any one of claims 1-10, further comprising a sensor for acquiring the environmental data.

12. A data processing apparatus, comprising:

the data receiving circuit is used for receiving characteristic data sent by the data acquisition equipment, the characteristic data are obtained by extracting characteristics of environment data through a neural network model by an analog circuit in the data acquisition equipment, and the data processing device is deployed outside the data acquisition equipment;

and the data processing circuit is used for processing the characteristic data to obtain a data processing result.

13. The data processing apparatus of claim 12, wherein the data processing circuitry is to perform one or more of the following tasks based on the characteristic data: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

14. A data processing apparatus according to claim 12 or 13, wherein the data processing circuit is a digital circuit.

15. A data processing method, wherein the method is applied to a data acquisition device, the method comprising:

performing feature extraction processing on the environmental data through a neural network model to obtain feature data, wherein the feature extraction processing on the environmental data is realized on the basis of an analog circuit, and the feature data is an analog signal;

converting the characteristic data into a digital signal and transmitting the digital signal to a data processing device;

16. The method according to claim 15, wherein the data processing means is specifically adapted to process the characteristic data based on digital circuitry.

17. A method according to claim 15 or 16, wherein the data acquisition device and the data processing apparatus are deployed at different locations in a vehicle, respectively.

18. The method of claim 17, wherein the data acquisition device is a camera and the environmental data is image data;

19. A method according to claim 17 or 18, wherein the characteristic data is used for the data processing apparatus to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

20. The method according to any one of claims 15 to 19, wherein the performing feature extraction processing on the environmental data includes:

performing a multi-layer convolution operation on the environmental data to obtain the feature data;

and in the process of executing the multi-layer convolution operation on the environment data, storing the output data of any one layer of convolution operation so as to enable the output data to be added with the output data of the target layer of convolution operation, and obtaining the input data of the next layer of convolution operation of the target layer of convolution operation.

21. The method of claim 20, wherein the feature extraction process is performed on the environmental data, further comprising:

and in the process of performing multi-layer convolution operation on the environment data, performing activation function processing on the output data of one or more layers of convolution operation.

22. A data processing method, the method being applied to a data processing apparatus, the method comprising:

the method comprises the steps of receiving characteristic data sent by data acquisition equipment, wherein the characteristic data are obtained by extracting characteristics of environment data through a neural network model by an analog circuit in the data acquisition equipment, and the data processing device is deployed outside the data acquisition equipment;

and processing the characteristic data to obtain a data processing result.

23. The method of claim 22, wherein the method is applied to perform one or more of the following tasks: image classification, object detection, object recognition, semantic segmentation, and depth estimation.

24. The method according to claim 22 or 23, wherein said processing said feature data to obtain a data processing result comprises:

and processing the characteristic data based on the digital circuit to obtain a data processing result.

25. A data processing apparatus comprising a memory and a processor; the memory stores code, the processor being configured to execute the code, the data processing apparatus performing the method of any of claims 15 to 24 when the code is executed.

26. A vehicle comprising a data acquisition device according to any one of claims 1-11.

27. The vehicle according to claim 26, characterized in that the vehicle further comprises a data processing device according to any of claims 12-14.

28. A video surveillance system comprising a data acquisition device according to any one of claims 1-11 and a data processing apparatus according to any one of claims 12-14.

29. The video surveillance system according to claim 28, the data processing device being deployed in a server in the cloud.

30. A computer storage medium storing instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 15 to 24.

31. A computer program product, characterized in that it stores instructions that, when executed by a computer, cause the computer to implement the method of any one of claims 15 to 24.