CN112465156A

CN112465156A - System, method and apparatus for neural network computation to optimize predictive maintenance of a vehicle

Info

Publication number: CN112465156A
Application number: CN202010806256.7A
Authority: CN
Inventors: 罗伯特·理查德·诺埃尔·比尔比; 普尔纳·卡利
Original assignee: Micron Technology Inc
Current assignee: Micron Technology Inc
Priority date: 2019-08-12
Filing date: 2020-08-12
Publication date: 2021-03-09
Also published as: US11586194B2; US20210048808A1

Abstract

The present application is directed to systems, methods, and apparatus for neural network computations that optimize predictive maintenance of a vehicle. For example, a data storage device of a vehicle includes: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; at least one storage media component having non-volatile memory; and a controller. The non-volatile memory is configured into a plurality of partitions (e.g., namespaces) having different sets of memory operation settings configured for different types of data related to the artificial neural network ANN. The partition includes a model partition configured to store model data of the ANN. The sensor data stream is applied in the ANN to predict maintenance services for the vehicle. The memory cells of the model partition may be configured for reading, infrequent updating, improved storage capacity, and/or for access in parallel with the input/output of the ANN.

Description

System, method and apparatus for neural network computation to optimize predictive maintenance of a vehicle

Technical Field

At least some embodiments disclosed herein relate generally to maintenance services for vehicles and, more particularly, but not exclusively, to storage and access of neural network models for predictive maintenance of automobiles.

Background

Conventionally, vehicle maintenance is scheduled based on predetermined operating milestones. For example, routine maintenance services may be scheduled every three or six months, or after traveling a predetermined distance (e.g., 3000 miles, 6000 miles, or 15000 miles).

Such accidents can present a safety hazard when a component of the motor vehicle fails or malfunctions during operation of the vehicle. After such an accident, the journey is made to obtain vehicle service as soon as possible, even at inconvenient times.

Recent developments in the field of autonomous driving technology allow computing systems to operate control elements of motor vehicles under at least some conditions without assistance from a human operator of the vehicle.

For example, sensors (e.g., cameras and radar) may be mounted on a motor vehicle to detect the surroundings of the vehicle traveling on a roadway. With or without any input from the vehicle's human operator, a computing system mounted on the vehicle analyzes the sensor inputs to identify conditions and generate control signals or commands for autonomous adjustment of the vehicle's direction and/or speed.

In some arrangements, when the computing system recognizes a situation in which the computing system may not be able to continue operating the vehicle in a safe manner, the computing system alerts a human operator of the vehicle and requests that the human operator take over control of the vehicle and manually drive instead of allowing the computing system to autonomously drive the vehicle.

United states patent No. 9,533,579 entitled "Electronic Control Apparatus for electric Vehicle (Electronic Control Apparatus for electric-drive Vehicle)" published on 3.1.2017 discloses an Electronic Control Apparatus of a Vehicle having a self-diagnostic function.

Autonomous driving and/or Advanced Driving Assistance Systems (ADAS) typically involve an Artificial Neural Network (ANN) for identifying events and/or objects captured in sensor inputs.

Generally, an Artificial Neural Network (ANN) uses a network of neurons to process inputs to the network and produce outputs from the network.

For example, each neuron in the network receives a set of inputs. Some of the inputs to the neurons may be outputs of certain neurons in the network; and some of the inputs to the neurons may be inputs provided to the neural network. Input/output relationships among neurons in the network represent neuron connectivity in the network.

For example, each neuron may have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function may be in the form of a step function, a linear function, a log-sigmoid function, and the like. Different neurons in the network may have different activation functions.

For example, each neuron may generate a weighted sum of its input and its bias and then generate an output that is a function of the weighted sum, the output being calculated using the activation function of the neuron.

The relationship between the inputs and outputs of an ANN is generally defined by an ANN model that contains data representing the connectivity of neurons in the network, as well as the bias, activation function, and synaptic weights of each neuron. Using the given ANN model, the computing device computes the output of the network from the given set of inputs to the network.

For example, input to the ANN network may be generated based on camera input; and the output from the ANN network may be an identification of an item such as an event or object.

The Spiking Neural Network (SNN) is a type of ANN that closely mimics the natural neural network. When the level of activation of a neuron is high enough, the SNN neuron generates a pulse as an output. The activation level of SNN neurons mimics the membrane potential of natural neurons. The output/pulse of an SNN neuron may alter the level of activation of other neurons receiving the output. The current activation level of an SNN neuron over time is typically modeled using differential equations and is considered to be the state of the SNN neuron. Afferent impulses from other neurons may push the activation level of the neuron higher to reach the threshold of the impulse. Once a neuron generates a pulse, its activation level is reset. Prior to generating the pulse, the activation level of the SNN neuron may decay over time, as governed by a differential equation. The temporal elements in the behavior of SNN neurons make SNNs suitable for processing spatiotemporal data. The connectivity of SNNs is typically sparse, which is beneficial for reducing the computational workload.

In general, an ANN may be trained using a supervised approach in which parameters in the ANN are adjusted to account for errors between known outputs produced from respective inputs and calculated outputs produced from applying the inputs to the ANN. Examples of supervised learning/training methods include reinforcement learning, and learning to perform error correction.

Alternatively or in combination, the ANN may be trained using an unsupervised approach in which the precise output produced by a given set of inputs is unknown prior to completion of the training. The ANN may be trained to classify items into multiple categories, or to classify data points into clusters.

Multiple training algorithms may be employed for complex machine learning/training paradigms.

Disclosure of Invention

In one aspect, the present disclosure is directed to a data storage device of a vehicle, comprising: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; a non-volatile memory configured into a plurality of partitions, the partitions each having a different set of memory operation settings configured for different types of data related to an artificial neural network, wherein the partitions include a model partition configured to store model data for the artificial neural network; and a controller configured to operate memory units in the partitions according to the set of memory operation settings; wherein the artificial neural network is configured to predict maintenance services for the vehicle based on the stream of sensor data.

In another aspect, the present disclosure is directed to a vehicle comprising: at least one processor; at least one sensor configured to generate a sensor data stream; and a data storage device having: a host interface configured to receive the sensor data stream; a non-volatile memory configured into a plurality of partitions, the partitions each having a different set of memory operation settings configured for different types of data related to an artificial neural network, wherein the partitions include a model partition configured to store model data for the artificial neural network; and a controller configured to operate memory units in the partitions according to the set of memory operation settings; wherein the artificial neural network is configured to predict maintenance services for the vehicle based on the stream of sensor data.

In another aspect, the present disclosure is directed to a method comprising: configuring non-volatile memory of a data storage device into a plurality of partitions; different sets of memory operation settings that store different types of data related to an artificial neural network, respectively for the partitions, wherein the partitions include model partitions configured to store model data for the artificial neural network; receiving, in the data storage device, a sensor data stream from at least one sensor disposed on a vehicle; operating memory cells in the partition according to the set of memory operation settings in response to the sensor data stream; and predicting maintenance services for the vehicle based on the sensor data stream using the artificial neural network.

Drawings

Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements.

FIG. 1 illustrates a system in which a vehicle is configured with a data storage device to collect and process sensor data, according to some embodiments.

FIG. 2 illustrates an autonomous vehicle having a data storage device, according to one embodiment.

Figures 3-5 illustrate training of an artificial neural network for maintaining service predictions, in accordance with some embodiments.

FIG. 6 illustrates a method of predictive maintenance, according to one embodiment.

FIG. 7 illustrates a data store accelerating neural network computations, according to one embodiment.

Figure 8 illustrates a storage media component that accelerates neural network computations, according to one embodiment.

FIG. 9 illustrates a method of accelerating neural network computations in a motor vehicle, according to one embodiment.

FIG. 10 illustrates a data storage device configured to support neural network computations, according to one embodiment.

FIG. 11 illustrates a configuration of a namespace for an Artificial Neural Network (ANN) model, according to one embodiment.

FIG. 12 illustrates a configuration of a namespace for inputs to artificial neurons, according to one embodiment.

FIG. 13 illustrates a configuration of a namespace for output from an artificial neuron, in accordance with one embodiment.

14-16 illustrate a method of predictive maintenance supported by model partitioning, input partitioning, and output partitioning, according to one embodiment.

Detailed Description

At least some embodiments disclosed herein provide systems, methods, and apparatus to process sensor data generated in a motor vehicle, or another vehicle with or without an Advanced Driving Assistance System (ADAS), to facilitate predictive maintenance.

Before a component of a motor vehicle fails or malfunctions during operation of the vehicle, there may be an indication of whether the component needs to be replaced or serviced. Such indications may not be noticeable to a typical driver or passenger. However, sensor data may be collected and analyzed to predict the probability of component failure. The predictions may be used to schedule maintenance services, which may reduce or eliminate the chance of an accident that a component of the vehicle will fail or malfunction during operation of the vehicle on the roadway. Furthermore, the prediction allows service trips to be scheduled at convenient times.

For example, sensors may be installed in an automobile system to collect data during routine operation thereof; and the sensor data can be used to predict whether and how long after a component needs to be replaced or serviced. The sensor data may be provided as input to an Artificial Neural Network (ANN), such as a Spiking Neural Network (SNN), of an Artificial Intelligence (AI) system to train itself (e.g., using unsupervised machine learning techniques) during periods of time in which the vehicle is expected to operate normally. The training customizes the neural network for the particular operating environment of the driver, passenger, or user of the vehicle and the personalized operating habits of the vehicle passenger. Subsequently, the artificial neural network may detect an abnormal condition when the operational data deviates from the normal pattern. The AI system may be used to suggest maintenance services and/or identify components that may need replacement or maintenance.

The system of fig. 1 includes a vehicle 111 having a data storage device 101. Optionally, the vehicle 111 has an Advanced Driving Assistance System (ADAS)105 and one or more sensors 103 that provide sensor data inputs to the ADAS105 and/or the data storage device 101. The data storage device 101 is configured to use an Artificial Neural Network (ANN)125 to predict/identify the need for maintenance services based on data collected by the sensors 103. The ADAS105 may be omitted without affecting the predictive maintenance feature. In some embodiments, at least a portion of the data generated by the sensors 103 is in the ADAS105 for driving assistance and the ANN125 for maintenance prediction. Optionally, the output of the ANN 124 may be used in the data storage device 101 and in the ADAS 105.

The sensors 103 may include digital cameras, light, radar, ultrasonic sonar, brake sensors, speed sensors, acceleration sensors, airbag sensors, Global Positioning System (GPS) receivers, audio sensors/microphones, vibration sensors, force/stress sensors, deformation sensors, motion sensors, temperature sensors, and the like. Some of the sensors 103 may be primarily configured to monitor the environment of the vehicle 111; and the other sensors 103 may be primarily configured to monitor operating conditions of one or more components of the vehicle 111 (e.g., an internal combustion engine, an exhaust system, an electric motor, brake pads, tires, a battery, etc.).

The time-varying outputs of the sensors 103 are provided as a stream of sensor data to the ADAS105 and/or the ANN125 to provide driving assistance (e.g., autonomous driving) and maintenance prediction.

For example, vehicle 111 may have a wireless communication device that communicates with remote server 119 via wireless signal 113 and communication network 117. The remote server 119 is typically deployed at a location remote from the roadway 102 on which the vehicle 111 is operating. For example, the vehicle 111 may provide some sensor data 121 to the server 119 and receive updates of the ANN125 from the server 119.

One example of a communication network 117 is a cellular telephone network having one or more base stations (e.g., 115) that receive wireless signals (e.g., 113). Another example of a communication network 117 is the internet, where wireless local area network signals (e.g., 113) transmitted by the vehicle 113 are received in an access point (e.g., 115) for further transmission to the server 119. In some embodiments, the vehicle 111 uses a communication link 107 to a satellite 109 or a communication balloon in communication with a server 119.

The server 119 may also communicate with one or more maintenance service facilities (e.g., 127) to receive maintenance service data 123 for the vehicle (e.g., 111). The maintenance service data 123 may include inspection records and/or service records of components of the vehicle (e.g., 111). For example, the inspection record and/or service record may indicate a degree of wear of a component being inspected at a maintenance service facility (e.g., 127) during service of the component, an identification of a faulty or malfunctioning component, and so forth. Sensor data 121 and maintenance service data 123 for a period of time prior to service of a vehicle (e.g., 111) may be used to train the ANN125 to predict the probability that a component needs maintenance service. The updated ANN125 may be used to predict and recommend maintenance services for the vehicle 111 based on the sensor data 121 received over the recent time period. Alternatively, the updated ANN125 may be transmitted to the vehicle 111; and the vehicle 111 may use the data generated from the sensors 103 during routine operation of the vehicle 111 to predict and recommend maintenance services.

The data storage device 101 of the vehicle 111 may be configured to record sensor data over a period of time used in the ANN for predictive maintenance. Maintenance forecasts are typically for a relatively long period of time (e.g., days, weeks, and/or months). In contrast, sensor data recorded for review of an accident, collision, or near-collision involving an autonomous vehicle is typically for a short period of time (e.g., 30 seconds to minutes). Thus, a typical black box data recorder configured to record sensor data for use in reviewing/analyzing an accident or crash is not sufficient for predictive maintenance.

Optionally, the data storage device 101 stores sensor data regarding a time period that caused a trip to a maintenance service (e.g., 127). A maintenance service facility (e.g., 127) may download sensor data 121 from data storage 101 and provide sensor data 121 and corresponding maintenance service data 123 to server 119 to facilitate training of ANN 125.

Optionally, or in combination, the data storage device 101 is configured with a machine learning module to customize and/or train the ANN125 installed in the vehicle 111 for predictive maintenance.

For example, the machine learning module of the data storage device 101 may be used to calibrate the ANN125 to account for typical/everyday environments in which the vehicle 111 is operated and/or driving preferences/habits of the driver of the vehicle 111.

For example, during periods when the vehicle is expected to operate with healthy components in a typical/everyday environment, sensor data generated by the sensors 103 may be used to train the ANN125 to recognize patterns of sensor data that represent unobstructed operation. Such modes may differ for different vehicles (e.g., 111) based on their routine operating environment and the driving habits/characteristics of their drivers. The training allows the ANN125 to detect deviations from the recognized normal pattern and report maintenance predicted anomalies.

For example, the ANN125 may include an SNN configured to classify and/or detect time-based changes in sensor data from known sensor data patterns of the vehicle 111 operating in normal/healthy conditions but in a personalized environment (e.g., a driver/passenger's daily route) and/or operating in a personalized driving habit/pattern.

FIG. 2 illustrates an autonomous vehicle 111 having a data storage device 101 according to one embodiment. For example, the vehicle 111 in the system of FIG. 1 may be implemented using the autonomous vehicle 111 of FIG. 2.

The vehicle 111 of fig. 2 is configured with an Advanced Driving Assistance System (ADAS) 105. The ADAS105 of the vehicle 111 may have an Artificial Neural Network (ANN)125 for object detection, recognition, identification, and/or classification. The ANN125 and/or another neural network (e.g., configured in the data storage device 101) may be used to predict the probability that a component of the vehicle 111 requires maintenance service (e.g., repair, replacement, or adjustment).

Preferably, the data storage device 101 is configured to process the sensor data at least partially for predictive maintenance, with reduced computational burden on the processor 133 responsible for operating the ADAS105 and/or other components (e.g., infotainment system 149).

The vehicle 111 typically includes an infotainment system 149, a communication device 139, one or more sensors 103, and a computer system 131 connected to some control of the vehicle 111 (e.g., a steering control 141 for the direction of the vehicle 111, a brake control 143 for stopping the vehicle 111, an acceleration control 145 for the speed of the vehicle 111, etc.). In some embodiments, the vehicle 111 in the system of fig. 1 has a similar configuration and/or similar components.

Operation of the ADAS105 requires some of the sensors 103; and some of the sensors 103 are used to collect data related to the health of components of the vehicle 111 that may not be used in the ADAS 105. Optionally, sensor data generated by the sensor 103 may also be used to predict the likelihood of an impending component failure. Such predictions may be used in the ADAS105 to take emergency action to bring the vehicle to a safe state (e.g., by slowing down and/or stopping).

The computer system 131 of the vehicle 111 includes one or more processors 133, data storage 101, and memory 135 that stores firmware (or software) 147, including computer instructions and data models for the ADAS 105.

The one or more sensors 103 of the vehicle may include a visible light camera, an infrared camera, a light radar, a radar or sonar system, a peripheral device sensor, a Global Positioning System (GPS) receiver, a satellite positioning system receiver, a brake sensor, and/or an airbag sensor. Further, the sensors 103 may include audio sensors (e.g., microphones), vibration sensor pressure sensors, force sensors, stress sensors configured to monitor noise from various components and orientations in the vehicle 111, and/or deformation sensors configured to measure loads on components of the vehicle 111, accelerometers and/or gyroscope sensors that measure motion of some components of the vehicle 111, and so forth. Such sensors may be used to monitor the operating state and/or health of the component for predictive maintenance.

Sensor 103 may provide a real-time sensor data stream to computer system 131. The sensor data generated by the sensors 103 of the vehicle 111 may include capturing an image of an object using a camera, or sonar, radar, or light-reaching system, that is a camera that captures the image using light visible to the human eye, or that captures the image using infrared light. The image data obtained from the at least one sensor of the vehicle is part of the collected sensor data recorded in the data storage device 101 and/or as input to the ANN 125. For example, a camera may be used to obtain roadway information for the vehicle 111 to travel, which the ANN125 may process to generate control signals for the vehicle 111. For example, the camera may be used to monitor the operating state/health of components of the vehicle 111, which the ANN125 may process to predict or schedule maintenance services.

The sensor data generated by the sensors 103 of the vehicle 111 may include an audio stream that captures characteristics of sound at an orientation on the vehicle 111 (e.g., an orientation near an engine, a motor, a transmission system, wheels, doors, windows, etc.). The audio data obtained from the at least one sensor 103 of the vehicle 111 may be part of the collected sensor data recorded in the data storage device 101 and/or as input to the ANN 125. For example, the audio stream may be used to monitor the operating state/health of components of the vehicle 111 (e.g., internal combustion engine, exhaust system, electric motor, brakes) that the ANN125 may process to predict or schedule maintenance services.

The infotainment system 149 may be used to present predicted or scheduled maintenance services. Optionally, the communication device 139 may establish a connection to the driver's mobile device of the vehicle 111 to notify the driver of recommended maintenance services and/or recommended service data, to schedule appointments, and the like.

When the vehicle 111 is configured with the ADAS105, the output of the ADAS105 may be used to control (e.g., 141, 143, 145) the acceleration of the vehicle 111, the speed of the vehicle 111, and/or the direction of the vehicle 111 during autonomous driving.

In fig. 3, a module 171 for supervising machine learning is used to train the artificial neural network 125 to minimize the difference between the service prediction 129 generated from the sensor data 121 and the maintenance service data 123.

For example, the maintenance service data 123 may identify measured wear of components over time to predict when a recommended service will be performed. The sensor data 121 may be used in the ANN125 to generate a predicted time at which the recommended service is to be performed. The supervised machine learning module 171 may adjust the artificial neural network 125 to reduce/minimize the difference between the time predicted based on the sensor data 121 and the time calculated from the wear measurements.

For example, the maintenance service data 123 may identify components that are replaced or repaired in the maintenance service facility 127. Sensor data 121 recorded during a period of time prior to replacement or repair of a component may be used to calculate the time at which the replacement or repair is to be performed. Further, segments of the sensor data stream over a period of time prior to a replacement or repair may be used in the ANN125 to generate a prediction of when a replacement or repair is to be performed. Supervised learning 171 may be used to adjust the ANN125 to reduce the predicted time to perform a replacement or repair and the actual time to perform a replacement or repair.

Supervised learning 171 of fig. 2 may be applied in server 119 to generate a generic ANN for a population of vehicles based on sensor data of the population of vehicles and maintenance service data 123 thereof.

The supervised learning 171 of fig. 2 may be applied in the vehicle 111 to generate a customized/personalized ANN for a population of vehicles based on the vehicle's sensor data and its maintenance service data 123. For example, a general purpose ANN may be initially used in the vehicle 111; and the sensor data of the vehicle 111 and its maintenance service data 123 may be used to further train the vehicle's ANN125 for customization/personalization of the ANN125 in the vehicle 111.

In fig. 4, unsupervised machine learning module 175 is used to train or optimize artificial neural network 125 to facilitate anomaly detection 173. The unsupervised machine learning module 175 is configured to adjust the ANN (e.g., SNN) for the classifications, clusters, or recognized patterns in the sensor data 121 such that a degree of deviation from the classifications, clusters, or recognized patterns in the sensor data 121 generated over the most recent time period may be used to signal the detection 173 of the anomaly. Anomaly detection 173 allows for scheduling vehicle 111 for inspection in maintenance service 127. Optionally, after inspection, as in fig. 3, maintenance service data 123 may be used to apply supervised learning 171 to produce a more accurate prediction of service.

Generally, it may be assumed that the vehicle 111 is operating under normal/healthy conditions for a certain period of time. For example, after initial delivery of use of a new vehicle 111, it may be assumed that the vehicle 111 provides barrier-free service for at least a period of time (e.g., months). For example, after a period of time following replacement or repair of a component, it may be assumed that the component provides unobstructed service (e.g., months or years) for at least a period of time. Thus, the sensor data 121 obtained during this time period may be pre-classified as "normal" to train the ANN125 using unsupervised learning 175 in fig. 4 or supervised learning 171 in fig. 5.

For example, sensor data 121 collected via during "normal" service periods of the vehicle 111 or component may be classified into clusters via unsupervised learning 175. Different clusters may correspond to different types of normal conditions (e.g., driving on different routes, driving on roads with different road conditions, driving under different weather conditions, driving during different time periods of the day, driving during different days of the week, driving under different moods of the driver's driving habits). When the continuous sensor data 121 is classified outside of the "normal" cluster, an anomaly is detected.

Optionally, as illustrated in fig. 5, supervised machine learning 171 may be used to train the ANN 125. Expected classification 177 may be used to label sensor data 121 during a "normal" service period of vehicle 111 or a component. Supervised learning 171 may be used to minimize classification differences between predictions 179 from sensor data 121 using ANN125 and expected classifications 177. Further, when sensor data 121 is known to be "abnormal" (e.g., in maintenance service 127 or after a diagnosis by a user, driver, or passenger of vehicle 111), it is contemplated that classification 177 may change to "abnormal" for further training ANN125 to directly identify the abnormality (e.g., instead of relying on deviations from a known "normal" cluster to infer the abnormality).

Accordingly, the ANN125 may be trained to identify abnormal sensor data and schedule maintenance services by estimating the severity of the abnormality.

FIG. 6 illustrates a method of predictive maintenance, according to one embodiment. For example, the method of fig. 6 may be implemented in the data storage device 101 in the vehicle 111 of fig. 1 or 2 or the computer system 131 in the vehicle 111 of fig. 2.

At block 201, a sensor (e.g., 103) installed in a vehicle 111 generates a sensor data stream (e.g., 121) during operation of the vehicle 111 on a road 102.

At block 203, the sensor data stream (e.g., 121) is provided into an Artificial Neural Network (ANN) 125. For example, the ANN125 may include a Spiking Neural Network (SNN).

At block 205, an Artificial Neural Network (ANN)125 generates a prediction of maintenance services based on the sensor data stream (e.g., 121).

At block 207, the data storage device 101 disposed on the vehicle stores at least a portion of the sensor data stream (e.g., 121).

At block 209, an Artificial Neural Network (ANN) is trained using a stream of sensor data (e.g., 121) collected over a predetermined period of time since the vehicle left the plant or maintenance service 127.

For example, an Artificial Neural Network (ANN) may be configured to identify components of the vehicle 111 that need repair or replacement in a maintenance service and/or to identify a predicted time period to component failure or malfunction, or a suggested time period to a recommended maintenance service for the component prior to the component failure or malfunction. Thus, the performance of the predicted maintenance service may avoid the occurrence of a component failure or malfunction when the vehicle 111 is operating on the roadway 102.

For example, the sensor 103 may be a microphone mounted near the component, a vibration sensor attached to the component, a pressure sensor mounted in the component, a force or stress sensor mounted to or attached to the component, a deformation sensor attached to the component, an accelerometer configured to measure a parameter of motion of the component.

Optionally, the data storage device 101, the vehicle 111 of the computer system 131, and/or the server 119 remote from the vehicle may have a machine learning module configured to train an Artificial Neural Network (ANN)125 during a period of time when the vehicle 111 is assumed to be in a healthy state (e.g., a predetermined period of time from when the vehicle 111 leaves the plant or maintenance service 127).

For example, as illustrated in fig. 4, the machine learning module may train the ANN125 using unsupervised machine learning 175 to recognize/classify normal patterns of the sensor data 121 and thus have the ability to detect anomalies based on deviations from the normal patterns. Alternatively, as illustrated in fig. 3 or 5, supervised machine learning 171 may be used.

For example, unsupervised machine learning 175 may be applied by the data storage device 101 or the computer system 131 of the vehicle 111 during a predetermined period of time in which the vehicle and/or component is known to be operating without obstruction or degradation.

Alternatively or in combination, some of the sensor data 121 stored in the data storage 101 of the vehicle 111 may be uploaded to the server 119 for use in training the ANN 125.

In at least some embodiments disclosed herein, the data storage device 101 is configured to accelerate computations by an Artificial Neural Network (ANN)125 of the vehicle 111.

For example, in addition to typical operations that support data access and storage, the data storage device 101 may be further configured to perform at least a portion of calculations involving an Artificial Neural Network (ANN)125, such as generating predictions (e.g., 129 or 173) or classifications (e.g., 179) from the sensor data 121 and/or adjusting the ANN125 by unsupervised machine learning 175 (e.g., as illustrated in fig. 4) and/or supervised machine learning 171 (e.g., as illustrated in fig. 3 or 5).

For example, computations configured in the data storage device 101 may be used to reduce the amount of data transmitted to the processor 133 for use or application by the ANN125 and/or to reduce the computational tasks of the processor 133 when evaluating the output of the ANN125 and/or when training the ANN 125. Such an arrangement may result in faster output from the data storage device 101 and/or reduced energy usage, as data to and from the memory does not have to be moved to a dedicated independent neural network accelerator. The computational capabilities of the data storage device 101 in processing data related to the ANN125 allow the computer system 131 of the motor vehicle 111 to monitor the health of the vehicle components (e.g., in a non-real-time manner, or pseudo-real-time manner), with reduced or no impact on the processing of mission-critical tasks (e.g., autonomous driving via the ADAS 105). Furthermore, the computing power of the data storage device 101 may be used to accelerate the processing of sensor data for the ADAS105 and thus improve the processing of mission critical tasks.

FIG. 7 illustrates a data storage device 101 that accelerates neural network computations, according to one embodiment. For example, the data storage device 101 of FIG. 7 may be used to implement the data storage device 101 of the vehicle 111 illustrated in FIG. 1 or 2

In fig. 7, data storage device 101 has a host interface 157 configured to communicate with a processor (e.g., 133). For example, communication between the processor (e.g., 133) and the host interface 157 may be in accordance, at least in part, with a communication protocol for a peripheral component interconnect express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Universal Serial Bus (USB) bus, and/or a Storage Area Network (SAN).

For example, the host interface 157 may be used to receive sensor data 121 generated by the sensors 103 of the vehicle 111 to optionally store a portion of the sensor data 121 in the storage media components 161-163.

For example, each of the storage media components 161-163 can be a memory integrated circuit configured to store data. For example, media components 161 or 163 may include one or more integrated circuit dies embedded in an integrated circuit package. An integrated circuit die may have a plurality of memory cells formed thereon to store data.

Generally, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and can retain stored data even when not powered.

Examples of non-volatile memory include flash memory, memory cells formed based on NAND (NAND) logic gates, NOR (NOR) logic gates, Phase Change Memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point storage devices, and memory devices. Cross-point memory devices use memory elements with fewer transistors, each of which has memory cells and selectors stacked together in columns. Columns of memory elements are connected via two vertical wire layers, with one layer above and the other layer below the columns of memory elements. Each memory element may be individually selected at the intersection of one wire on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include Read Only Memory (ROM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), and Electronically Erasable Programmable Read Only Memory (EEPROM) memory, among others. Examples of volatile memory include Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM).

The data storage device 101 may have a controller 151 that includes volatile local memory 153 and at least one processing device 155.

The local memory of the controller 151 may be an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines that control the operation of the processing device 155, including handling communications between the data storage device 101 and a processor (e.g., 133) of the vehicle 111, as well as other functions described herein. The local memory 151 of the controller 151 may include Read Only Memory (ROM) for storing microcode and/or memory registers to store, for example, memory pointers, fetched data, etc., and/or volatile memory such as Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM).

In FIG. 7, data storage device 101 includes a neural network accelerator 159 coupled to controller 151 and/or storage media components 161-163.

For example, the neural network accelerator 159 may be configured to perform matrix arithmetic calculations. The calculations involve the ANN125 having matrix multiply and accumulate operations, which may be computationally intensive for general purpose processors (e.g., 133). Performing matrix arithmetic calculations using the neural network accelerator 159 may reduce the data to be transmitted to the processor 133 of the vehicle 111 and reduce the computational workload of the processor 133.

For example, when the ANN125 includes a Spiking Neural Network (SNN), the simulation of differential equations used to control the activation levels of SNN neurons may be computationally intensive for a general purpose processor (e.g., 133). Neural network accelerator 159 may use specialized hardware to model differential equations and thus improve the computational efficiency of computer system 131 as a whole.

In some implementations, the neural network accelerator 159 is an integrated circuit device that is separate from the controller 151 and/or the storage media components 161-163. Alternatively or in combination, the neural network accelerator 159 is integrated with the controller 151 in an integrated circuit package. Further, as illustrated in fig. 8, the neural network accelerator 159 may be integrated in at least one of the storage media components 161-163.

Figure 8 illustrates a storage media component 160 that accelerates neural network computations, according to one embodiment. For example, each or some of the storage media components 161-163 of FIG. 7 may be implemented using the storage media component 160 of FIG. 8.

In fig. 8, the storage media component 160 may be housed within an integrated circuit package. An input/output (I/O) interface 171 of the storage media assembly 160 is configured to handle input/output signals in pins of the integrated circuit package. For example, the input/output signals may include address signals that specify a position in media unit 175, and data signals that represent data written in media unit 175 at the position specified via the address signals, or data retrieved from the position in media unit 175.

In fig. 8, the neural network accelerator 159 is coupled with the control logic 173 and/or the media unit 175 to perform calculations used in evaluating the output of the ANN125 and/or in training the ANN 125.

For example, the input/output interface 171 may receive an address identifying a matrix stored in the media unit and operated on via the neural network accelerator 159. The storage media component 160 may provide the results of the calculations of the neural network accelerator 159 as output data responsive to the address, store the output data in a buffer for further operations, store the output data into a position in the media unit 175 specified via the address signal. Thus, the calculations performed by the neural network accelerator 159 may be within the storage media component 160, the storage media component 160 being proximate to the media units 175 in which the matrix data is stored. For example, each of the media units 175 may be an integrated circuit die having memory units of non-volatile memory formed thereon.

For example, state data for SNN neurons may be stored in media unit 175 according to a predetermined pattern. The neural network accelerator 159 may automatically update the state of the SNN neuron over time according to a differential equation for controlling the activation level of the SNN neuron. Optionally, the neural network accelerator 159 is configured to process pulses of neurons in the neural network. Alternatively, the neural network accelerator 159 and/or the processor 133 of the data storage device 101 may be configured to process pulses of neurons and/or accumulations of inputs to the SNN.

FIG. 9 illustrates a method of accelerating neural network computations in a motor vehicle, according to one embodiment. For example, the method of fig. 9 may be implemented in the vehicle 111 of fig. 1 or 2 using the data storage device 101 of fig. 7 and/or the storage media component 160 of fig. 8. For example, the method of fig. 9 may be used in combination with the method of fig. 6.

At block 221, the data storage device 101 of the vehicle 111 receives a sensor data stream from at least one sensor (e.g., 103) disposed on the vehicle 111.

At block 223, the data storage device 101 stores at least a portion of the sensor data stream.

At block 225, a neural network accelerator 159 configured within the data storage device 101 performs at least a portion of the calculations based on the artificial neural network 125 and the stream of sensor data.

At block 227, maintenance services for the vehicle 111 are predicted based at least in part on calculations performed by the neural network accelerator 159 configured within the data storage device 101.

Optionally, at block 229, in the vehicle 111, an Artificial Neural Network (ANN) is trained, at least in part, using the neural network accelerator and using the stream of sensor data collected over a predetermined period of time (e.g., a period of time after the shipment of a new vehicle 111 or after a component replacement in the maintenance service 127).

For example, the neural network accelerator 159 may be configured on an integrated circuit device that is separate from the controller 151 of the data storage device and/or separate from the storage media components 161-163.

For example, neural network accelerator 159 may be configured on an integrated circuit device that includes controller 151 of data storage device 101, or on an integrated circuit device that includes storage media components 160, 161, or 163 of data storage device 101.

For example, the neural network accelerator 159 may be configured to perform calculations using data stored in the data storage device 101, such as matrix arithmetic calculations for ANN and/or differential equation simulations for SNN.

Examples of matrix arithmetic calculations include matrix multiply and accumulate operations. After performing calculations using data stored in the data storage device 101 to produce results of matrix arithmetic calculations, the neural network accelerator 159 may provide the results as output by the data storage device 111 when retrieving data (e.g., in response to a read command). Alternatively or in combination, the results of the matrix arithmetic calculations may be buffered in the data storage device 101 as operands for a next matrix calculation to be performed in conjunction with a data matrix retrieved from non-volatile memory via a read command received in the host interface 157.

When the Artificial Neural Network (ANN)125 comprises a Spiking Neural Network (SNN), the neural network accelerator may be configured to simulate a differential equation that controls the activation level of neurons in the Spiking Neural Network (SNN). Optionally, the storage medium component is configured to store the state of neurons in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to automatically update the state of the neuron over time according to the differential equation. For example, the neural network accelerator 159 may be configured to detect anomalies via an unsupervised machine learning training impulse neural network (SNN).

The calculations performed by the neural network accelerator 159 in accordance with the Artificial Neural Network (ANN)125 involve different data types having different usage patterns for the data storage device 101.

For example, using the Artificial Neural Network (ANN)125 to predict includes using data specifying a model of the Artificial Neural Network (ANN)125, input data provided to the artificial neuron, and output data generated by the artificial neuron.

The storage capacity of the data storage device 101 may be partitioned into different portions of the different types of ANN-related data. The different portions may be individually configured to optimize access to and storage of corresponding data according to usage patterns of the neural network accelerator 159 and/or the processor 133 of the computer system 131 in which the data storage device 101 is configured.

A model of an Artificial Neural Network (ANN)125 may include parameters that specify static attributes of individual artificial neurons in the ANN125 and neuron connectivity in the ANN 125. The model data of the ANN125 is static and does not change during prediction calculations performed using the ANN 125. Therefore, it is usually the usage pattern in which the model data is read. However, when the updated ANN125 is installed, the model data of the ANN125 may change. For example, the vehicle 111 may download the updated ANN125 from the server 119 to the data storage 101 of the vehicle 111 to update its predictive capabilities. The model data of the ANN125 may also change during or after training the ANN125 using machine learning techniques (e.g., 171 or 175). It is preferred that individual partitions or namespaces of the data storage device 101 are configured to store the model data, wherein the partitions or namespaces operate according to configuration parameters that optimize memory cells for a particular usage pattern (e.g., mostly read, rarely updated) for the model data. For example, when memory cells are implemented using NAND logic gate based flash memory, the memory cells in the ANN model partition/namespace can be configured to operate in a multi-level cell (MLC) mode, a three-level cell (TLC) mode, or a four-level cell (QLC) mode, where each memory cell stores two, three, or four bits for increased storage capacity.

The input data provided to the artificial neurons in the ANN125 may include external inputs and internal inputs. The external input is typically generated by the sensors 103 of the vehicle 111 rather than by artificial neurons in the ANN 125. The external input may be saved in a round robin fashion so that the input data for the latest time period of the predetermined driving length may be found in the data storage device 101. Thus, it is preferred that a separate partition or namespace of the data storage device 101 is configured to store the external input data, wherein the partition or namespace operates according to configuration parameters that optimize memory cells (e.g., enhance durability of cyclic overwrites) for the storage pattern of the external input data. For example, when memory cells are implemented using NAND logic gate based flash memory, the memory cells in the ANN input partition/namespace can be configured to operate in a Single Level Cell (SLC) mode, where each memory cell stores one bit of data for improved endurance in a cyclic overwrite operation.

In some embodiments, the artificial neuron may have a state variable that changes over time in response to input during the prediction computation. For example, the activation level of a spiking neuron may change over time and be considered as a dynamic state variable of the spiking neuron. In some embodiments, such state variable data of the artificial neuron has a similar memory usage pattern as external input data; and thus, the state variable data may be stored in a partition or namespace configured for external input data. In other embodiments, state variable data for the artificial neuron is maintained in a buffer and stored less frequently than external inputs; and thus, another partition/namespace may be configured to store dynamic state variable data for the artificial neuron.

Output data generated by artificial neurons in the ANN125 may be buffered for further access by the neural network accelerator 159 and/or the processor 133 of the computer system 131. The output data may include external outputs and internal outputs. The external input is generated by an artificial neuron as a result of an output from the ANN125, such as a classification or prediction by the ANN 125. The output of the ANN125 is typically further processed by a processor 133 of the computer system 131. The external input may be saved periodically (e.g., in a manner similar to storing state variable data). Some of the internal outputs and/or external outputs may be internal inputs to artificial neurons in the ANN 125. In general, it may not be necessary to store the internal output from the buffer of the data storage device to the storage media component. In some embodiments, data storage device 101 may expand the capacity of the buffer using the swap partition/namespace when the buffering capacity of data storage device 101 is insufficient to hold the entire state variable data and/or internal output. The switch partition/namespace may be configured for optimizing random access and for improving durability.

The dynamic state of the external outputs and/or neurons may be saved in a round robin fashion in separate output partitions or namespaces so that the external output data and/or dynamic state of the neurons may be stored periodically and the latest set of external outputs and/or dynamic states may be found in the data storage 101. The dynamic state of the external outputs and/or neurons may be selectively stored, as some of this data may be regenerated by the ANN from external inputs stored in the input partition or namespace. Preferably, the output partition or namespace is configured to store one or more sets of external outputs and/or dynamic states that cannot be generated from external inputs stored in the input partition or namespace. When data is stored in an input/output partition or namespace in a round robin fashion, the oldest stored dataset is erased to make room for the newest dataset. The ANN input/output partition/namespace may be configured for optimized continuous writing of the stream to copy data from a buffer of the data storage device into memory cells in a storage media component of the data storage device.

FIG. 10 illustrates a data storage device 101 configured to support neural network computations, according to one embodiment. For example, the data storage device 101 may be used in the vehicle 111 in fig. 1 or 2 to facilitate predictive maintenance and/or support of the ADAS 105.

Similar to data storage device 101 of FIG. 7, data storage device 101 of FIG. 10 includes a host interface 157 and a controller 151.

Similar to storage media components 161-163 in data storage device 101 of FIG. 7, storage capacity 181 of data storage device 101 of FIG. 10 can be implemented using a set of storage media components.

The set of namespaces 183, 185, 187, … can be created on the storage capacity 181 of the data storage device 101. Each of the namespaces (e.g., 183, 185, or 187) corresponds to a named portion of the storage capacity 181. Logical addresses are defined within each namespace. Address map 191 is configured to map between logical addresses defined in namespaces 183, 185, 187 … to physical addresses of memory cells in storage media components (e.g., 161-163 illustrated in FIG. 7).

Address map 191 may contain namespace optimization settings 192 for namespaces 183, 185, and 187.

For example, the ANN model namespace 183 may be a memory/storage partition configured to model data for an Artificial Neural Network (ANN) 125. Namespace optimization settings 192 optimize memory operations in the ANN model namespace 183 based on the data usage pattern of the ANN model (e.g., mostly read, rarely update-centric).

For example, the neuron input namespace 185 may be a memory/storage partition configured for external input data to the Artificial Neural Network (ANN) 125. Namespace optimization setting 192 optimizes memory operations in neuron input namespace 185 according to a data usage pattern of external input data (e.g., enhanced durability to support circular overwriting of a continuous input data stream for continuous writes).

For example, the neuron output namespace 187 may be a memory/storage partition/configured for external output data provided from an Artificial Neural Network (ANN) 125. Namespace optimization setting 192 optimizes memory operations in neuron output namespace 187 according to a data usage pattern of external output data (e.g., improving endurance of periodically overwriting data by random read/write access).

The data storage device 101 includes a buffer 152 configured to store temporal/intermediate data of an Artificial Neural Network (ANN)125, such as internal input/output of artificial neurons in the ANN 125.

Optionally, a swap namespace may be configured in the storage capacity 181 to expand the capacity of the buffer 152.

Optionally, address map 191 comprises a mapping between logical memory addresses received in host interface 157 to access data of the artificial neuron and an identity of the artificial neuron. Thus, a read or write command that accesses one type of data of an artificial neuron in one namespace may cause controller 151 to access another type of data of an artificial neuron in another namespace.

For example, in response to a request to write external input data for a neuron into the storage capacity 181 of the data storage 185, the address map 191 may be used to calculate addresses of model parameters for neurons in the ANN model namespace 183 and read the model parameters into the buffer 152 to allow the neural network accelerator 159 to perform calculations of outputs of the neurons. The output of a neuron may be saved in buffer 152 as an internal input to other neurons (e.g., to reduce write amplification). Additionally, the identities of other neurons connected to the neuron may also be retrieved from the ANN model name space 183 into the buffer 152, which allows the neural network accelerator 159 and/or processor to further process the propagation of the output in the ANN 125. The retrieval of model data from the ANN model namespace 183 may be performed in parallel with the storage of external input data into the neuron input namespace 185. Thus, the processor 133 of the computer system 131 of the vehicle 111 does not have to explicitly send a read command to retrieve model data from the ANN model namespace 183.

Similarly, in response to reading the output data of the neuron, address map 191 may be used to compute the addresses of the model parameters of the neuron stored in ANN model namespace 183 and read the model parameters into buffer 152 to allow neural network accelerator 159 to apply the internal inputs in buffer 152 to perform the computation of the output of the neuron. The computed output may be provided as a response to the output data of the read neuron without having to cause the data storage device 101 to store the output data in a storage medium component (e.g., 161-163). Thus, the processor 133 and/or the neural network accelerator 159 may control the computation of the neurons via writing inputs to the neurons and/or reading outputs from the neurons.

In general, the external input data passed into the ANN125 may be raw sensor data 121 generated directly by the sensors 103 without processing by the processor 133 and/or the neural network accelerator 159. Alternatively, indirect sensor data 121 that has been processed by the processor 133 of the ANN125 from signals from the sensors 103 may be provided as external input data. Incoming external input data may be accepted in the host interface 157 and written into the neuron input namespace 185 in a round-robin fashion, and automatically buffered in the buffers 152 of the neural network accelerator 159 to generate neuron outputs using the models stored in the ANN model namespace 183. The output generated by the neural network accelerator 159 may be further buffered as internal input to further apply the model in the ANN model namespace 183. When the external output becomes available, data storage device 101 may report completion of the write request by an indication of the availability of the external output. Optionally, the controller 151 and/or the neural network accelerator 159 may generate internal read commands to propagate signals in the ANN125 when generating external outputs. Alternatively, host processor 133 may control the propagation of signals in ANN125 by selectively reading the output of neurons; and the data storage device 101 may actively buffer data that may be needed in the buffer 152 to speed up the ANN calculation.

FIG. 11 illustrates a configuration of a namespace 183 for an Artificial Neural Network (ANN) model, according to one embodiment. For example, the configuration of FIG. 11 may be implemented in the data storage device 101 illustrated in FIGS. 7 and/or 10. For example, the settings 193 of FIG. 11 can be part of the namespace optimization settings 192 of FIG. 10.

The configuration of fig. 11 maps ANN model namespace 183 to at least one storage media component a 161. Preferably, at least one storage media component a 161 is available for use by controller 151 in parallel with storage media components (e.g., 163) of other namespaces (e.g., 185 and 187) hosting the ANN data. For example, storage media component a 161 can be in an integrated circuit package separate from integrated circuit packages used for other namespaces (e.g., 185 and 187). Alternatively, storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 can be formed on separate regions of the integrated circuit die, where the separate regions can operate substantially in parallel (e.g., for reading, for erasing, and/or for writing).

In fig. 11, the setting 197 is optimized to a usage pattern with mostly reads and few updates.

FIG. 12 illustrates a configuration of a namespace 185 for inputs to artificial neurons, according to one embodiment. For example, the configuration of FIG. 11 may be implemented in the data storage device 101 illustrated in FIGS. 7 and/or 10. For example, the setting 195 of FIG. 11 can be part of the namespace optimization setting 192 of FIG. 10.

The configuration of FIG. 12 maps the neuron input namespace 185 to at least one storage media component B163. Preferably, at least one storage media component B163 is available to controller 151 in parallel with storage media components (e.g., 161) of other namespaces (e.g., 183 and 187) hosting the ANN data. For example, storage media component B163 may be in an integrated circuit package that is separate from integrated circuit packages for other namespaces (e.g., 183 and 187). Alternatively, storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 can be formed on separate regions of the integrated circuit die, where the separate regions can operate substantially in parallel (e.g., for reading, for erasing, and/or for writing).

In fig. 12, a usage pattern optimized to enhance durability in recording a cyclic continuous overwrite of a continuous input data stream sampled at fixed time intervals is set 197.

Fig. 13 illustrates a configuration of a namespace 187 for output from an artificial neuron, in accordance with one embodiment. For example, the configuration of FIG. 11 may be implemented in the data storage device 101 illustrated in FIGS. 7 and/or 10. For example, the settings 197 of FIG. 11 may be part of the namespace optimization settings 192 of FIG. 10.

The configuration of fig. 13 maps neuron output name space 187 to at least one storage media component C162. Preferably, at least one storage media component C162 is available to controller 151 in parallel with storage media components (e.g., 161 and 163) of other namespaces (e.g., 183 and 185) hosting the ANN data. For example, storage media component C162 may be in an integrated circuit package separate from integrated circuit packages used for other namespaces (e.g., 183 and 185). Alternatively, storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 can be formed on separate regions of the integrated circuit die, where the separate regions can operate substantially in parallel (e.g., for reading, for erasing, and/or for writing).

In fig. 13, the setting 197 is optimized to a usage pattern that periodically overwrites the buffered data by random access. For example, the memory cells are configured via optimization settings 193-197 to update/overwrite in the neuron output namespace 187 at a frequency higher than in the ANN model namespace 183, but lower than in the neuron input namespace 185.

FIG. 14 illustrates a method of predictive maintenance supported by model partitioning, according to one embodiment. For example, the method of fig. 14 may be implemented in the vehicle 111 of fig. 1 or 2 using the data storage device 101 of fig. 7 or 10 and/or the storage media component 160 of fig. 8. For example, the method of fig. 14 may be used in combination with the methods of fig. 6 and/or 9.

At block 241, the non-volatile memory of data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187, …). For example, non-volatile memory can have the same type of memory cells (e.g., NAND flash memory cells) used to store data; and the same type of memory cells in different partitions (e.g., 183-187) may be configured differently to optimize their performance according to the usage pattern of the data stored in the different partitions (e.g., 183-187).

At block 243, the data storage device 101 stores, for partitions (e.g., 183, 185, 187, …), respectively, different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125, wherein the partitions (e.g., 183, 185, 187, …) include model partitions (e.g., 193) configured to store model data for the artificial neural network 125.

At block 245, the data storage device 101 receives a sensor data stream (e.g., 121) from at least one sensor 103 disposed on the vehicle 111.

At block 247, controller 151 of data storage device 101 operates the memory cells in partitions 183, 185, 187, … according to a set of memory operation settings (e.g., 193, 195, 197) in response to the stream of sensor data (e.g., 121).

At block 249, the computer system 131 having the data storage device 101 predicts maintenance services for the vehicle 111 based on the sensor data stream (e.g., 121) using the artificial neural network 125.

For example, the memory operation settings configure the model partition (e.g., 183) to store three or more bits per memory cell. The memory operation settings may include an address map 191 that maps between neurons in the ANN125 and inputs to the neurons. When a first address is received that is destined for an input to a neuron in the artificial neural network 125, the first address in the input partition (e.g., 185) that is separate from the model partition (e.g., 183) may be translated into at least one second address of model data associated with the neuron, such that attributes of the neuron and an identification of the neuron connected to the neuron may be retrieved from the model partition (e.g., 183) without an explicit command from the processor 133. Controller 151 may, in response to receiving the first address, automatically retrieve model data associated with the neuron from a model partition (e.g., 183) using at least one second address. The neural network accelerator 159 may generate an output of a neuron from inputs to the neuron and model data associated with the neuron. In general, the inputs to the neurons may include outputs from a plurality of neurons connected to the neurons in the ANN 125. Controller 151 may save the output of the neuron in a buffer 152 in data storage device 101 to facilitate accelerated access to the output by host processor 133 and/or neural network accelerator 159.

Typically, model data does not change during the calculations used to predict maintenance services. For example, the model data may include neuron connectivity data for the artificial neural network and static attributes of neurons in the artificial neural network. The memory operation settings (e.g., 192) may configure the model partition (e.g., 183) to store more than one bit per memory cell in the non-volatile memory based on a usage pattern in which model data is mostly read and rarely updated.

For example, a partition (e.g., 183, 185, 187, …) in a data store may be implemented as a namespace in which logical addresses are defined; and address map 191 in the data store is configured to map namespaces 183, 185, 187, … to individual storage media components (e.g., 161, 163, 162, …).

The model data in model namespace 183 may be updated during training via machine learning 171 or 175 or during over-the-air updates to ANN125 from server 119.

In some embodiments, controller 151 is configured to retrieve model data associated with neurons in the artificial neural network from a model zone via address map 191 in response to an input to or an output from a neuron being addressed in a zone separate from model zone 183. Further, controller 151 may retrieve model data associated with neurons from model zone 183 in parallel with neurons storing inputs into zones (e.g., 185) separate from model zone 183.

FIG. 15 illustrates a method of predictive maintenance supported by input partitioning, according to one embodiment. For example, the method of fig. 15 may be implemented in the vehicle 111 of fig. 1 or 2 using the data storage device 101 of fig. 7 or 10 and/or the storage media component 160 of fig. 8. For example, the method of fig. 15 may be used in combination with the methods of fig. 6, 9, and/or 14.

At block 261, the non-volatile memory of data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187, …). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) implemented in multiple storage media components (e.g., 161-163).

At block 263, the data storage device 101 stores, for partitions (e.g., 183, 185, 187, …), respectively, different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125, wherein the partitions (e.g., 183, 185, 187, …) include an input partition (e.g., 185) configured to cyclically store input data for the artificial neural network 125.

For example, the input partition 185 may be configured to store external inputs rather than internal inputs for the artificial neural network 125. The input data stored in the input partition 185 is independent of the output from the neurons in the artificial neural network 125.

For example, the input data stored in the input partition 185 may include a portion of a stream of sensor data (e.g., 121). In some embodiments, the input data stored in the input partition 185 is computed from a stream of sensor data (e.g., 121) for a subset of neurons in the artificial neural network 125.

For example, the memory operation settings (e.g., 195) configure the input partition 185 to store one bit per NAND memory cell in non-volatile memory for enhancing the endurance of repeated data erases and data programming.

For example, the memory operation settings (e.g., 195) configure the controller to write input data into the input partition 185 in sequence and overwrite the oldest input data in the input partition 185 with the newest input data received in the data storage device 101.

At block 265, the data storage device 101 receives a sensor data stream (e.g., 121) from at least one sensor 103 disposed on the vehicle 111.

At block 267, controller 151 of data storage device 101 operates the memory cells in partitions 183, 185, 187, … according to a set of memory operation settings (e.g., 193, 195, 197) in response to the stream of sensor data (e.g., 121).

At block 269, the computer system 131 having the data storage device 101 predicts maintenance services for the vehicle 111 based on the sensor data stream (e.g., 121) using the artificial neural network 125.

FIG. 16 illustrates a method of predictive maintenance supported by input partitioning, according to one embodiment. For example, the method of fig. 16 may be implemented in the vehicle 111 of fig. 1 or 2 using the data storage device 101 of fig. 7 or 10 and/or the storage media component 160 of fig. 8. For example, the method of fig. 16 may be used in combination with the methods of fig. 6, 9, 14, and/or 15.

At block 281, the non-volatile memory of data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187, …). For example, non-volatile memory can have the same type of memory cells (e.g., NAND flash memory cells) used to store data.

At block 283, the data storage device 101 stores, for partitions (e.g., 183, 185, 187, …), respectively, different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125, wherein the partitions (e.g., 183, 185, 187, …) include an output partition (e.g., 187) configured to store output data for the artificial neural network 125.

For example, the output data stored in the output partition (e.g., 187) may include state data for neurons in the artificial neural network 125. For example, state data of neurons in an artificial neural network may identify activation levels of the neurons for generating pulses in a spiking neural network. The level of activation can be controlled via a differential equation. Thus, the activation level may change in response to an input to the artificial neural network 125 and/or in response to the passage of time.

For example, the output data may include predictions or classifications generated by the artificial neural network 125 in response to the sensor data streams.

For example, the memory operation settings configure the output partition to store no more than two bits per memory cell in the non-volatile memory.

At block 285, the data storage device 101 receives a sensor data stream (e.g., 121) from at least one sensor 103 disposed on the vehicle 111.

At block 287, the controller 151 of the data storage device 101 operates the memory cells in the partitions 183, 185, 187, … according to a set of memory operation settings (e.g., 193, 195, 197) in response to the stream of sensor data (e.g., 121).

At block 289, the computer system 131 having the data storage device 101 predicts maintenance services for the vehicle 111 based on the sensor data stream (e.g., 121) using the artificial neural network 125.

For example, data storage device 101 may include buffer 152. The buffer 152 may be implemented via volatile memory (e.g., SRAM or DRAM) for faster access performance compared to non-volatile memory (e.g., NAND flash memory) of the data storage 101. The memory operation settings configure controller 151 to store output data in buffer 152 for access by a processor (e.g., 133) via host interface 157 during or after storage of the output data to output partition 187.

For example, the data storage device 101 may include a neural network accelerator 159 coupled to the controller 151. The neural network accelerator is configured to apply inputs provided to neurons in the artificial neural network 125 to model data of the artificial neural network 125 to generate output data by one or more output neurons in the artificial neural network 125. In response to the neural network accelerator 159 completing the computation of the output data, the controller is configured to provide an indication to the processor (e.g., 133) of the availability of the output data generated by the artificial neural network 125 so that the processor (e.g., 133) can request the data storage device 101 to transmit the output data.

Optionally, the controller 151 is configured to provide the output data to the processors in parallel with storing the output data into the output partitions. For example, the controller 151 may be configured to automatically discard output data calculated for a previous segment of the sensor data stream if the processor (e.g., 133) does not request that the output data be transmitted to the processor (e.g., 133) within a predetermined time period or before a next version of the output data is available. Optionally, after reporting the availability of the output data to the processor (e.g., 133), the controller 151 may be configured to selectively discard the output data calculated for the previous segment of the sensor data stream based on the processor (e.g., 133) response to the output data sent to the processor (e.g., 133). For example, in some cases, a processor (e.g., 133) may request that output data be transmitted to the processor (e.g., 133) without saving the output data into an output partition (e.g., 187); and in other cases, the processor (e.g., 133) may request that the output data be transmitted to the processor (e.g., 133) and stored into the output partition (e.g., 187).

Optionally, output data from the artificial neural network 125 may also be stored in a round robin fashion into output partitions (e.g., for segments of output data within a time period selected by a processor (e.g., 133)).

For example, external inputs to the artificial neural network 125 may be continuously recorded in the input namespace 185 for the last time period T1. When the sensor data is sampled at predetermined time intervals T2, the input namespace 185 may save the latest T1/T2 input data set. In contrast, external outputs from the artificial neural network 125 may be selectively recorded into the output namespace 187 (e.g., once every predetermined time period T3, where T3 is a multiple of T2). The output data may be recorded into output namespace 187 at a lower frequency; and an output namespace 187 may be allocated to store a predetermined number of output data sets (e.g., the last output data set is maintained via successive writes and writes that occur in a round robin fashion).

Server 119, computer system 131, and/or data storage device 101 may each be implemented as one or more data processing systems.

The present disclosure includes methods and apparatus to perform the methods described above, including data processing systems that perform these methods, and computer readable media containing instructions that when executed on data processing systems cause the systems to perform these methods.

A typical data processing system may include interconnects (e.g., buses and system core logic) that interconnect a microprocessor and a memory. The microprocessor is typically coupled to a cache memory.

The interconnect interconnects the microprocessor and the memory together and also to input/output (I/O) devices via an I/O controller. The I/O devices may include display devices and/or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I/O devices (e.g., printer, scanner, mouse, and/or keyboard) are optional.

The interconnect may include one or more buses connected to each other through various bridges, controllers, and/or adapters. In one embodiment, the I/O controller includes a USB (Universal Serial bus) adapter for controlling USB peripheral devices, and/or an IEEE-1394 bus adapter for controlling IEEE-1394 peripheral devices.

The memory may include one or more of the following: read-only memory (ROM), volatile Random Access Memory (RAM), and non-volatile memory such as hard drives, flash memory, and the like.

Volatile RAM is typically implemented as dynamic RAM (dram), which requires power continually in order to refresh or maintain the data in the memory. The non-volatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other type of memory system that maintains data even after power is removed from the system. The non-volatile memory may also be a random access memory.

The non-volatile memory may be a local device coupled directly to the rest of the components in the data processing system. Non-volatile memory that is remote from the system, such as a network storage device coupled to the data processing system through a network interface such as a modem or Ethernet interface, may also be used.

In the present disclosure, some functions and operations are described as being performed by or caused by software code to simplify description. However, this expression is also used to specify that the function is produced by a processor, such as a microprocessor, executing code/instructions.

Alternatively or in combination, the functions and operations as described herein may be implemented using special purpose circuitry, with or without software instructions, such as with an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). Embodiments may be implemented using hardwired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the data processing system.

While one embodiment may be implemented in fully functional computers and computer systems, the various embodiments are capable of being distributed as a computing product in a variety of forms, and are capable of being applied regardless of the particular type of machine or computer-readable media used to actually carry out the distribution.

At least some aspects of the disclosure may be embodied, at least in part, in software. That is, the techniques may be performed in a computer system or other data processing system in response to its processor, such as a microprocessor, executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage).

The routines executed to implement the embodiments, may be implemented as part of an operating system or a specific application, component, program, article, module, or sequence of instructions referred to as a "computer program". The computer programs typically include one or more sets of instructions at various times in various memories and storage devices in the computer, and which, when read and executed by one or more processors in the computer, cause the computer to perform the necessary operations to execute elements relating to the various aspects.

A machine-readable medium may be used to store software and data which when executed by a data processing system causes the system to perform various methods. Executable software and data may be stored in various places including, for example, ROM, volatile RAM, non-volatile memory, and/or cache memory. Portions of this software and/or data may be stored in any of these storage devices. Further, the data and instructions may be obtained from a centralized server or a peer-to-peer network. Different portions of the data and instructions may be obtained from different centralized servers and/or peer-to-peer networks at different times and in different communication sessions or the same communication session. All data and instructions may be obtained prior to execution of the application. Alternatively, portions of the data and instructions may be obtained dynamically and in time as needed for execution. Thus, it is not required that the data and instructions be entirely on the machine-readable medium at a particular time.

Examples of computer readable media include, but are not limited to, non-transitory recordable and non-recordable type media such as volatile and non-volatile memory devices, Read Only Memory (ROM), Random Access Memory (RAM), flash memory devices, flexible and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read only memory (CD ROM), Digital Versatile Disks (DVD), etc.), among others.

The instructions may also be embodied in digital and analog communications links for electrical, optical, acoustical or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc. However, a propagated signal, such as a carrier wave, an infrared signal, a digital signal, etc., is not a tangible machine-readable medium and cannot be configured to store instructions.

In general, a machine-readable medium includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.).

In various embodiments, hard-wired circuitry may be used in combination with software instructions to implement the techniques. Thus, the techniques are not limited to any specific combination of hardware circuitry and software nor to any particular source for the instructions executed by the data processing system.

The above description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in this disclosure do not necessarily refer to the same embodiment; and such references mean at least one.

In the foregoing specification, the disclosure has been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

1. A data storage device of a vehicle, comprising:

a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle;

a non-volatile memory configured into a plurality of partitions, the partitions each having a different set of memory operation settings configured for different types of data related to an artificial neural network, wherein the partitions include a model partition configured to store model data for the artificial neural network; and

a controller configured to operate memory units in the partitions according to the set of memory operation settings;

wherein the artificial neural network is configured to predict maintenance services for the vehicle based on the stream of sensor data.

2. The data storage device of claim 1, wherein the model data does not change during calculations to predict the maintenance service.

3. The data storage device of claim 2, wherein the model data includes neuron connectivity data for the artificial neural network.

4. The data storage device of claim 3, wherein the model data additionally includes static attributes of neurons in the artificial neural network.

5. The data storage device of claim 2, wherein the memory operation settings configure the model partition to store more than one bit per memory cell in the non-volatile memory.

6. The data storage device of claim 5, further comprising:

a neural network accelerator coupled to the controller, the neural network accelerator configured to apply inputs provided to neurons in the artificial neural network to the model data to produce outputs from neurons in the artificial neural network.

7. The data storage device of claim 6, wherein the artificial neural network includes spiking neurons.

8. The data storage device of claim 7, wherein the partition is a namespace in which logical addresses are defined; and an address mapping in the data store is configured to map the namespace to individual storage media components.

9. The data storage device of claim 8, wherein the model partition is a model namespace; the controller is configured to retrieve model data associated with neurons in the artificial neural network from the model namespace, the retrieving being addressed in a namespace that is spaced apart from the model namespace in response to inputs to or outputs from the neurons.

10. The data storage device of claim 9, wherein the controller is configured to retrieve the model data associated with the neuron from the model namespace in parallel with storing inputs to the neuron in the namespace that is separate from the model namespace.

11. The data storage device of claim 10, wherein the model data associated with the neuron includes a static attribute of the neuron and an identification of neurons connected to the neuron.

12. A vehicle, comprising:

at least one processor;

at least one sensor configured to generate a sensor data stream; and

a data storage device having:

a host interface configured to receive the sensor data stream;

13. The vehicle of claim 12, wherein the model data is updatable in the model zone during training via machine learning.

14. The vehicle of claim 13, wherein the model data includes neuron connectivity data for the artificial neural network and static attributes of neurons in the artificial neural network.

15. The vehicle of claim 13, wherein the memory operation settings configure the model partition to store three or more bits per memory cell.

16. The vehicle of claim 15, wherein the controller is configured to retrieve model data associated with neurons in the artificial neural network from the model partition, the retrieving being addressed in a partition separate from the model partition in response to an input to or an output from the neurons.

17. The vehicle of claim 16, wherein the controller is configured to retrieve the model data associated with the neuron from the model partition in parallel with storing an input to the neuron in the partition separate from the model partition.

18. A method, comprising:

configuring non-volatile memory of a data storage device into a plurality of partitions;

different sets of memory operation settings that store different types of data related to an artificial neural network, respectively for the partitions, wherein the partitions include model partitions configured to store model data for the artificial neural network;

receiving, in the data storage device, a sensor data stream from at least one sensor disposed on a vehicle;

operating memory cells in the partition according to the set of memory operation settings in response to the sensor data stream; and

predicting, using the artificial neural network, maintenance services for the vehicle based on the sensor data stream.

19. The method of claim 18, wherein the memory operation settings configure the model partition to store three or more bits per memory cell; and the method additionally comprises:

receiving a first address of an input to a neuron in the artificial neural network, the first address being in a partition separate from the model partition;

converting the first address to at least one second address of model data associated with the neuron; and

in response to the receiving the first address, retrieving, from the model partition, the model data associated with the neuron using the at least one second address.

20. The method of claim 19, further comprising:

generating an output of the neuron from the input issued to the neuron and the model data associated with the neuron; and

buffering the output of the neuron in the data storage.