JPH0683717A - Large fault-resistant nonvolatile plural port memories - Google Patents

Abstract

PURPOSE: To minimize an influence of an occurring error to realize a long-time operation free from error. CONSTITUTION: A device consists of a large-capacity semiconductor memory (DRUM) consisting of many symbol planes, triple processing cores 13, 14, and 15, and an I/O channel adapter which connects contents of processing cores to an external device, and the large-capacity semiconductor memory has a stripe constitution over plural symbol planes, and each symbol plane includes a failure confining area of the memory, and processing cores perfrom error check and correction of all fetched data to generate correction and detection code bits of data to be stored, and each processing code is provided with an ECC/voting function selection mechanism to continuously monitor three or more input links from plural symbol planes.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION This invention relates to large-volume data storage such as online transaction processing system (OLTPS).
Large semiconductor-based fault tolerant and non-volatile memory systems of the order of 4 to 128 gigabytes used for base applications, especially in error relief operation or operation and sharing of data between applications and between multiple computers in size and speed In addition, it relates to a memory system, which is a significant design criterion.

[0002]

Cross Reference This invention refers to a tee called "Nested Frame Communication Protocol" assigned to the same assignee as this application.
Relevant to the invention disclosed in Application No. 07 / 698,685 filed May 10, 1991 by B. Smith. The disclosure of Application No. 07 / 698,685 application is incorporated herein by reference.

This occurrence is also due to the invention disclosed in US Pat. No. 5,020,023, which is also assigned to the same assignee of the present application as "Automatic Vernier Synchronization of Skew Data Streams". Is also relevant. US Patent No. 5,02
The disclosure of 0,023 is also incorporated herein by reference.

[0004] BACKGROUND Semiconductor storage or memory device of the present invention as a main storage component of a conventional computer, and has been used as a storage medium of a cache disk controller. Many of the main storage devices in online transaction processing (OLTP) systems are used to buffer the data blocks of the disk storage device which are applied to be similar in nature to the cache function of the disk controller.

The buffer and cache function (hereinafter referred to as cache) intercepts a read request to the disk and supplies the requested data from the main memory buffer or the cache of the disk controller to determine the number of physical disk accesses. Can be minimized. The main memory buffer stores the computer I / O subsystem and disk storage because the read requests intercepted by the main memory buffer do not cause I / O activity or disk activity.
The load on the actuator can be considerably reduced.

A read request satisfied by the disk controller cache still requires the computer to initiate an I / O channel operation, but the actual disk actuator activity is exempt. And, since the latency for reading when served from the cache is short, the I / O channel occupancy is greatly reduced.

Each disk actuator is simply 2 per second
Reducing physical disk activity is particularly important because it can only service 0-40 random access requests (the exact number depends on the disk type and the particular access pattern). With improved processor speed and increased transaction speed and complexity,
Disks are reduced by reducing the number of physical disk accesses.
It is important to meet the economical number of actuators.

For example, if the system processes 1000 transactions per second and each transaction accesses (reads or writes) 40 data items, then 40,000 times per second if the disk subsystem is not cached or buffered. Will need to support the operation of the disk.

Given that these accesses could be spread or distributed without skew across all available disk actuators, there could be as many as 2,000 disk actuators.
Actuators will be required. In most systems the effects of skew need to be minimized. 90% of read requests can be intercepted by the main memory buffer (or buffer) or disk cache, and 15% of access requests
If only one is write, the load on the disk actuator is reduced by 75%.

In addition to reducing disk access, both of the above methods substantially reduce the latency to availability of requested data. This will be more efficient and will reduce the likelihood of collisions between transactions executing in parallel, which will reduce the level of multiple programming in the computer relatively.

It also reduces the response time of transactions, and the disk controller cache, which has a somewhat longer latency than the buffer in main memory, when disk data is shared among several computers,
It allows for straightforward sharing of data and thus has a significant benefit over main memory buffers. The buffering of main memory is not straightforward in that if disk data residing there is modified by another computer, some mechanism must be provided to invalidate the buffered data on one computer.

As the size of the buffer and cache increases, a larger portion of the read request can be intercepted. With extremely large memory,
Read requests can be satisfied almost entirely from a buffer or cache. On such systems, most of the disk activity is updates to the disk,
It is governed by the need to write all changes or additions.

This requirement to reflect all writes to a disk is a requirement for disk (magnetic) storage when comparing disk storage with cache memory (semiconductor storage) of traditionally designed main or disk controllers. Would have broader and better protective integrity properties and would be promoted.

Since the write or update activity constitutes a significant portion of the disk demand for most OLTP workloads, the size and throughput of conventional OLTP systems is at a premium even when large amounts of semiconductor memory are installed. Will be limited by the capacity and characteristics of the disk actuator that it supports.

Accordingly, the primary motivation for the present invention is to improve the integrity characteristics of large scale semiconductor base storage subsystems so that they can be used as data storage devices that do not reflect changes or updates to the disk. . Since disk data is often duplicated in other co-existing systems, its memory must be fault tolerant and non-volatile in order to effectively counter the integrity profile of the disk storage device. .

This allows the database to be stored entirely in semiconductor memory without the assistance of a disk. It is a write-back of disk-based database
Enable caching. The write-back cache, which replaces the write-through cache, can intercept both reads and writes, thus continuously reducing the number of disk accesses as memory is added. Ultimately, disk accesses can all be effectively removed when the cache becomes large enough.

The preferred embodiment according to the invention disclosed herein further allows this fault tolerant and non-volatile memory to be easily shared between multiple client computers, facilitating the construction of a large capacity transaction processing system. Explore positioning to where you can. To achieve this goal, fault-tolerant processor components are incorporated into fault-tolerant and non-volatile memory to share between several client computers for a shared cache of data and for the complete storage of the containing database. An intelligent fault tolerant non-volatile memory subsystem is provided.

Description of Prior Art Next, a specific prior art will be described. There are several fault tolerant computer designs that appear to be the closest prior art. In particular, Stratus Computer Inc. and Tandem Computer Inc. have increased the appropriate software and I / O attachments to provide the desired integrity for shared data storage and cache functions. Fault-tolerant computers that give profiles (eg Stratus X respectively
A2000 computer model and tandem integrity computer model) are manufactured and sold.

Making such an increase is a routine system integration task and is within the state of the art at the time of invention. Moreover, the FTCX computer as described in "High Performance Fault Tolerant Real-Time Computer Architecture" published in the 1986 digest of the IEEE 16th International Fault Tolerant Computer Symposium (Vienna, Austria) is a fault tolerant processor component of the present invention. It is a good prior example of the basic 3 overlap processing core technology used as.

Each of these prior art examples differs most significantly from the present invention in the design of their main memory components. The basic technique for protecting the main memory used in the prior art is a simple duplication method. In the Stratus machine and the tandem machine, both main memories are simply duplicated or duplicated, and the main memory of the FTCX is duplicated three times.

[0021]

Although such duplication is simple, it has a number of other drawbacks not present in the present invention.
The first is that it is not economical in practice. That is, the present invention may choose to reduce the overhead to protect the storage device from all failures, including failures of supporting systems such as power or clock components and failures of control or sequences among others, and Rigid (or strong) multiple symbol planes with error correction code
Synchronous parallel operation can be used.

The overhead to provide this protection in prior art systems is 100% for duplicates,
200% for triplicates. This difference is clear when compared to the 18% overhead for equivalent protection in the preferred embodiment of the invention. It can be seen that the cost of the above prior art system is dominated by the cost of the semiconductor memory, so that the present invention can save a considerable amount.

Second, the tight synchronous parallel operation of multiple symbol planes using the present invention has its own high memory bandwidth. That is, the performance of the prior art memory system is roughly equivalent to that of a single memory module of conventional design. The main memory bandwidth of the present invention is many times higher than the bandwidth of a single module.

The effective bandwidth in the preferred embodiment of the present invention is as much as 16 times the bandwidth of a single module. This combination of low cost and high performance according to the present invention would make the present invention better suited for the large scale shared memory applications described above.

Significant components of the overall memory architecture according to the present invention incorporate and use certain components known in the art.

The Basic 3 Overlap Processing Core is essentially an IEEE
E "High-performance fault-tolerant real-time computer architecture" published in the digest of the 16th International Fault-Tolerant Computer Symposium (Vienna, Austria, June 1986), and FTCS digest papers (14-19)
Page).

Connections made through a serial voting I / O channel are referred to above in US Pat. No. 5,020,023, "Automatic vernier synchronization of skew data structures", and in US patent application Ser. No. 07 referenced above. / 698,685 (Nested Frame Communication Protocol).

Communication between the tri-duplication processing core and the symbol plane is also a technique described in US Pat. No. 5,020,023 for compensating for skew between a plurality of symbol planes and the tri-duplication core. use.

The error correction code used here is IS.
Elle. U.S. patent application Ser. No. 07 entitled "Low Cost Symbol Error Correction Coding and Decoding" which is an example of a special Reed-Soloman ECC (Error Check Correction) code filed by Chen on March 6, 1989. / 318,983. It is believed that one of ordinary skill in the art will be able to select other error correction codes that, with the structure according to the present invention, may be better optimized for a particular size or application of the present invention.

The fault tolerant clock system used to provide a synchronized time reference for the system's independent symbol planes and processing rails is basically an IEEE 16th Annual International Conference on Fault Tolerant Computer Systems. As described in this paper by the inventor of the present invention, entitled "High Performance Fault Tolerant Real-Time Computer Architecture", submitted to Symposium (Vienna, Austria).

The present invention is in many ways distinguished from the known prior art. For example, U.S. Pat. No. 4,653,050 discloses means for correcting a failure of a memory module, a memory mapping mechanism for replacing the failed module, and the like. The use of ECC technology is similar to the present invention which uses error correction code to recover data lost due to the failure of a single memory module.

Error correction is not unique to any invention and is quite common in the industry. The present invention is also distinguished from U.S. Pat. No. 4,653,050 in the means of correcting or allowing a broad class of control or sequence failures or faults. This is described in detail below in accordance with the present invention.
CC / Voting function (voter) Specified by the selection circuit.

The present invention also relates to US Pat. No. 4,653,05.
Using a more robust (different) point-to-point connection topology than that described in No. 0, and using a shared bus topology with multiple single points of failure in the connection according to the description of US Pat. No. 4,653,050. In contrast, the interconnection mechanism according to the present invention can tolerate or withstand any single point of failure.

The object of the invention Accordingly, a first object of the present invention, in particular, provides a very large highly reliable non-volatile semiconductor memory system suitable for use as a central storage facility for large-line transaction processing system and others That is.

Furthermore, it is an object of the invention to provide, in essence, a memory system as described above which enables long error free operations.

Another object of the invention is the use of wide error correction and detection codes in the mass memory array itself.
A system capable of achieving error relief operations through three modular duplications in many control and communication modules, and the disciplined use of fault containment areas or partitions to prevent unwanted propagation of errors caused by faults. Is to provide.

Still another object of the present invention is that the application of the error correction code not only guarantees the error release data contents from the mass memory array, but also in the control circuit closely related to the mass memory. To provide a means for detecting and correcting failures or faults in a memory system.

[0038]

In order to solve the above problems and achieve the object, the present invention is configured as described below. Then, the details will be described later with reference to the description of the embodiments and the drawings, which will be made clear.

The present invention is broadly accomplished by its design by designing a large high reliability semiconductor data storage system ideally composed of three unique components. The first is a large capacity semiconductor memory array (DRAM),
The second is an optimally tripled processing core, and the third is a plurality of channel adapters that connect memory to external devices.

Each of these components is partitioned into a plurality of fault containment areas adapted to surround the fault in a particular fault containment area in which the fault has occurred. The mass memory is striped or connected across multiple symbol planes, each symbol plane including a mass memory fault containment area, and each symbol plane containing at least one designated memory word accessed by the system. Remember the bits.

The processing core error-checks and corrects all data fetched from memory (EC
C), and error detection and correction means for generating correction and detection code bits for all data to be stored in memory. Each symbol plane includes means for generating a FETCH-RESPONSE control field that uniquely identifies that data as a response prior to fetching the data from memory.

An ECC / voting function selection mechanism is provided in each processing core, and at its input, fetch response command
Continuously monitor more than two input links from multiple symbol planes connected to the processing cores that identify the fields. A majority voting mechanism is used to determine if the majority of the incoming input links being monitored carry a fetch-response command field.

Error correction / correct all data fields that follow from all active symbol planes of the memory array.
Activating the appropriate switch means for processing through the detection circuit. If the ECC / voting function selection mechanism detects that one or more fetch-response fields do not contain an appropriate fetch-response command when a majority vote of inputs is made, the operation in the control circuit of the defective symbol plane is When an error occurs, it is flagged for subsequent diagnostic testing.

Furthermore, according to one aspect of the invention, the processing core can obtain greater reliability of the system by trimodular redundancy (TMR). An I / O channel adapter connected to the communication hardware,
Or by the symbol plane of the mass memory,
Correct operation of the processing core can be guaranteed by TMR checking all transmissions received from the processing core.
Also, the use of "vernier skew correction" throughout the system allows for significantly better error relief output at high data rates.

[0045]

Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. First, the outline of the present invention will be described.
The present invention is an important component for fault-tolerant and non-volatile large-capacity semiconductor memory devices. Such storage is shared by multiple computers and is used as a file of data, catalogs and / or other permanent data storage in place of conventional disk storage to minimize system performance requirements on the disk subsystem. It can be used for write-back cache on the disk.

Due to the significant difference in performance between semiconductor memory and disk, the combined performance of large computers using such memory systems can be dramatically better than that of disk-based equivalent systems.
Neither storage performance nor data integrity is affected by either a single point of storage failure or possible multiple failures. The reliability of the storage or storage device can be at least as good as that of a fully duplicated disk storage device.

Next, the present invention will be described with reference to FIG. FIG. 1 is a diagram showing the overall organization of a storage device composed of four main subsystems. Each of these subsystems: An I / O channel adapter subsystem typically consisting of several independent channel adapters, each channel being a separate channel to the CPU of each client,

2. Three independent identical processing rails (r
aile), three overlapping control or processing cores,

3. A large memory subsystem consisting of multiple independent symbol planes operating in tight clock synchronization with each other;

4. And a dual power system consisting of two independent AC-DC converters providing coarsely regulated DC power to two primary DC distribution buses.

Each of these subsystems comprises several fault containment regions (FCRs). An FCR, simply stated, is a predefined block of a circuit, designed to contain the physical effects of internal faults in that block and not be physically affected by faults in other FCRs. The size and the partitioning of the system into FCRs are shown in FIG. 1 for each type of FCR in the system, with each FCR surrounded by a dotted line.

The four types of FCR are the same as the above-mentioned four main subsystems. That is, 1. I / O channel adapter FCR. The system has one I / O channel adapter per installed I / O channel.

2. Processing rail FCR. In the preferred embodiment of the system, there are three processing rails.

3. Symbol plane FCR. In the embodiment according to the invention, there are 19 symbol planes, but this number can vary depending on the function of the error correction code used in the particular embodiment.

4. Primary power converter FCR. There are two primary power converters in this system.

All logical FCRs of the system (not applicable to power supplies) operate with tight clock synchronization to each other and each FC
R is broadcast from a 3-duplication clock system, a portion of which is embedded in each rail of the 3-duplication processing core.
It serves to individually derive local clock signals from the overlapping clock signals.

All communication between these individual FCRs is via a dedicated point-to-point link, and vernier skew compensation is used to eliminate the effects of inherent clock skew between the elements. Although only a few dedicated communication links are shown in FIG. 1 to avoid obscuring the invention, each FCR in the system is connected to each rail of a three-rail processing core by a dedicated point-to-point link.

That is, each rail 13, 1 of the processing core
Dedicated links 10, 1 between 4, 15 and each symbol plane
1 and 12, each rail 13, 14 and 15 of the processing core
, A dedicated link 16, between each channel adapter
There are 17, 18 and there are dedicated links (not shown) that connect all three rails of the processing core to each other. There are no links connecting the symbol planes to each other, and no links connecting the channel adapters to each other.

Moreover, each FCR draws regulated power from a dedicated DC-DC regulator that is part of that FCR and is individually powered. The DC-DC regulator can provide regulated power to the FCR as long as power is maintained on at least one of the two primary DC power buses.

The storage device is a customer or client connected through an individual I / O channel adapter 19.
It provides services to computers and is shared by each of them. Each channel adapter 19 is simplex. That is, there are no overlapping elements and they operate independently. If a duplicate connection is desired between the storage device and the client computer, the computer can connect to the storage device via multiple channels.

The channel adapter replicates the input data stream and duplicates the rails 13, 1 of the triplication processing core.
It serves to distribute the same duplicate to each of the four and fifteen. It also functions to create a single output data stream to vote or multiple trips from a triple duplication processing core from a channel adapter.

This voting function masks (and detects) error transmissions from one of the system's processor rails. In this preferred embodiment, each channel is a U.S. patent application Ser. No. 07 / 698,68 referenced above.
Process the serial data stream at 100 megabytes per second using the nested link protocol as described in No. 5.

The characteristics of this channel protocol are not central to the invention and can be interchanged with many other channel protocols, such as the IBM ESCON fiber optic serial channel protocol can be used. it can. For performance and data latency reasons,
And to take full advantage of semiconductor memory, the channel operates at the highest data rate possible,
It is desirable to optimize with the least latency.

The message between the data storage device and the client computer is a triple duplication processing core 13,1.
4,15. Each rail of this processing core operates in tight clock synchronization with the other two rails and performs the same function for the same data in each rail. A single rail fault that causes an error transmission from that rail can be masked by the receive FCR of the voting circuit.

The design and operation of the three-overlap processing core is similar to that described in the aforementioned publication that referred to the FTCX computer with increased and improved vernier skew compensation. Vernier skew compensation enables high bandwidth transmission between the rails of the core and between the core and the surrounding FCR and high speed operation of the core. This is because skew otherwise limits bandwidth and operating speed. Therefore,
It can be modified to increase both bandwidth and operating speed.

The skew compensation module (skew circuit) in this embodiment is arranged at the receiving ends of all links.
They would also be placed in the receive circuit for each memory port in each symbol plane, as well as in the receive circuit for each link in each memory port controller of the processing core.

Further, a skew circuit is provided in the receiving circuit of each channel controller in each processing core rail and in the receiving circuit of each channel adapter.

Further, in FIG. 1, a link including a skew compensation module as a part of those receiving circuits is shown by being marked with an enlarged dot attached on the link. These skew modules are typically physically located within a functional block that acts as part of the block's receive circuitry. Each parallel data link is shown as a single line, each link 10, 11, 12 is 9 bits wide (8 data bits and 1 control bit), and each link 16, 17, 18
Is 128 bits wide (8 bits bytes x 16).

Under these conditions, the present invention provides parallel operation and control of multiple symbol planes FCR by the memory port controller of the processing core. The data can be reconstructed in multiple ways such that the data lost due to the failure of a single symbol plane can be reconstructed by the voting means in the memory port control circuit of the processing core and the combination means with the error correction circuit. Data can be stored economically by stripes (or connected in stripes) across symbol planes.

The parallel operation of multiple symbol planes is also one with inherently high memory bandwidth. 2 and 3 illustrate means for striping data across multiple symbol planes. In the preferred embodiment, 16 symbol planes are used to store data symbols and 3 symbol planes are used to store error correction symbols.

Further, in this embodiment, each symbol is 8 bits.
It is a data byte. In fact, each 16 bytes (128
A (bit) data word is striped across 16 symbol planes to store one byte in each of the 16 data planes. In addition, the memory port controller
3 error correction symbols per 16 byte word (EC
S) and strip them across the three ECS planes and store them when storing the data. The connection between the memory port controllers on the rails of the processing core and each symbol plane is provided by a dedicated point-to-point serial link.

FIG. 4 shows a port controller format for symbol plane commands and data streams. This data stream can be regarded as consisting of a serial stream of 9-bit control / data symbols. Bit 0 of that symbol indicates that the control / data symbol is either a control symbol or a data symbol. Bits 1-8 of the symbol are either data bytes (bit 0 = 0) or control bytes (bit 0 = 1).

The idle (IDLE) control symbol is transmitted between frames. First of frame for a simple STORE request
The symbol contains a storage control code, followed by four data symbols containing the address of the word in which to store the data, followed by the actual data to be stored. The length of the data block to be stored is variable and the end of the data block is limited by an idle (IDLE) symbol or other control symbol that marks the beginning of a new frame.

Serial data in this preferred embodiment
The transmission rate of the stream is 25 million symbols per second.
The transmission of control symbols is the same as the transmission of the address field for each symbol plane. That is, these fields for all 19 transmissions for each symbol plane are the same.

And since all symbol planes operate exactly in synchronization with each other, they must all receive the same request as far as the required operation is concerned. Therefore, the effective transmission rate of the command portion of the frame is the basic symbol transmission rate per link. In this example, it is 25 million symbols per second.

Since different data is stored in each symbol plane, the effective data rate for one transmission of the data field is that each symbol plane only receives a unique copy of the data to be stored in that plane. , Even faster. In this embodiment with 16 data planes, the effective rate of data transmission to the high capacity symbol plane memory subsystem is 40,000 per second.
10,000 data bytes (75 million ECS bytes per second for the +3 ECS plane).

The transmission bandwidth for the individual symbol planes is still only 25 million data bytes per second. In this embodiment with a triple redundant processing core, each symbol plane receives a triple redundant copy of each command and data frame. It detects and masks erroneous transmissions from one of the rails, thus voting 3 duplicate transmissions on a symbol basis for that symbol (either command or data).

The processing cores in this preferred embodiment range from 1 to 4 depending on the number of installed I / O channels and letters.
A memory port controller can be located. Each symbol plane can be configured to support 1 to 4 independent ports to match their processor configuration.

Each port is similar, each of which is a duplicate of the single port just described and contains three duplicate port controllers in three duplicate processing rails, with each symbol plane from each port controller. Contains a dedicated link to the memory port.

FIG. 5 shows the format of a fetch (FETCH) request. The first 4 symbols are the data symbols, containing the address of the first 16 byte word to be fetched,
Followed by two data symbols including the block size.
This fetch request is implicit and the data symbol following the idle (IDLE) is the first address byte of the implicit fetch.

This reduces the fetch latency by one cycle. If the fetch should immediately follow another frame, then that frame is delimited by inserting an explicit fetch control symbol before the fetch address. Also, the address is sent before the block size.

This starts a memory cycle (assuming that the memory is idle at this time), so that as soon as the necessary information is available, the fetch
It is possible to start the operation and reduce the latency of the fetch. The response format from the symbol plane to the memory port controller is shown in Figure 6.
Shown in. Like the fetch command format, the response format is optimized for good fetch latency,
The symbol plane simply initiates the transmission of data blocks.

If the first symbol of the response frame following idle is a data symbol, this is an implicit fetch-response (FETCH-
RESPONSE) frame. Like the fetch request frame, this response frame is separated in two by an explicit fetch-response control code if it should immediately follow the preceding response transmission.

FIG. 7 illustrates the structure and data flow of the receiver of the processing core port controller which is a positively significant feature of the present invention. This is a mechanism called the ECC / voting function selection mechanism that implements the means of correcting both the symbol plane data and control or sequence information failures. That is, the data failure is corrected by applying the error correction code, and the failure of the control or sequence information is corrected by the majority voting logic.

The ECC / voting function selection mechanism is exempt from the application of error correction codes to the data and the failure of one link or symbol plane that interconnects the port controller and the memory port of the symbol plane. A means for selectively switching between a majority vote for sequence information is provided. This means will be described later in detail.
Such a mechanism is disposed on each rail of the tripled processing core of this preferred embodiment.

It should also be noted that the symbol plane can be multi-ported, which is accomplished by having a match-only memory port controller in the processing core for each memory port. In the preferred embodiment, there are four memory ports.
Therefore, a total of 12 ports are arranged on each rail, which is a total of 12 ports.

Moreover, in the above means, each processing core is TM
Whether R or not, it is fully operational and effective against symbol plane or link failures.

More specifically, since the symbol planes operate in tight synchronization with each other, the port controller will receive a hard synchronization response from all of the symbol planes simultaneously in response to a fetch request. They are processed through the ECC / voting function select circuit and all symbol plane failures are masked or corrected.

FIG. 5 illustrates the structure of this ECC / voting function selection circuit. Response control or sequence information symbol is 1
Although always the same across all 9 parallel data streams, the data symbols being fetched are always different for each symbol plane. A subset of symbol plane transmissions, one bit, is used to determine if a symbol is data or control information.

In this preferred embodiment, 3 ECS
Control / data symbols from the symbol plane are vote functions 50
Are individually examined in order to determine whether they are control or sequence information symbols. Voting between these three symbol planes is performed for a single bit in the voting function 50 to determine whether the symbol is a command or a data symbol.
When it is determined that the symbol is such a command symbol, the voting function 51 makes the next vote and determines the symbol.

Actually, the mechanism of the voting function 51 is 3 ECS.
Voting between the symbol planes constitutes the command stream, which is used to drive the receive state machine 52 of the port controller. The control stream of this visible configuration is all command symbols (idle, fetch-reply, etc ...) Received, as well as a single pseudo control symbol, data that can be replaced by any data symbol in the received data stream.

Since this command stream is constructed by voting between 3 symbol planes and it is assumed that no more than one symbol plane can fail at a time, a command stream of this configuration will have no single plane control. It is corrected despite the failure of the control circuit due to the failure. If the failed symbol plane is an ECS plane, it can win by voting (or majority). A failure in any data plane does not destroy the constructed command stream because it has no role in its construction.

The symbols from the 16 data planes can be additionally compared with the command stream constructed to detect and report all data symbol plane failures for the sake of clarity, but this is the subject of the present invention. It is not central to the main function and is excluded from explanation. The choice of which 3-symbol plane is used for the construction (voting) of the response control stream is arbitrary and the choice of the 3ECS plane is made on a sensory basis in this preferred embodiment.

The reception state machine 52 of the port controller is commanded by the outputs of the voting function 51 and the multiplexer (MUX) 55, and the error correction circuit 53 or the data selection circuit 5 is sent.
4 is responsible for outputting the data. In the case of a fetch, the port controller waits for response data, starting with the idle or fetch response code, starting with the first data symbol 19
The byte-width data stream is output to the error correction circuit 53.

This error correction circuit uses the input 16 data
Bytes and 16 bytes extracted from 3 ECS bytes
A data word is output every cycle. The error correction code implemented in this preferred embodiment is capable of completely correcting any missing or erroneous bytes (either single data bytes or single ECS bytes) and is capable of detecting any two-byte failure. it can.

The error correction circuit is selectively enabled or disabled. That is because not all symbol positions in the input data stream contain data striped from the symbol plane. This selective enabling and disabling of the inputs to the error correction circuit is
It is the responsibility of the receive state machine and associated circuitry of the port controller, including the voting function selection mechanism.

In Fetch-Response, the ECC circuit is disabled or idle between the receive cycles of the Fetch-Response command, for data symbol streams that continue until the end of a transmission marked with an Idle or Response command symbol for the next transmission. Are available or active. The ECC circuit is disabled when receiving this idle or command symbol.

In addition to explicit store or store and fetch (FETCH) symbol plane request / response operations, the port controller also fetches some symbol plane status information from the symbol plane and the configuration for the symbol plane. And a mechanism for storing control information.

The status information differs between the often-used planes. For example, this preferred embodiment uses conventional error correction mechanisms inside the symbol plane to mask soft errors in the symbol plane memory array DRAM (most often caused by alpha particle emission). The error is corrected and the error event is recorded in the symbol plane status array within the symbol plane.

Memory from the symbol plane to the processing rail
This type of error event is invisible from outside the symbol plane because the data sent to the port controller has already been modified. Also, it is unlikely that two symbol planes will have the same soft error at the same time, so it is unique to the symbol plane in which the error occurred.

The port controller implements the functionality that allows fetching this internal status array from the symbol plane. The port controller performs a status fetch from the symbol plane as follows. It is the situation-fetch (STAT
US-FETCH) command to the target symbol plane, and DUMMY-STATUS-FETCH to other symbol planes in parallel.

This allows the dummy operation to be performed in parallel with the real situation-fetch operation of the target symbol plane in order to maintain a tight synchronization between all symbol planes for that dummy-state-fetch. To do. Therefore, all symbol planes are
TATUS-FETCH-REPLY) but only the target symbol plane returns status data. The other symbol planes return 0s in the data field to maintain synchronous operation.

The port controller receive status machine 52 directs parallel transmissions from all symbol planes to the data selector 54,
That selector must be controlled to select the transmission from the target symbol plane. The identity of the target symbol plane is known to the port controller because that symbol plane is sending requests and waiting for responses.

The port controller constitutes an internal symbol plane circuit and executes a control-store (CONTROL-STORE) command for controlling and a dummy-control-store (DUMMY-CONTROL-STORE) command. These are the symbols
Various error status indicators, such as might have been generated by a drum (DRAM) error, can be used to reset for a routine reconfigure command.

A good example is changing and adjusting the size and configuration of the memory installed in the symbol plane.
In the preferred embodiment, the control-memory frame is also used to synchronize the refresh activity of the DRAM across all symbol planes during power-up initialization.

It is not possible for the port controller to predict the actual response time for every request for the symbol plane. Therefore, the processing of the response command symbol performed via the response command voting function is the center of the present invention. The timing of the fetch response depends on the impact of many interferences such as drum (DRAM) refresh, or interference from other ports. The port receiving circuit relies on this voting function to extract the necessary sequence information and drive the control status machine.

In the case of status-fetch (STATUS-FETCH),
Sequence information is most often derived from transmissions from the symbol planes that are actually performing the dummy-status-fetch operation. In particular, in this preferred embodiment, when the status-fetch is output from the data plane, the sequence information is only extracted from the symbol plane which performs the dummy-status-fetch operation.

Extensions to the Basic Invention Certain optimizations and features may be added to the underlying context of the invention to optimize the implementation for a particular application. In the following, three specific invention extensions will be described. 1 is symbol plane sharing, 2 is exchange error correction code, and 3 is symbol plane error removal or scrub which facilitates parallel repair and upgrade of symbol plane memories.

It is truly possible to have a means of doing an electronic repair of a system with a pre- failed symbol plane. This situation is
It can occur most often when the symbol plane memory system is very large, and when the system is so far away that it may be desired to defer repair.

This is to replace the failed plane,
This can be facilitated by providing one or more additional spare symbol planes that can be electronically exchanged. In the preferred embodiment, when provided with a single spare plane, this plane constituted the 20th plane (spare). This plane can be exchanged for any of the other 19 symbol planes.

FIG. 8 illustrates a preferred embodiment that extends the present invention to enable electronic repair. A single 19: 1 multiplexer 60 for transmission on the symbol plane selects one of 19 parallel port controller transmissions for transmission on the spare symbol plane.

This multiplexer is constituted by a maintenance process (software) running in the processing core and supports a duplicate transmission of the transmissions for the exchanged symbol planes for additional transmissions to the spare planes. For example,
If plane 7 is to be replaced, the multiplexer is arranged to switch the transmission for plane 7 and select the transmission for the spare plane.

The port controller further includes nineteen 2: 1 multiplexers 62. These multiplexers are typically configured to transmit from their respective symbol planes to the port controller. Transmission from the spare symbol plane is switched to transmission from the fault symbol plane when electronic repair is required. For example, if plane 7 is to be replaced, the 2: 1 multiplexer for plane 7 is configured to replace the signal received from the spare symbol plane with the received signal from plane 7.

Therefore, the electronic processing means should consist of a single 19: 1 multiplexer 60 and one of 19 2: 1 multiplexers 62 for electronically exchanging the failed symbol plane with a spare plane. The contents of the spare symbol plane can be loaded by reading the entire contents of the mass memory from the symbol plane array and writing it back to the symbol plane memory.

During the read operation, the missing information from the failed symbol plane is reconstructed by the error correction circuitry of the port controller. This reconstructed data is written to the spare plane throughout the write cycle. At the end of this error removal operation, the spare plane is loaded with the reconstructed data that was lost when the original symbol plane failed. This error elimination operation can be performed by a simple program that loops all reads and writes of memory, or it can be hardware assisted as described below.

Exchange Error Correction Code This preferred embodiment is the S. Elle. "Single byte error correction / double byte correction" as described in Chen, U.S. patent application Ser. No. 07 / 318,983.
Error detection error correction code "is used. The replacement error correction code can be used to optimize different applications.

For example, the double error detection function of this code requests some applications (such that the exposure is only until repair is done) when the exposure to the double error is short. It is not possible. The two-error correction symbol is only required for single-byte error correction without double-byte error detection. This reduces duplication overhead from 18% to 12% and is more economical.

Hardware Auxiliary Memory Error Removal Many of these configurations change the requirement that all of the memory be error removed, that is, the entire contents of the memory be read from the symbol plane and rewritten. can do. The above procedure example of exchanging with a spare symbol plane is only one example of a number of situations where the need for error elimination may occur.

Simultaneous repairs and additions to memory also require error removal operations. The time to error clear from memory may be very long for large memory if the memory is driven entirely by program execution of fetch / store loops.
This fetch / store loop fetches words to prevent otherwise intervening normal stores or stores from being overwritten by the store portion of fetch / store operations. /
Because it must be locked during the store, it can arbitrate with the operation of other software on the machine.

The symbol plane and memory port controller in this preferred embodiment uses unused memory bandwidth and includes hardware aids to perform high speed software transparent error elimination of memory. Hardware error elimination is initiated in parallel by a CONTROL-STORE command for all symbol planes. The starting address and length of the memory range to be error-removed is stored in the special error-removal control register of each symbol plane.

The symbol plane then initiates and controls error removal. Each word is fetched from the symbol plane and sent to the memory port controller. FIG. 9 shows the format. When the memory port controller receives the error elimination word or word stream, it sends the word through the error correction circuit and rewrites it in the symbol plane.

FIG. 10 is a diagram showing a format for retransmission on the symbol plane. When the symbol plane receives the error-corrected data, it overwrites the previous contents of that memory location and effectively reloads the contents of that word with the corrected data. Error-rejected transmissions in either direction can be overridden at any time to allow normal memory traffic, and access to the memory array itself in the symbol plane is on a priority basis to normal memory traffic. First, it is allowed.

Therefore, the effect of error elimination on the normal use of the memory is minimal. Since there is some pipeline latency between when the word was first read by the error elimination mechanism and when it was rewritten,
Normal memory traffic is monitored by the symbol plane during this latency period to determine if an attempt was made to store to a memory location that was erroneously removed.

Although an attempt was made to write to a memory location read by the error elimination hardware,
If it has not been rewritten, normal writing proceeds and the rewriting of the error-removed data for that location is canceled. That is, the error removal hardware supports the error removal from that point, saves, and discards the pipeline data.

The effect of the pipeline sending to the memory port controller through the error correction circuit and returning from the port controller to the symbol plane is that several error elimination words can be in transit at any given time. It should be noted that it does. Memory location conflicts between normal memory activity and error elimination are extremely rare, so the error elimination input does not affect error elimination performance.

The hardware-assisted error elimination in this preferred embodiment is well pipelined in the background and can proceed at full port bandwidth when no other memory activity is present on the port. this is,
It enables error removal of the entire memory at near 400 megabytes per second with minimal impact on the performance of the system's normal software running.

It should be noted that the memory required to support hardware assisted error removal of the memory in the background
The control of the port controller and the addition of the data path structure can be easily understood.

Although the embodiments of the present invention have been described in detail above, it is not intended to limit the present invention thereto alone, and many changes and modifications can be made without departing from the spirit of the present invention. . And it can be scaled to achieve more reliability, speed, and versatility.

[0129]

According to the present invention, by configuring as described above, the processing core is duplicated in three modules to reduce the probability of reprocessing due to an error, and the error occurring in each module and storage plane is reduced. It was possible to achieve very long error freeing operations by using wide error correction and detection code in the memory array itself, as well as containing it in a non-intrusive manner.

[Brief description of drawings]

FIG. 1 is a high-level organizational chart showing the main functional components according to the present invention.

FIG. 2 is an explanatory diagram showing a connection between a processing core / memory port controller (processing core) and a symbol plane for a storage / fetch operation.

FIG. 3 is an explanatory diagram showing a connection between a processing core / memory port controller (processing core) and a symbol plane for a storage / fetch operation.

FIG. 4 is an explanatory diagram showing a store or store (STORE) command transmission format of the port control device.

FIG. 5 is an explanatory diagram illustrating a fetch (FETCH) command transmission format of a port control device.

[Figure 6] Fetch response of the symbol plane (FETCH RESPONSE)
Explanatory diagram illustrating a transmission format

FIG. 7 is a connection diagram showing a control and data path structure of a memory / port control device generally including an ECC / voting function selection circuit.

FIG. 8 is an illustration showing an increase in control and data path structure of a memory port controller required to support the exchange of spare symbol planes.

FIG. 9 is an explanatory diagram showing an error removal transmission format from the symbol plane to the port control device.

FIG. 10 is an explanatory diagram showing an error elimination transmission format from a port controller to a symbol plane.

[Explanation of symbols]

 10-12, 16-18 Dedicated link 13-15 Processing core rail 19 I / O channel 50,51 Voting function 52 Port controller state machine 53 Error correction circuit 54,55 Multiplexer 60 19: 1 Multiplexer 62 2: 1 Multiplexer

Claims (9)

[Claims] 1. A memory function is striped across a plurality of symbol planes and includes a memory fault containment area,
Each said memory fault containment area has at least one designated memory word accessed in the storage system.
What is claimed is: 1. A large fault tolerant semiconductor data storage (memory) system including the symbol plane for storing bits, the storage system comprising at least one memory port controller for accessing the plurality of symbol planes in parallel. A memory core controller including a processing core, wherein the memory port controller inspects all data fetched from the memory for errors and generates error correction / detection code bits for all data to be stored in memory.
A detection mechanism, each said symbol plane including means for generating fetch-response sequence information that signals fetched from memory has arrived at the memory port controller and signaling to the requesting processor and the link, said sequence information being Generated on each symbol plane, the storage system further continuously monitors a plurality of input links from a plurality of signal planes to the processing core to identify fetch-response sequence information input to each processing core. An ECC / voting function selection mechanism, the ECC / voting function selection mechanism including a majority voting means, determining that a majority of the monitored input links include the fetch-response sequence information, the storage system further comprising: , All data appearing on the input link from the symbol plane in response to the discrimination Means for enabling error detection / correction circuit means of the processing core to check for errors after receipt of the fetch-response sequence information for which the error detection circuit is disabled until the sequence information is detected. Large-scale fault-tolerant highly reliable semiconductor data storage system characterized by being maintained in a state. 2. The completed processing cores are overlapped three to form three processing core rails, each connected on one side to all of the symbol planes and to the data link and on the other side said A rail connected to the data link between the processing core and the channel adapter for communicating with other components connected to the memory system, each between itself and each of the three core rails. A symbol plane and channel adapter having a data path and further including voting means in the three data paths for selecting a "majority voting" output for the data and control information received from the three processing rails. The large-scale fault-tolerant high-reliability semiconductor data storage system according to claim 1. 3. The processing core is directly connected from all of the symbol planes to its inputs,
Only one selectable multiplexer switch can be selected as a data input, and means for causing all unselected symbol planes to create dummy data even if the symbol plane data is not selected. The large-scale fault-tolerant high-reliability semiconductor data storage system according to claim 1. 4. A plurality of parallel bit data links between each said symbol plane and each processing core comprises means for vernier skew adjustment, all said vernier skew adjustment means being a single master system system. Clocked and all data transmitted between the processing core and the symbol plane of the data storage system synchronized and aligned with the master system clock to be better than a predetermined maximum skew tolerance, The large-scale fault-tolerant high-reliability semiconductor data storage system according to claim 1, characterized in that a skew not less than the maximum value provided by the vernier skew adjusting means is compensated. 5. The storage system comprises at least one spare symbol plane in the memory and a switch circuit operable under the control of a diagnostic circuit to form a data path between the spare symbol plane and a processing core input. And that the spare symbol plane receives all data destined for it and normally returns all data that would have come from the symbol plane designated as "unavailable" by the diagnostic circuitry. The large-scale fault-tolerant high-reliability semiconductor data storage system according to claim 1. 6. Each of the symbol planes has a plurality of ports, the same number of processing core port controllers as the ports of the symbol planes are provided, and each of the processing core port controllers has a channel for each symbol plane. Link means for connecting to a predetermined port of the device, a completed ECC / voting function selection mechanism, an error detection / correction mechanism, and means for activating the error detection / correction mechanism after receiving a fetch-response control signal. The method according to claim 1, wherein
The described large-scale fault-tolerant highly reliable semiconductor data storage system. 7. A large fault tolerant high reliability semiconductor data store that stripes memory functions across multiple symbol planes, each symbol plane storing at least one bit of a designated memory word accessed in a storage system. A system, wherein the storage system selectively connects memory to a high speed communication link via the channel adapter to communicate with at least one memory entity attached to another functional entity attached to the data storage system. A processing core module including a port control function and a channel port control function, wherein the processing core error-checks all data fetched from the memory, and an error correction and detection code for all data to be stored in the memory. Includes error correction / detection mechanism to generate bits , Each said symbol plane is fetched from memory and comprises means for generating fetch-response sequence information which must precede the data returned to the processing core, said sequence information being generated in each symbol plane, said storage system , Further, in each said processing core,
Fetches from multiple signal planes-ECs that continuously monitor multiple input links to identify response sequence information
C / voting function selection mechanism, said selection mechanism comprising majority voting means for determining that a majority of the monitored input links include such sequence information field, said storage system further comprising fetch-response sequence information In a large fault tolerant high reliability semiconductor data storage system including switch means for enabling an error detection / correction mechanism of the processing core to check for errors in all data appearing on the input link from the symbol plane after reception. A method of ensuring a corrective operation in a control circuit embedded in the symbol plane, comprising: 1. Determine that the sequence information is on multiple supervisory input links from the symbol plane, 2. Determine that the same sequence information was detected on the majority of input links, A method of guaranteeing corrective operation, comprising the steps of causing the error detection / correction circuit to process all subsequent data transmissions to the processing core. 8. The method of guaranteeing corrective operation of claim 7, wherein the method of guaranteeing corrective operation includes an error notification in the sequence information for the diagnostic module if one of the monitored links does not match the other. 9. The modified operation guarantee method supplies a control-store command that supplies a start address and the number of words to be error-removed to all symbol planes, to a symbol plane port controller of the processing core. To the symbol plane transmitting sequential words to the symbol plane, and the symbol plane port controller recognizes the error elimination command using the ECC / voting function selection mechanism and recognizes from the other symbol planes. Reconstructing missing data from uninitialized or improperly loaded symbol planes by using data and ECC symbols, the symbol plane port controller recalculating the ECC symbols and modifying the data and The recalculated ECC symbol is returned to the symbol plane port, where the symbol plane is the data and ECC. Modify operation assurance method of claim 8 wherein the overwriting and storing the error symbols, characterized in that it comprises the steps of No..