WO2022105448A1

WO2022105448A1 - Method and system for maintaining pcie signal connection by using in-place signal, device, and medium

Info

Publication number: WO2022105448A1
Application number: PCT/CN2021/121429
Authority: WO
Inventors: 王凌骏
Original assignee: 苏州浪潮智能科技有限公司
Priority date: 2020-11-20
Filing date: 2021-09-28
Publication date: 2022-05-27
Also published as: CN112416828A

Abstract

A method and system for maintaining a PCIE signal connection by using an in-place signal, a device, and a medium. The method comprises: in response to interruption of a PCIE link between a master controller and an expansion cabinet, determining, on the basis of the in-place signal of each port of the expansion cabinet, whether cables between the master controller and the expansion cabinet are all in place (S1); in response to the cables between the master controller and the expansion cabinet being all in place, sending the in-place signal to a master control chip of the master controller (S2); in response to the master control chip receiving the in-place signal, releasing a reset signal and releasing a reset state of the PCIE link of an uplink port of the master control chip (S3); and in response to finishing the reset state of the PCIE link of the uplink port of the master control chip, communicating the PCIE link to enable the master controller and the expansion cabinet to be in a connection state (S4). The method sends the in-place signal to the master control chip of the master controller, and releases the reset signal and releases the reset state of the PCIE link of the uplink port of the master control chip, thereby achieving the purpose of maintaining the connection between the master controller and the expansion cabinet, and improving the product reliability.

Description

Method, system, device and medium for maintaining PCIE signal connection by using in-position signal

This application claims the priority of the Chinese patent application filed on November 20, 2020 with the Chinese Patent Office, the application number is 202011311071.5, and the invention title is "Method, System, Device and Medium for Maintaining PCIE Signal Connection Using In-Place Signals", which The entire contents of this application are incorporated by reference.

technical field

The present application relates to the field of servers, and more particularly, to a method, a system, a computer device and a readable medium for maintaining a PCIE signal connection by using an in-position signal.

Background technique

With the increasing number of Internet companies, more and more industries are involved. For example, the e-commerce industry, short video and micro video industry, online live broadcast industry, online movies and online music, etc., have all moved physical products to the Internet. Now people can enjoy these products anytime and anywhere as long as they have a mobile phone. These virtual things all exist in large storage servers. The storage server processes various problems through CPU computing, and then stores the information on the mounted hard disk. In some storage servers, the CPU motherboard and the hard disk are together, and in some storage servers, the CPU motherboard and the hard disk are two controllers, that is, the CPU and the hard disk are not on the same motherboard, and can be connected through a network or a cable.

Some special applications may require PCIe devices to run continuously and uninterruptedly with high reliability. For this reason, the PCIe bus adopts Hot Plug technology to realize the replacement of PCIe card devices without turning off the system power. During the normal operation of the equipment, various problems will occur, including cable damage or the operator inserting the wrong cable. Such problems will cause the main controller and the expansion cabinet to be disconnected at the corresponding nodes, resulting in control The controller changes from dual control to single control, which seriously affects product performance. However, closing the main control cabinet and the expansion cabinet to replace cables will not only delay time, but also affect normal business interaction.

SUMMARY OF THE INVENTION

In view of this, the purpose of the embodiments of the present application is to propose a method, system, computer equipment and computer-readable storage medium for maintaining a PCIE signal connection by utilizing an in-position signal, by sending an in-position signal to the main control chip of the main controller; And release the reset signal and release the reset state of the PCIE link of the upstream port of the main control chip, realize the purpose of maintaining the connection between the main controller and the expansion cabinet, and improve the reliability of the product.

Based on the above purpose, an aspect of the embodiments of the present application provides a method for maintaining a PCIE signal connection by using an in-position signal, including the following steps: in response to the interruption of the PCIE link between the main controller and the expansion cabinet, based on each expansion cabinet The presence signal of each port judges whether the cables between the main controller and the expansion cabinet are all in place; in response to all the cables between the main controller and the expansion cabinet are in place, the The main control chip of the main controller sends the in-position signal; in response to the main control chip receiving the in-position signal, the reset signal is released and the reset state of the PCIE link of the upstream port of the main control chip is released; and in response After the PCIE link of the upstream port of the main control chip ends in a reset state, the PCIE link is connected so that the main controller and the expansion cabinet are in a connected state.

In some embodiments, the determining whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet includes: acquiring the presence signal of each port of the expansion cabinet , and perform AND operation with other in-position signals after inversion of each in-position signal to determine whether the result is a preset value.

In some embodiments, the method further includes: in response to the cables between the main controller and the expansion cabinet not being all in place, resetting the PCIE signal of the main control chip of the expansion cabinet.

In some embodiments, the method further includes: in response to the main control cabinet and the expansion cabinet being in a connected state, detecting whether a sequence of devices with all ports exists in the sequence list; and in response to the absence of devices with all ports in the sequence list Sequence, set the connection bandwidth according to the device sequence in the sequence list.

Another aspect of the embodiments of the present application further provides a system for maintaining a PCIE signal connection by using an in-position signal, including: a judgment module, configured to respond to the interruption of the PCIE link between the main controller and the expansion cabinet, based on the expansion The presence signal of each port of the cabinet determines whether the cables between the main controller and the expansion cabinet are all in place; the sending module is configured to respond to the connection between the main controller and the expansion cabinet. The cables are all in place, and send an in-position signal to the main control chip of the main controller; the reset module is configured to release the reset signal and release the main control chip in response to the main control chip receiving the in-position signal. The reset state of the PCIE link of the upstream port of the control chip; and the connection module, configured to end the reset state in response to the PCIE link of the upstream port of the main control chip, and communicate the PCIE link so that the main controller and the The expansion enclosure is in a connected state.

In some embodiments, the judging module is configured to: obtain the in-position signal of each port of the expansion cabinet, and perform AND operation with other in-position signals after inversion of each in-position signal to determine whether the result is a preset value.

In some embodiments, a second reset module is further included, configured to: in response to that the cables between the main controller and the expansion cabinet are not all in place, reset the PCIE of the main control chip of the expansion cabinet Signal.

In some embodiments, a detection module is further included, configured to: in response to the main control cabinet and the expansion cabinet being in a connected state, to detect whether there are device sequences of all ports in the sequence list; There is a device sequence for all ports, and the connection bandwidth is set according to the device sequence in the sequence list.

In yet another aspect of the embodiments of the present application, a computer device is also provided, including: at least one processor; and a memory, where the memory stores computer instructions that can be executed on the processor, and the instructions are executed by the processor. The processor implements the steps of the above method when executed.

In another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, where the computer-readable storage medium stores a computer program that implements the above method steps when executed by a processor.

The present application has the following beneficial technical effects: by sending an in-position signal to the main control chip of the main controller; and releasing the reset signal and releasing the reset state of the PCIE link of the upstream port of the main control chip, the main controller and the expansion cabinet are maintained. The purpose of the connection is to improve the reliability of the product.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other embodiments can also be obtained according to these drawings without creative efforts.

1 is a schematic diagram of an embodiment of a method for maintaining a PCIE signal connection using an in-position signal provided by the present application;

FIG. 2 is a schematic diagram of a hardware structure of an embodiment of a computer device that maintains a PCIE signal connection by using an in-position signal provided by the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the following describes the embodiments of the present application in detail with reference to the accompanying drawings and specific embodiments.

It should be noted that all expressions using "first" and "second" in the embodiments of the present application are for the purpose of distinguishing two entities with the same name but not the same or non-identical parameters. It can be seen that "first" and "second" It is only for the convenience of expression and should not be construed as a limitation on the embodiments of the present application, and subsequent embodiments will not describe them one by one.

Based on the above purpose, in a first aspect of the embodiments of the present application, an embodiment of a method for maintaining a PCIE signal connection by using an in-position signal is proposed. FIG. 1 shows a schematic diagram of an embodiment of a method for maintaining a PCIE signal connection by using an in-position signal provided by the present application. As shown in Figure 1, the embodiment of the present application includes the following steps:

S1. In response to the interruption of the PCIE link between the main controller and the expansion cabinet, determine whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet;

S2. In response to all cables between the main controller and the expansion cabinet being in place, send an in-position signal to the main control chip of the main controller;

S3, in response to the main control chip receiving the in-position signal, release the reset signal and release the reset state of the PCIE link of the upstream port of the main control chip; and

S4. In response to the completion of the reset state of the PCIE link of the upstream port of the main control chip, the PCIE link is communicated so that the main controller and the expansion cabinet are in a connected state.

The main controller and the expansion cabinet interact through the Mini SAS cable, and they communicate through the PCIE protocol. Although it is the PCIE protocol, because the Mini SAS cable is used, there are only x16 differential signals inside the cable. Other sideband (sideband) signals, so the PCIE protocol between the main controller and the expansion cabinet is incomplete. The cabinet loses interaction, and the state machine enters the Polling state. Since there is no other sideband, the main controller cannot send a PERST (reset) signal to the expansion cabinet. Only restart the main control cabinet or the expansion cabinet to solve the problem. In this embodiment, when the PCIE link between the main controller and the expansion cabinet is simulated, the cables can be hot-plugged arbitrarily. For example, only any number of cables on the expansion cabinet can be hot-plugged, or only the main Any number of cables on the controller side can also be hot-swapped between the main controller and the expansion cabinet.

In response to the interruption of the PCIE link between the main controller and the expansion cabinet, it is determined whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet. In the standard Mini SAS cable, there will be a cable presence signal. When a cable is inserted into the expansion cabinet, the in-position signal of this port will be pulled down, and the pull-down signal will be input to the main control after a delay of 3 seconds. In the CPLD of the switch or expansion cabinet, the purpose of the delay is to prevent the operator from jittering and inactive and mistaking it for cable insertion. In addition, the Polling state of the PCIE state machine has a judgment of 24ms. If the Polling_Active is not completed within 24ms, it will jump out. The Polling state returns to the Detect state.

In some embodiments, the determining whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet includes: acquiring the presence signal of each port of the expansion cabinet , and perform AND operation with other in-position signals after inversion of each in-position signal to determine whether the result is a preset value. The CPLD of the main controller "inverts" the in-position signal of the port and registers it in the register, and performs an "AND" operation with the in-position signal of other ports. In the same principle, when the cables of all ports of the expansion cabinet are completely hot-inserted, the CPLD "inverts" the in-position signals of all ports one by one and performs an "AND" operation. Because the in-position signal is at a low level, it is "0" for CPLD, and when "inverting" the in-position signal of other ports and then performing the "AND" operation, if it is "0", it means that at least one cable is not connected. Insert, if it is "1", it means that all cables have been inserted into the expansion cabinet.

In response to the presence of all cables between the main controller and the expansion cabinet, a presence signal is sent to the main control chip of the main controller. In some embodiments, the method further includes: in response to the cables between the main controller and the expansion cabinet not being all in place, resetting the PCIE signal of the main control chip of the expansion cabinet. Define a 4-bit wire type variable prsnt_mini_sas[3:0], each bit represents the in-position status of each port, when the cable is in position, the corresponding variable is "0", when the cable is not in position, the corresponding The variable is "1", regardless of whether the cable is in place, prsnt_mini_sas[3:0] must be inverted and output, at this time, define a variable perst, and input the in-position information of the 4 ports into the variable: perst=( prsnt_mini_sas_dly[3:0]==4'b1111)? 1'b1:1'b0. If perst=0, it means that the cable is not in place, and the PCIE signal of the main control chip of the expansion cabinet is always reset; if perst=1, it means that all four cables are in place, and the PCIE signal of the main control chip of the expansion cabinet is released.

In response to the master control chip receiving the in-position signal, the reset signal is released and the reset state of the PCIE link of the upstream port of the master control chip is released. The CPLD outputs the in-position signals of all ports to the main controller or the main control chip of the expansion cabinet, which is used to control the PERST signal of the upstream port of the main control chip. When the cables of all ports are in place, the CPLD will eventually output " 1” (high level) to the main control chip, when the main control chip detects that it is a high level, the PERST signal will be released, and the PCIE link of the upstream port of the main control chip will be de-reset. When at least one cable of all ports is not in place, and the CPLD will eventually output "0" (low level) to the main control chip, the main control chip detects that it is a low level, and will always hold the PERST signal, so that the main control chip will always hold the PERST signal. The PCIE of the upstream port of the control chip is always in the reset state and will not be Linked, and the state machine is always in the Detect state.

In response to ending the reset state of the PCIE link of the upstream port of the main control chip, the PCIE link is connected so that the main controller and the expansion cabinet are in a connected state.

In some embodiments, the method further includes: in response to the main control cabinet and the expansion cabinet being in a connected state, detecting whether a sequence of devices with all ports exists in the sequence list; and in response to the absence of devices with all ports in the sequence list Sequence, set the connection bandwidth according to the device sequence in the sequence list. When all cables are inserted, the link initialization and training in the PCIe bus starts to enter the state machine mode and perform the Link process. For example, there are 4 ports, and each port has a bandwidth of x4. It is inserted from port 1. When port 3 or port 4 is inserted, the state machine will go from Detect to Polling to Configuration (configuration) and then to L0. If there is no Detect sequence For the device sequence of port 3 and port 4, PCIE does not have the bandwidth of x12, so only Link can be the bandwidth of the first two ports, that is, the bandwidth of x8. However, port 3 and port 4 are eventually inserted into the main controller and the expansion cabinet. If both the main controller and the expansion cabinet support the high-speed transmission of x16Gen3, they will automatically enter the Re-training (retraining) state to switch the speed again. , and ultimately the Gen3 rate and x16 bandwidth. If port 1 is inserted last, because the PCIE mechanism starts Link from Lane0, when port 1 is not inserted, it will not be linked. Only when port 1 is inserted, it will enter the state machine, which is directly the bandwidth of x16. No Re-training mechanism is required.

When the main controller and the expansion cabinet are in the normal Link state, any hot-plugging of a cable will cause the state machine to jump to the Detect state, and the PCIe device will detect whether there are other PCIe devices at the other end of its Link (connection). In other words, it is to detect whether there are other PCIe devices connected to it. When the cable is restored and it detects that there is a PCIE device on the opposite end, it will enter the next state from the Detect state until the L0 state to complete the Link.

If this judgment mechanism is not used, the probability of abnormal device connection will increase, because in the process of hot plugging, it will fall into a certain state of the state machine, and it will never be able to jump out. The main control chip must be reset or powered off. This greatly reduces work efficiency and greatly increases the probability of data loss. If this mechanism is used, recurrence of the above-mentioned problems is avoided, labor is saved, it is convenient for operators to replace cables, and the reliability of products is improved.

It should be particularly pointed out that the steps in each of the above-mentioned embodiments of the method for maintaining the PCIE signal connection by using the in-position signal can be crossed, replaced, added, and deleted. The method for maintaining the signal connection of the PCIE signal should also belong to the protection scope of the present application, and the protection scope of the present application should not be limited to the embodiments.

Based on the above purpose, a second aspect of the embodiments of the present application proposes a system for maintaining a PCIE signal connection by using an in-position signal, including: a judgment module configured to respond to the PCIE chain between the main controller and the expansion cabinet If the circuit is interrupted, based on the presence signal of each port of the expansion cabinet, it is judged whether the cables between the main controller and the expansion cabinet are all in place; the sending module is configured to respond to the main controller and the expansion cabinet. All cables between the expansion cabinets are in place, and send an in-position signal to the main control chip of the main controller; a reset module is configured to release the reset signal in response to the main control chip receiving the in-position signal And release the reset state of the PCIE link of the upstream port of the main control chip; And the connection module is configured to end the reset state in response to the PCIE link of the upstream port of the main control chip, and communicates the PCIE link so that all The main controller is in a connected state with the expansion cabinet.

Based on the above purpose, in a third aspect of the embodiments of the present application, a computer device is proposed, including: at least one processor; and a memory, where the memory stores computer instructions that can be executed on the processor, and the instructions are executed by the processor to The following steps are implemented: S1, in response to the interruption of the PCIE link between the main controller and the expansion cabinet, determine whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet; S2 . In response to all cables between the main controller and the expansion cabinet being in place, send an in-position signal to the main control chip of the main controller; S3. In response to the main control chip receiving the in-position signal, release the reset signal and release the main control chip. The reset state of the PCIE link of the upstream port of the control chip; and S4, in response to the end of the reset state of the PCIE link of the upstream port of the control chip, connect the PCIE link to make the main controller and the expansion cabinet in a connected state.

In some embodiments, the step further includes: in response to the cables between the main controller and the expansion cabinet not being all in place, resetting the PCIE signal of the main control chip of the expansion cabinet.

In some embodiments, the steps further include: in response to the main control cabinet and the expansion cabinet being in a connected state, detecting whether there are device sequences of all ports in the sequence list; and in response to the absence of devices of all ports in the sequence list Sequence, set the connection bandwidth according to the device sequence in the sequence list.

As shown in FIG. 2 , it is a schematic diagram of the hardware structure of an embodiment of the above-mentioned computer device using an in-position signal to maintain a PCIE signal connection provided by the present application.

Taking the device shown in FIG. 2 as an example, the device includes a processor 301 and a memory 302 , and may further include an input device 303 and an output device 304 .

The processor 301 , the memory 302 , the input device 303 and the output device 304 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 2 .

As a non-volatile computer-readable storage medium, the memory 302 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules. For example, in the embodiments of the present application, the PCIE signal is maintained by using the in-position signal. The program instruction/module corresponding to the connected method. The processor 301 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 302, that is, the method of maintaining the PCIE signal connection by using the in-position signal in the above method embodiments is realized. method.

The memory 302 may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required by at least one function; created data, etc. Additionally, memory 302 may include high speed random access memory, and may also include nonvolatile memory, such as at least one magnetic disk storage device, flash memory device, or other nonvolatile solid state storage device. In some embodiments, memory 302 may optionally include memory located remotely from processor 301, which may be connected to local modules via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 303 can receive input information such as user name and password. The output device 304 may include a display device such as a display screen.

One or more program instructions/modules corresponding to the method of using the in-position signal to maintain the PCIE signal connection are stored in the memory 302, and when executed by the processor 301, perform any of the above method embodiments using the in-position signal to maintain the PCIE signal connection. Methods.

Any embodiment of the computer device that executes the above-mentioned method for maintaining a PCIE signal connection by using an in-position signal can achieve the same or similar effects as any of the foregoing method embodiments corresponding to it.

The present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program that executes the above method when executed by a processor.

Finally, it should be noted that those of ordinary skill in the art can understand that all or part of the process in the method of the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. In a computer-readable storage medium, when the program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, the storage medium of the program may be a magnetic disk, an optical disk, a read only memory (ROM) or a random access memory (RAM) or the like. The above computer program embodiments can achieve the same or similar effects as any of the foregoing method embodiments corresponding thereto.

The above are exemplary embodiments disclosed in the present application, but it should be noted that various changes and modifications may be made without departing from the scope of the disclosure of the embodiments of the present application defined by the claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements disclosed in the embodiments of the present application may be described or claimed in an individual form, unless explicitly limited to the singular, they may also be construed as a plurality.

It should be understood that, as used herein, the singular form "a" is intended to include the plural form as well, unless the context clearly supports an exception. It will also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The above-mentioned embodiments of the present application disclose the serial numbers of the embodiments only for description, and do not represent the advantages and disadvantages of the embodiments.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, and the like.

Those of ordinary skill in the art should understand that the discussion of any of the above embodiments is only exemplary, and is not intended to imply that the scope (including the claims) disclosed by the embodiments of the present application is limited to these examples; under the idea of the embodiments of the present application , the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present application, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

A method for maintaining PCIE signal connection by utilizing in-position signal, is characterized in that, comprises the following steps:

In response to the interruption of the PCIE link between the main controller and the expansion cabinet, determine whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet;

In response to all cables between the main controller and the expansion cabinet being in place, sending an in-position signal to the main control chip of the main controller;

In response to the main control chip receiving the in-position signal, releasing the reset signal and releasing the reset state of the PCIE link of the uplink port of the main control chip; and

In response to ending the reset state of the PCIE link of the upstream port of the main control chip, the PCIE link is connected so that the main controller and the expansion cabinet are in a connected state.
The method according to claim 1, wherein the determining whether all cables between the main controller and the expansion cabinet are in place based on the presence signal of each port of the expansion cabinet comprises:

Acquire the in-position signal of each port of the expansion cabinet, invert each in-position signal and perform an AND operation with other in-position signals to determine whether the result is a preset value.
The method of claim 1, further comprising:

In response to the cables between the main controller and the expansion cabinet not being all in place, the PCIE signal of the main control chip of the expansion cabinet is reset.
The method of claim 1, further comprising:

In response to the main control cabinet and the expansion cabinet being in a connected state, detecting whether there is a device sequence for all ports in the sequence list; and

In response to a device sequence in which all ports are not present in the sequence list, the connection bandwidth is set in accordance with the device sequence in the sequence list.
A system for maintaining a PCIE signal connection by utilizing an in-position signal, comprising:

The judgment module is configured to, in response to the interruption of the PCIE link between the main controller and the expansion cabinet, judge whether all the cables between the main controller and the expansion cabinet are all based on the in-position signal of each port of the expansion cabinet. reign;

a sending module, configured to send an in-position signal to the main control chip of the main controller in response to the presence of all cables between the main controller and the expansion cabinet;

A reset module, configured to release the reset signal and release the reset state of the PCIE link of the upstream port of the main control chip in response to the main control chip receiving the in-position signal; and

The connection module is configured to connect the PCIE link so that the main controller and the expansion cabinet are in a connected state in response to the PCIE link of the main control chip's uplink port ending the reset state.
The system according to claim 5, wherein the judging module is configured to:

Acquire the in-position signal of each port of the expansion cabinet, invert each in-position signal and perform an AND operation with other in-position signals to determine whether the result is a preset value.
The system of claim 5, further comprising a second reset module configured to:

In response to the cables between the main controller and the expansion cabinet not being all in place, the PCIE signal of the main control chip of the expansion cabinet is reset.
The system of claim 5, further comprising a detection module configured to:

In response to the main control cabinet and the expansion cabinet being in a connected state, detecting whether there is a device sequence for all ports in the sequence list; and

In response to a device sequence in which all ports are not present in the sequence list, the connection bandwidth is set in accordance with the device sequence in the sequence list.
A computer device, comprising:

at least one processor; and

a memory storing computer instructions executable on the processor, the instructions implementing the steps of the method of any one of claims 1-4 when executed by the processor.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method of any one of claims 1-4 are implemented.