CN116278880A

CN116278880A - Charging equipment and method for controlling mechanical arm to charge

Info

Publication number: CN116278880A
Application number: CN202111562748.7A
Authority: CN
Inventors: 陈辰; 楚亚奎; 薛景涛; 贺亚农; 陈晓鹏; 黄远灿; 张伟民
Original assignee: Huawei Technologies Co Ltd; Beijing Institute of Technology BIT
Current assignee: Huawei Technologies Co Ltd; Beijing Institute of Technology BIT
Priority date: 2021-12-20
Filing date: 2021-12-20
Publication date: 2023-06-23
Also published as: WO2023116667A1

Abstract

The embodiment of the application discloses charging equipment, include: the control structure and the mechanical arm main body are provided with a force sensor and a charging plug, and the force sensor is used for collecting acting force information between the side surface of the charging head and the charging port after the charging plug is in contact with the charging port; the control structure can control the pose of the mechanical arm main body according to the acting force information so as to insert the charging plug into the charging port. According to the method and the device, the pose control of the mechanical arm is carried out by acquiring the acting force information acquired by the force sensor, and the charging plug can adapt to the insertion direction of the charging port according to the force sense adjustment position and the pose under the condition that the charging plug is partially inserted and the visual information cannot acquire errors, so that the accuracy and the reliability of the plugging task are improved.

Description

Charging equipment and method for controlling mechanical arm to charge

Technical Field

The application relates to the technical field of vehicles, in particular to charging equipment and a method for controlling mechanical arm charging.

Background

With the development and large-scale use of new energy automobiles, mobile charging equipment is increasingly favored by users because of flexible charging resources, environment and parking space conditions. The mobile charging equipment is in a mobile form of a charging pile or a mobile chassis carrying a battery pack, for example, when the mobile charging equipment is a mobile charging car, a plurality of battery packs can be carried on the chassis of the mobile charging car, and after the mobile charging car receives a user request, the mobile charging car drives to a place where the user is located, and provides charging service for a car driven by the user, so that a charging task is completed.

Before the mobile charging device charges the vehicle to be charged, the mobile charging device needs to be moved to the vicinity of the vehicle to be charged so as to position a positioning opening area on the vehicle to be charged, and adjust the pose (the position of the mobile charging device and the pose of the charging arm) of the mobile charging device so as to charge the charging opening.

The current mainstream charging robot schemes all adopt structured light or binocular or ultrasonic combined mechanical arm planning modes to realize plugging through position control. Firstly, a charging port is photographed and identified in a perception range of a structured light camera, the pose of a charging seat relative to the camera is obtained, then the pose coordinates are converted into the pose of a charging socket at the tail end of the mechanical arm, and then a control instruction of the mechanical arm can be obtained, and the mechanical arm motion planning is carried out to complete the plugging.

Because in order to realize the accurate inserting of charging socket to the mouth that charges, the pose positioning accuracy that possesses sub-millimeter level (for example less than or equal to 2 degrees) just can guarantee to charge the smooth inserting of socket to the mouth that charges, and dock and do not damage connecting device, consequently, the scheme based on visual feedback just requires the vision detection module to just reach the position and discerns sub-millimeter level, pose discernment 2 degrees requirement, but current structure light technique basic accuracy can reach 1 ~ 2mm, pose is about 3 degrees, still can receive the restriction of apart from the angle of view, just hardly guarantee smooth plug by itself, even insert, because the existence of pose error also can lead to great plug power, easily destroy equipment, even damage arm. Therefore, the accuracy requirement of smooth insertion of the charging jack into the charging port is hardly met by performing the insertion and the extraction based on the visual feedback technology only.

Disclosure of Invention

The application provides a mechanical arm, a charging device and a method for controlling the charging of the mechanical arm, which can improve the accuracy and the reliability of a plugging task.

In a first aspect, the present application provides a robotic arm comprising: a control structure and a robot arm body; the mechanical arm comprises a mechanical arm main body, and is characterized in that a force sensor and a charging plug are arranged on the mechanical arm main body, the charging plug comprises a plurality of charging heads, the force sensor is used for collecting acting force information of a contact surface between the charging plug and the charging port, the contact surface comprises a side surface of at least one charging head in the plurality of charging heads, and the side surface is a surface parallel to the axial direction of the charging head; the control structure is used for controlling the pose of the mechanical arm main body according to the acting force information so as to insert the charging plug into the charging port.

The charging head can be of a cylindrical structure and can comprise a bottom surface and a side surface, the bottom surface can be a surface which is required to be contacted with the bottom of the charging hole during charging, and the side surface can be a surface which is required to be contacted with the side wall of the charging hole during charging.

In one possible implementation, the force information includes a force in a radial direction of the charging head.

In the existing scheme of performing pose control only based on vision, because of errors of the vision, the vision camera cannot continuously observe the charging port when the charging plug is near the charging port, if the situation that the charging plug cannot be plugged to the bottom and wedged is likely to occur only by adopting the vision pose observed at the optimal observation point, the pose control of the mechanical arm is performed by acquiring the acting force information acquired by the force sensor, and the charging plug can be adjusted to adapt to the plugging direction of the charging port according to the force sense under the condition that the vision information cannot acquire errors (such as the side surface is contacted with the charging port), so that the accuracy and the reliability of the plugging task are improved. The situation that the charging head cannot reach the bottom due to the fact that the side face of the charging head and the inner wall of the charging port are extruded or rubbed is reduced.

In one possible implementation, the effort information includes moment information of the charging port to the charging plug.

The force information may indicate, among other things, a 6-dimensional force of the charging port against the charging plug, the 6-dimensional force may include a force in three-dimensional space (e.g., along an x-axis, a y-axis, a z-axis) and a moment in three-dimensional space (e.g., rotating about the x-axis, the y-axis, the z-axis). Because moment information is included in the charging head, the blocking direction of the charging port to the charging head can be accurately and finely identified, and accurate force control is further provided for accurately inserting the charging head into the charging hole.

In one possible implementation, the force sensor is fixed to a side of the at least one charging head; or the mechanical arm comprises a joint with a rotation degree of freedom, the control structure is used for controlling the pose of the mechanical arm main body by controlling the rotation of the joint, and the force sensor is fixed on the joint; or, the force sensor is fixed between the charging plug and the mechanical arm main body.

In one possible implementation, the charging port includes a plurality of charging holes, and the plurality of charging heads are matched with the plurality of charging holes one by one; the control structure is specifically used for:

determining pose adjusting parameters according to the acting force information;

according to the pose adjustment parameters, correcting the pose of the mechanical arm main body from a first pose to a second pose; the first pose and the second pose are different, and the second pose is the pose of the mechanical arm main body when each charging head is aligned with a matched charging hole.

In one possible implementation, based on the force information, it may be determined whether the robotic arm body is in the correct pose (e.g., the second pose may be the correct pose). For example, when the force information indicates that there is a non-axial pressure (or a moment) of the charging port against the charging plug, it may be determined that the robotic arm body is not in the correct pose, in which case at least one of the plurality of charging heads is not aligned with the matching charging hole, i.e., the central axis of the charging head does not coincide with the central axis of the matching charging hole or the deviation is greater than a threshold value.

That is, the first pose may be a case where the central axis of the charging head does not coincide with the central axis of the matched charging hole or the deviation is greater than a threshold value (i.e., the above-described pose is not correct).

In the embodiment of the application, when the mechanical arm main body is not in the correct pose, the pose of the mechanical arm main body can be adjusted based on the acting force information, so that the mechanical arm main body is in the correct pose.

In one possible implementation, the control mechanism may correct the pose of the mechanical arm body (to a second pose, that is, a correct pose) according to the force information when the mechanical arm body is in the first pose, where the mechanical arm body may insert the charging plug into the bottom of the charging port in a correct manner with less resistance.

In one possible implementation, the first pose is a pose when bottom end surfaces of the plurality of charging heads are not in contact with bottoms of the charging holes, and the bottom end surfaces are surfaces perpendicular to an axial direction of the charging heads.

In one possible implementation, the aligning includes: the projection range of the charging head along the direction of the matched charging hole is within the range of the matched charging hole, and the central axis of the charging head coincides with the central axis of the matched charging hole or the deviation is smaller than a threshold value.

In one possible implementation, the control structure is further configured to: and after correcting the pose of the mechanical arm main body from the first pose to the second pose, controlling the pose of the mechanical arm main body so that the charging plug moves towards the bottom of the charging hole and along the direction of the central shaft of the charging head.

In one possible implementation, in the process of correcting the first pose to the second pose, the resistance of the charging port to the charging plug in a target direction is reduced, and the target direction is an axial direction or a radial direction of the charging head.

It will be appreciated that there may be a tendency for the charging port to reduce resistance to the charging head in a target direction, which may be a radial direction and/or an axial state (e.g., may be a radial direction, an axial state, or both). Here, the tendency of the charging port to decrease is understood to mean that the resistance of the charging port to the charging head in the target direction may not decrease all the time, but may increase locally, but the resistance of the charging port to the charging head in the target direction decreases in terms of tendency.

After correcting the pose of the mechanical arm main body from the first pose to the second pose, the charging plug is in an aligned pose with the charging port, in which case the charging plug may be moved towards the bottom of the charging hole and along the direction of the central axis of the charging head until the bottom, and in one possible implementation, the control structure may instruct the charging port to have only resistance to the charging plug along the axial direction of the charging plug based on the force information, so as to control the mechanical arm main body to maintain a stationary state, thereby completing the task of inserting the mechanical arm main body into the charging port.

In one possible implementation, during the correction from the first position to the second position, the charging port has a tendency to reduce the moment between the charging pins.

In one possible implementation, the pose adjustment parameter is specifically used to control the movement direction and the movement rate of the charging plug.

In one possible implementation, the control mechanism may acquire image information acquired by a vision sensor for an area of the charging port in real time; obtaining the real-time pose of the mechanical arm main body according to the position information of the charging port in the image information; and controlling the pose of the mechanical arm main body according to the real-time pose so as to guide a charging plug on the mechanical arm main body to the periphery of the charging port.

Because the image information that gathers in the visual servo is the real-time acquisition, can solve the real-time position appearance that obtains the arm main part based on the image information that acquires in real time (this real-time position appearance can be the relative position appearance between the terminal and the mouth that charges of waiting to charge the vehicle of arm main part), because when controlling the position appearance of arm main part, the base removes, the circumstances such as the hindrance of barrier can appear, leads to the position appearance error of arm main part, and this application can revise the position appearance of arm main part based on real-time position appearance, can improve the position appearance recognition accuracy of arm main part, and then can be more accurate with the charging plug in the arm main part guide to around the mouth that charges.

In one possible implementation, visual servoing is a closed-loop control method that visually incorporates motion. The main function is to realize the guiding and transporting work of the charging gun at the tail end of the mechanical arm to the charging port. The general flow is to take the pose result of the visual identification of the charging gun/charging seat as input, calculate the driving error by adopting the visual servo feedback control idea, and realize the accurate guidance of the tail end of the mechanical arm to the target pose by adopting the visual servo based on the pose. The self-adaptive gain method is adopted to accelerate the adjustment speed in small errors and realize quick visual servo. The robot arm will be at the best observation point at the end of the servo in this step, where the pose of the most accurate charging port has been obtained. The robotic arm will now be ready for the next step of force-controlled insertion based on guiding the charging socket to the vicinity of the charging port.

In one possible implementation, the accuracy of the target viewing pose is guaranteed because the optimal observation point needs to guarantee a certain stability. However, the pose of the mobile robot base relative to the charging stand may be constantly changing. In order to ensure the consistent pose of the optimal observation point relative to the charging seat under the change, the aim is fulfilled by adopting a quick visual servo of the pose. In one possible implementation, the visual servo thought of pose feedback can be adopted, and the joint speed control law is designed based on the Litsea stability criterion, so that the precise guidance of the tail end of the mechanical arm to the target position is realized.

Optionally, in one possible implementation, the vision sensor is disposed on the mechanical arm body.

Alternatively, in one possible implementation, the vision sensor may be disposed not on the robot arm body but at a position corresponding to the third party viewing angle.

In one possible implementation, in making the above determination of the relative pose, it is necessary that the vision sensor acquire image information for the charging port and determine the relative pose between the charging port and the vision sensor based on the image information.

In one possible implementation, the image information includes a 2D image and a 3D point cloud, and the control mechanism is further configured to: extracting a first 3D position corresponding to the charging port according to the 2D image and a preset charging port 3D model;

Extracting a second 3D position corresponding to the charging port according to the 3D point cloud; and fusing the first 3D position and the second 3D position to obtain the pose of the charging port.

According to the 2D image, two-dimensional pixel coordinates of the center point of the charging port can be obtained, and the number of each point is obtained through the mutual position relation of the center points; and obtaining the three-dimensional physical coordinates of the center point of the round hole under the target coordinate system according to the charging seat standard CAD model, and further obtaining the three-dimensional physical coordinates of each charging hole.

In one possible implementation, the first 3D location includes 3D locations of M first feature points on the charging port; the second 3D position includes 3D positions of M second feature points on the charging port; the M first feature points and the M second feature points are in one-to-one correspondence; the fusing the first 3D position and the second 3D position includes: and respectively fusing the 3D positions of the M first characteristic points with the 3D positions of the corresponding second characteristic points. Wherein the first feature point and the second feature point may be charging holes.

The charging port pose accurate identification method based on 2D image-3D point cloud information can adopt a 2D depth image detection network to accurately detect two-dimensional key points of a charging port in a sub-pixel level, adopts three-dimensional key point detection to reliably detect offset based on a 3D Hough network on the three-dimensional key points of the charging port, and enables projection errors of the two-dimensional key points and the three-dimensional key points to be minimum through fusion (such as nonlinear optimization), so that the accurate pose of the charging port is obtained, and the problem of pose high-accuracy identification of the charging port under the conditions of large illumination change and low target contrast is solved.

In one possible implementation, the mechanical arm body further includes a target interface for detachably connecting the charging plug. Alternatively, the target interface may be a component that is based on the detachable connection of the cleaning implement. The magnetic structure is adopted to grasp the charging gun, so that reliable butt joint of the charging plug and the mechanical arm is ensured, and then the next charging operation is carried out.

In one possible implementation, the target interface is connected with an initial charging plug before the control structure controls the pose of the mechanical arm main body according to the acting force information;

the control structure is also used for controlling the pose of the mechanical arm main body and the connection state of the target interface, and the initial charging plug is detached from the target interface and replaced by the charging plug; wherein, the liquid crystal display device comprises a liquid crystal display device,

the target charging plug is a plug supporting quick charging, and the charging plug is a plug supporting slow charging; or alternatively, the process may be performed,

the target charging plug is a plug supporting slow charging, and the charging plug is a plug supporting fast charging.

In this embodiment of the application, charging equipment can be compatible to fill two kinds of modes of charging of rifle soon and slow charge rifle, can possess quick change mechanism, can select the rifle that charges according to user's task demand.

In one possible implementation, the charging device further includes: a plug receiving area including a target receiving location for setting the charging plug;

the controlling the pose of the mechanical arm main body and the connection state of the target interface, and detaching and replacing the initial charging plug from the target interface to the charging plug, includes:

controlling the connection state of the target interface, and detaching the initial charging plug from the target interface;

and controlling the pose of the mechanical arm main body, moving the target interface to the target storage position, controlling the connection state of the target interface, and connecting the target interface with the charging plug.

In a second aspect, the present application provides a charging device comprising: mechanical arm, force sensor and charging plug;

the mechanical arm comprises a control structure and a mechanical arm main body;

the mechanical arm main body is provided with the force sensor and the charging plug, the charging plug comprises a plurality of charging heads, the force sensor is used for collecting acting force information of a contact surface between the charging plug and the charging port, the contact surface comprises a side surface of at least one charging head in the plurality of charging heads, and the side surface is a surface parallel to the axial direction of the charging head;

The control structure is used for controlling the pose of the mechanical arm main body according to the acting force information so as to insert the charging plug into the charging port.

In one possible implementation, the force sensor is fixed to a side of the at least one charging head; or alternatively, the process may be performed,

the mechanical arm comprises a joint with a rotation degree of freedom, the control structure is used for controlling the pose of the mechanical arm main body by controlling the rotation of the joint, and the force sensor is fixed on the joint; or alternatively, the process may be performed,

the force sensor is fixed between the charging plug and the mechanical arm main body.

In one possible implementation, the aligning includes: the projection of the charging head along the direction of the matched charging hole falls in the range of the matched charging hole, and the central axis of the charging head coincides with the central axis of the matched charging hole or the deviation is smaller than a threshold value.

In one possible implementation, the control structure is further configured to:

and after correcting the pose of the mechanical arm main body from the first pose to the second pose, controlling the pose of the mechanical arm main body so that the charging plug moves towards the bottom of the charging hole and along the direction of the central shaft of the charging head.

In one possible implementation, the control structure is specifically configured to:

and controlling the mechanical arm main body to keep a static state based on the acting force information indicating that the charging port only has resistance to the charging plug along the axial direction of the charging plug.

In one possible implementation, the control mechanism is further configured to:

acquiring image information acquired by a vision sensor aiming at the area of the charging port in real time before controlling the pose of the mechanical arm main body according to the acting force information;

obtaining the real-time pose of the mechanical arm main body according to the position information of the charging port in the image information;

and controlling the pose of the mechanical arm main body according to the real-time pose so as to guide a charging plug on the mechanical arm main body to the periphery of the charging port.

In one possible implementation, the visual sensor is disposed on the robotic arm body.

In one possible implementation, the image information includes a 2D image and a 3D point cloud, and the control mechanism is further configured to:

extracting a first 3D position corresponding to the charging port according to the 2D image and a preset charging port 3D model;

extracting a second 3D position corresponding to the charging port according to the 3D point cloud;

And fusing the first 3D position and the second 3D position to obtain the pose of the charging port.

In one possible implementation, the first 3D location includes 3D locations of M first feature points on the charging port; the second 3D position includes 3D positions of M second feature points on the charging port; the M first feature points and the M second feature points are in one-to-one correspondence;

the fusing the first 3D position and the second 3D position includes:

and respectively fusing the 3D positions of the M first characteristic points with the 3D positions of the corresponding second characteristic points.

In one possible implementation, the mechanical arm body further includes a target interface for detachably connecting the charging plug.

In one possible implementation, the charging device is a stationary charging stake, a mobile robot, or a charging vehicle.

In a third aspect, the present application provides a method for controlling charging of a robotic arm, where the method is applied to a control structure of the robotic arm, a force sensor and a charging plug are provided on a robotic arm main body, the charging plug includes a plurality of charging heads, the force sensor is used for collecting force information of a contact surface between the charging plug and the charging port, the contact surface includes a side surface of at least one charging head of the plurality of charging heads, and the side surface is a surface parallel to an axial direction of the charging head;

The method comprises the following steps:

and controlling the pose of the mechanical arm main body according to the acting force information so as to insert the charging plug into the charging port.

In one possible implementation, the charging port includes a plurality of charging holes, and the plurality of charging heads are matched with the plurality of charging holes one by one; according to the acting force information, controlling the pose of the mechanical arm main body comprises the following steps:

In one possible implementation, after correcting the pose of the robotic arm body from the first pose to the second pose, the method further comprises:

and controlling the pose of the mechanical arm main body so that the charging plug moves towards the bottom of the charging hole and along the direction of the central shaft of the charging head.

In one possible implementation, the method further comprises:

In one possible implementation, the image information includes a 2D image and a 3D point cloud, the method further comprising:

the fusing the first 3D position and the second 3D position includes:

The method further comprises the steps of: controlling the pose of the mechanical arm main body and the connection state of the target interface, and detaching and replacing the initial charging plug from the target interface to the charging plug; wherein, the liquid crystal display device comprises a liquid crystal display device,

In a fourth aspect, embodiments of the present application provide a computer readable storage medium comprising computer readable instructions which, when run on a computer device, cause the computer device to perform the third aspect and any optional method thereof described above.

In a fifth aspect, embodiments of the present application provide a computer program product comprising computer readable instructions which, when run on a computer device, cause the computer device to perform the third aspect and any optional method thereof.

In a sixth aspect, the present application provides a chip system comprising a processor for supporting an execution control mechanism to implement the functions involved in the above aspects, for example, to transmit or process data involved in the above method; or, information. In one possible design, the system-on-chip further includes a memory for holding program instructions and data necessary to execute the device or to train the device. The chip system can be composed of chips, and can also comprise chips and other discrete devices.

In a seventh aspect, the present application provides an apparatus for controlling charging of a robotic arm, the apparatus comprising one or more processors and a memory; wherein the memory has stored therein computer readable instructions; the one or more processors reading the computer readable instructions and performing the third aspect and any optional method thereof described above.

The application provides a charging device, its characterized in that includes: mechanical arm, force sensor and charging plug; the mechanical arm comprises a control structure and a mechanical arm main body; the mechanical arm main body is provided with the force sensor and the charging plug, the charging plug comprises a plurality of charging heads, the force sensor is used for collecting acting force information of a contact surface between the charging plug and the charging port, the contact surface comprises a side surface of at least one charging head in the plurality of charging heads, and the side surface is a surface parallel to the axial direction of the charging head; the control structure is used for controlling the pose of the mechanical arm main body according to the acting force information so as to insert the charging plug into the charging port. In the existing scheme of performing pose control only based on vision, because of errors of the vision, the vision camera cannot continuously observe the charging port when the charging plug is near the charging port, if the situation that the charging plug cannot be plugged to the bottom and wedged is likely to occur only by adopting the vision pose observed at the optimal observation point, the pose control of the mechanical arm is performed by acquiring the acting force information acquired by the force sensor, and the charging plug can be adjusted to adapt to the plugging direction of the charging port according to the force sense under the condition that the vision information cannot acquire errors (such as the side surface is contacted with the charging port), so that the accuracy and the reliability of the plugging task are improved. The situation that the charging head cannot reach the bottom due to the fact that the side face of the charging head and the inner wall of the charging port are extruded or rubbed is reduced.

Drawings

Fig. 1 is a schematic view of a scenario provided in an embodiment of the present application;

fig. 2 is a schematic view of a scenario provided in an embodiment of the present application;

FIG. 3 is a schematic view of a scenario provided in an embodiment of the present application;

fig. 4 is a schematic view of a scenario provided in an embodiment of the present application;

FIG. 5 is a schematic view of a scenario provided in an embodiment of the present application;

FIG. 6 is a schematic view of a scenario provided in an embodiment of the present application;

fig. 7 is a schematic view of a scenario provided in an embodiment of the present application;

fig. 8 is a schematic structural diagram of a charging device according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a search method according to an embodiment of the present application;

FIG. 10 is a schematic illustration of a detection result provided in an embodiment of the present application;

FIG. 11 is a schematic illustration of a detection result provided in an embodiment of the present application;

FIG. 12 is a schematic illustration of a pose calculation method according to an embodiment of the present application;

FIG. 13 is a schematic illustration of an information fusion algorithm according to an embodiment of the present application;

FIG. 14 is a schematic view of a model structure provided in an embodiment of the present application;

FIG. 15 is a schematic illustration of a detection result provided in an embodiment of the present application;

FIG. 16 is a coordinate system transformation diagram provided in an embodiment of the present application;

FIG. 17 is a schematic representation of 3D information extraction provided in an embodiment of the present application;

FIG. 18 is a schematic illustration of a method for controlling charging of a robotic arm according to an embodiment of the present disclosure;

FIG. 19 is a schematic illustration of a robotic arm provided in an embodiment of the present application;

FIG. 20 is a schematic diagram of a method for controlling charging of a robotic arm according to an embodiment of the present disclosure;

FIG. 21 is a schematic illustration of an RCC device provided in an embodiment of this application;

FIG. 22 is a schematic illustration of controlling charging of a robotic arm according to an embodiment of the present application;

FIG. 23 is a force indication provided by an embodiment of the present application;

fig. 24 is a schematic diagram of a device for controlling charging of a mechanical arm according to an embodiment of the present application;

fig. 25 is a schematic structural diagram of a chip according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the accompanying drawings in the embodiments of the present application. The terminology used in the description section of the present application is for the purpose of describing particular embodiments of the present application only and is not intended to be limiting of the present application.

The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which the embodiments of the application described herein have been described for objects of the same nature. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

New energy vehicles represent a new direction of green development in the automobile industry, and explosive development occurs in China and the world. With the steep increase of sales of new energy automobiles, owners need to power up in the way. However, the existing service area or parking lot charging piles are relatively severely insufficient, so that the phenomenon that users are in large queues and the waiting time of the queues is far longer than the charging time often occurs. Under the condition that a large amount of charging piles cannot be provided in a short time, mobile charging equipment (such as a charging robot) is adopted to automatically charge the new energy vehicle, a mobile charging device is provided for the new energy vehicle, and the problem that the charging piles are seriously insufficient can be effectively relieved in a short time. On the other hand, even if the empty charging pile exists in the parking lot, the time is particularly spent for finding the electric car charging pile in the parking lot, and meanwhile, the problem that the charging gun and the cable are heavy and high in hardness is commonly caused by the fast charging pile, so that the manual charging operation of a driver is difficult. The charging robot can automatically navigate to the front of the car owner and replace people to plug the charging gun, so that more efficient charging service and better charging experience can be brought to the user.

As shown in fig. 1, the mobile charging device may specifically be in a form of a mobile charging pile or a form of a battery pack carried on a mobile chassis, for example, when the mobile charging device is a mobile charging vehicle, a plurality of battery packs may be carried on the chassis of the mobile charging vehicle, and after the mobile charging vehicle receives a request of a user, the mobile charging vehicle drives to a place where the user is located, and provides charging service for a vehicle driven by the user, so as to complete a charging task.

The technical solution of the present application may be applied to a mobile charging system, as shown in fig. 2, which may include a server 11, at least one vehicle system 12, a mobile charging device 13, a charging pile 14, and the like. Each of the vehicle systems 12 may include therein a user 121, a terminal device 122, and a vehicle to be charged 123, as shown in fig. 3. The vehicle 123 is a vehicle that the user 121 is driving and that needs to be charged, and the holder of the terminal device 122 is the user 121.

Wherein the user 121 may be a driver, and when the driver finds that the electric quantity of the vehicle 123 driven by the driver is insufficient, a request message is sent to the server through the terminal device 122, wherein the request message is used for requesting to charge the vehicle 123. Alternatively, the user 121 generates a request on the APP of the terminal device 122 and then sends the request to the server 11.

The terminal device 122 may be a portable device, such as a smart terminal, a mobile phone, a notebook computer, a tablet computer, a personal computer (personal computer, PC), a personal digital assistant (personal digital assistant, PDA), a foldable terminal, a wearable device (e.g., a smart watch or a bracelet) with a wireless communication function, a user device (UE), an augmented reality (augmented reality, AR) or Virtual Reality (VR) device, etc., and the embodiment of the present application does not limit the specific device configuration of the terminal device.

The vehicle 123 may be an Electric Vehicle (EV) including a display screen, an on-vehicle processor, and a communication module, and may further include other components or units, which is not limited in this embodiment.

Alternatively, in a possible implementation manner, the vehicle system 12 may not include the terminal device 122, where the function of the terminal device 122 is implemented by a vehicle processor and a communication module in the EV, for example, the vehicle processor sends a request message to the server 11 through the communication module after receiving an instruction triggered by the user.

Referring to fig. 2, the server 11 may receive a request message sent by one or more vehicle systems 12, schedule at least one mobile charging device 13 in the system, and dispatch a charging task to the corresponding mobile charging device indicating its location to the user sending the request message to charge the requested vehicle. In addition, before the charge task is scheduled and distributed, the server 11 is further configured to acquire road condition information, perform advanced driving support system (advanced driver assistance systems, ADAS) calculation according to the road condition information, and generate a scheduling policy, etc.

Alternatively, the server 11 is a cloud server, and the cloud server may be one server, or a server cluster formed by a plurality of servers.

The mobile charging device 13 is configured to receive the indication signal sent by the server 11, drive to a position of a vehicle to be charged (also referred to as a target vehicle in the embodiment of the application) according to the indication signal, and charge the vehicle, and after the mobile charging device 13 completes a charging task, return to the charging pile 14 to perform power compensation. In addition, the mobile charging device 13 is further configured to report the current power situation of the carried battery pack in real time during the process of executing the charging task, and report its own position information to the server 11 in real time during the driving process, so as to provide a basis for the server 11 to grasp the situation of each mobile charging device in the system in real time, and for the dispatch of the subsequent charging task and the scheduling of the electric energy resource.

In addition, each mobile charging device 13 includes a vehicle processor, a communication module, a rechargeable battery or a battery pack, where the number of battery packs may be one or more, and the amount of electricity carried by each battery pack is not limited in this embodiment.

Alternatively, the mobile charging device 13 may be an EV, or a fuel-powered car.

Next, an architectural illustration of the mobile charging device 13 will be described.

The embodiment of the present application may be applied to the mobile charging device 13 as shown in fig. 4. As shown in fig. 4, the mobile charging device 13 may include: sensor module 110, drive device 120, charging device 140, and master control platform 130.

The sensor module 110 may include one or more visual sensors 111 (e.g., a camera) (which may also be referred to as a sensor in the embodiments of the present application for short), for example, may include a common optical camera, or may be an infrared camera, a structured light sensor, or a time-of-flight (ToF) sensor. For example, sensor module 110 may include a single conventional RGB camera or red-yellow-blue (red yellow yellow blue, RYYB) camera, and camera module may also include multiple cameras or sensor-based RGB-D depth camera solutions. For example, the RGB-D depth camera scheme may include a binocular scheme formed by two RGB cameras, a structured light scheme formed by one RGB camera and one structured light sensor, or a ToF scheme formed by one RGB camera and one ToF sensor, which is not specifically limited in this embodiment of the present application. The vision sensor 111 (e.g., camera) may be a fixed-focus camera or a zoom camera, and may have, for example, phase focusing, laser focusing, or the like.

It should be appreciated that the vision sensor 111 (e.g., camera) may be carried on a motion unit that is used to carry the vision sensor 111 (e.g., camera) and drive the vision sensor 111 (e.g., camera) to rotate. In one embodiment, the motion unit may drive the vision sensor 111 (e.g., camera) to rotate in two degrees of freedom, and if the direction pointed by the Z axis is right in front of the camera 11, the two degrees of freedom rotation may include rotation of the vision sensor 111 (e.g., camera) about the x axis and rotation of the vision sensor 111 (e.g., camera) about the y axis. The motion unit driving the rotation of the vision sensor 111 (e.g. camera) may be achieved by rotation of a steering engine or a servo motor, for example, when the driving device is used to drive the vision sensor 111 (e.g. camera) to rotate in two degrees of freedom, the motion unit may comprise two driving mechanisms driver 1 and driver 2, such as two steering engines or two servo motors, wherein one steering engine 1 or servo motor 1 is used to control the rotation of the vision sensor 111 (e.g. camera) about the x-axis and the other steering engine 2 or servo motor 2 is used to control the rotation of the vision sensor 111 (e.g. camera) about the y-axis. In other embodiments, the motion unit may drive the vision sensor 111 (e.g. camera) to rotate in three degrees of freedom, that is, increase the rotation of the vision sensor 111 (e.g. camera) about the z-axis, and correspondingly, the motion unit may further include three driving mechanisms driver 1, driver 2 and driver 3, for example, three steering engines or three servo motors, where one steering engine 1 or servo motor 1 is used to control the rotation of the vision sensor 111 (e.g. camera) about the x-axis and the other steering engine 2 or servo motor 2 is used to control the rotation of the vision sensor 111 (e.g. camera) about the y-axis and the other steering engine 3 or servo motor 3 is used to control the rotation of the vision sensor 111 (e.g. camera) about the z-axis.

The sensor module 110 may further include a motion sensor 112, where the motion sensor 112 may be an odometer, an accelerometer, a speedometer, an inertial measurement unit, etc. for collecting mileage information, such as travel, track, speed, etc., of the mobile charging device 13 during traveling.

The force sensor 113 may be a sensor for detecting the application of force to the end of the robotic arm 142. As the force sensor 113, a pressure sensor capable of detecting a force in a uniaxial direction and a force sensor or a moment sensor capable of detecting a component of forces in a plurality of axial directions can be used. In the present embodiment, as the force sensor 113, a six-axis force sensor may be used. The six-axis force sensor detects the magnitudes of forces parallel to three detection axes orthogonal to each other in the intrinsic sensor coordinate system and the magnitudes of moments about the three detection axes. The force sensor 113 may be provided at a position other than the position of the distal end of the arm 142, for example, at one or more joints in the arm 142.

The drive 120 may include components that provide powered movement of the mobile charging device 13. In one embodiment, the drive 120 may include an engine, an energy source, a transmission, and wheels/tires. The engine may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine of a gasoline engine and an electric motor, or a hybrid engine of an internal combustion engine and an air compression engine. The engine converts the energy source into mechanical energy.

Examples of energy sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source may also provide energy to other systems of the mobile charging device 13.

The transmission may transmit mechanical power from the engine to the wheels. The transmission may include a differential and a drive shaft. In one embodiment, the transmission may also include other devices, such as clutches. Wherein the drive shaft may comprise one or more axles coupleable to one or more wheels.

The main control platform 130 is a data processing and control center of the device, and the main control platform 130 is in communication connection with the charging device 140, the sensor module 110 and the driving device 120, for example, can receive image data collected by the sensor module 110, process the image data, send a moving instruction to the driving device 120, and the like. The charging device 140 may include a charging head 141 (or referred to as a charging plug) and a robot arm 142. In some embodiments, the master platform 130 may be an embedded computer platform including, but not limited to, a computer chip and software system designed based on the X86 instruction set, ARM instruction set, RISC-V instruction set, MIPS instruction set, or the like.

In one embodiment, the computer chip may include a processor 131 and a memory 132, for example, where the processor 131 may include, for example: a central processor (central processing unit, CPU), a system on a chip (SoC), an application processor (application processor, AP), a microcontroller (microcontroller), a Neural network processor (Neural-network Processing Unit, NPU), and/or a graphics processor (graphics processing unit, GPU), etc.; the memory 132 may include, for example, a nonvolatile memory such as a flash memory (flash memory), including NAND flash, a solid state disk, and the like, and a volatile memory such as a Synchronous Dynamic Random Access Memory (SDRAM), and the like.

In one embodiment, the software system may include an operating system and program instructions 133 running in the operating system. When the processor executes the program instructions, the apparatus shown in fig. 3 or fig. 4 is caused to execute the steps of the method for controlling the charging of the mechanical arm provided in the embodiment of the present application.

In some embodiments, memory 132 may contain program instructions 133 (e.g., program logic) that program instructions 133 may be executed by processor 131 to perform various functions of mobile charging device 13, including those described above. The memory 132 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and/or control one or more of the drive 120, sensor module 110, control system, and peripherals.

In addition to program instructions 133, memory 132 may also store data such as road maps, route information, position, direction, speed of autopilot, and other such autopilot data, as well as other information. Such information may be used by mobile charging device 13 during operation of mobile charging device 13 in autonomous, semi-autonomous, and/or manual modes.

The wireless communication system 150 may communicate wirelessly with one or more devices (e.g., the server 11) directly or via a communication network. For example, the wireless communication system 150 may use 3G cellular communication, such as code division multiple access (code division multiple access, CDMA), EVD0, global system for mobile communications (global system for mobile communications, GSM)/general packet radio service technology (general packet radio service, GPRS), or 4G cellular communication, such as long term evolution (long term evolution, LTE), or 5G cellular communication. The wireless communication system 150 may communicate with a wireless local area network (wireless local area network, WLAN) using WiFi. In some embodiments, the wireless communication system 150 may communicate directly with the device using an infrared link, bluetooth, or ZigBee. Other wireless protocols, such as various autopilot communication systems, for example, the wireless communication system 150 may include one or more dedicated short range communication (dedicated short range communications, DSRC) devices that may include public and/or private data communications between autopilots and/or roadside stations.

Alternatively, one or more of these components may be mounted separately from or associated with the mobile charging device 13. For example, the memory 132 may exist partially or completely separate from the mobile charging device 13. The above components may be communicatively coupled together in a wired and/or wireless manner.

Alternatively, the above components are just an example, and in practical applications, components in the above modules may be added or deleted according to actual needs, and fig. 4 should not be construed as limiting the embodiments of the present application.

In the embodiment of the present application, the execution body of the method for controlling the mechanical arm charging may be the mobile charging device 13, or may be the cloud-side server 11, referring to fig. 5, in the system shown in fig. 5, the mobile charging device 13 and the server 11 may be in communication connection (for example, implemented by the wireless communication system 150), where the mobile charging device 13 may transmit data collected by a sensor to the server 11, and the server 11 implements the method for controlling the mechanical arm charging in the embodiment of the present application based on the data collected by the sensor by the mobile charging device 13, and transmits the relevant processing result for controlling the mechanical arm charging to the mobile charging device 13.

Next, an architectural illustration of the server 11 will be described.

The embodiment of the present application further provides a server, please refer to fig. 6, fig. 6 is a schematic structural diagram of the server provided in the embodiment of the present application, specifically, the server 600 is implemented by one or more servers, and the server 600 may be relatively different due to configuration or performance, and may include one or more central processing units (central processing units, CPU) 66 (e.g. one or more processors) and a memory 632, and one or more storage media 630 (e.g. one or more mass storage devices) storing the application 642 or the data 644. Wherein memory 632 and storage medium 630 may be transitory or persistent storage. The program stored on the storage medium 630 may include one or more modules (not shown), each of which may include a series of instruction operations on a server. Still further, the central processor 66 may be configured to communicate with a storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

The server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input/output interfaces 658; or one or more operating systems 641 such as Windows ServerTM, mac OS XTM, unixTM, linuxTM, freeBSDTM, or the like.

The mechanical arm for charging provided by the embodiment of the application can realize the plug of the charging socket to the charging port with high precision.

Next, the charging device in the embodiment of the present application will be described in terms of the morphology of the product.

Referring to fig. 7, fig. 7 is a charging scene illustration of a charging device in an embodiment of the present application, where the charging device may include a robotic arm. The mechanical arm can adopt a multi-axis serial mechanical arm to realize the position and pose control of the charging gun, and the form of the mechanical arm is not limited to serial connection, so long as the six-dimensional pose guide of the terminal charging gun can be realized. The charging device may comprise a force sensor (or may be a torque sensor) which in one possible implementation is capable of sensing forces and moments (Fx, fy, fz, tx, ty, tz) experienced by the tip of the gun head, and of sensing lateral force sensors. The charging device may include a camera capable of acquiring 2D images and/or 3D point clouds of the charging port for identifying a 6D pose of the charging dock. The charging equipment can include charging gun (including charging plug) and fixing device, and the charging equipment can include moving platform, expands charging device's movable range.

Referring to fig. 8, fig. 8 is a schematic diagram of an embodiment of a charging device provided in an embodiment of the present application, where the mobile charging device may be a fixed charging pile, a mobile charging vehicle, a mobile charging robot, or the like.

As shown in fig. 8, a charging device provided in an embodiment of the present application may include a mechanical arm 802, where the mechanical arm 802 may include: control structure and robot 802 body. The control mechanism may be the processor 131 described above.

In one possible implementation, the control mechanism may control the pose of the main body of the mechanical arm 802 to implement a charging task, where the charging task may be, for example: a charging plug 803 at the end of the mechanical arm 802 is inserted into a charging port of the vehicle.

In one possible implementation, during the beginning stage of the charging task, optionally, the charging device may receive a charging instruction (for example, may receive a charging instruction sent from a server, or directly receive a charging instruction sent by a terminal device of a user), and optionally, the charging instruction may carry information such as a position of a charged vehicle, vehicle type information of the charged vehicle, and speed charging information. The charging device may initiate execution of the charging task based on receiving the charging instruction, triggering an initiate movement state.

In one possible implementation, the charging device needs to be moved near the charging port of the vehicle to be charged to start the operation of the mechanical arm 802, so as to ensure that the charging port is within the reachable space of the mechanical arm 802, and this part can be implemented by adopting a navigation positioning mode or an automatic driving technology.

In one possible implementation, the charging device may be compatible with two charging modes, i.e. a fast charging gun and a slow charging gun, so that the charging device may be provided with a fast-changing mechanism, and the charging gun may be selected according to the task requirements of the user, where the charging gun is grabbed by adopting a magnetic structure, so as to ensure reliable butt joint of the charging gun and the mechanical arm 802, and then perform the next charging operation.

In one possible implementation, after the charging device selects the docking point, the task of moving the chassis is basically achieved, that is, the operation of the mechanical arm 802 is about to be performed, and then the approximate position of the charging stand (including the charging port, which may also be referred to as the charging port for convenience of description in this embodiment) needs to be known, so as to control the mechanical arm 802 to perform the position search of the charging port in this area. The approximate position of the charging port can be detected according to the vehicle type, the vehicle type of the charged vehicle can be identified through a charging equipment camera, the approximate position of the charging port can be calculated by combining the visual information of the charged vehicle with the vehicle model information, the position is sent to the mechanical arm 802, and the mechanical arm 802 can perform the next flow in the area, namely the searching stage.

In one possible implementation, there may also be a limit to the camera field of view, such as the field of view (FOV), due to positional errors in the resting position of the mobile platform. It is difficult to ensure that the charging port is visible within the camera at the end of the robotic arm 802, so it is necessary to perform a "search" strategy for the charging port, perform a preliminary positioning or a coarse positioning of the charging port. Alternatively, the search scheme may be implemented jointly based on a fan-shaped path plan in the kinematics of the robotic arm 802 and a target detection algorithm in visual perception.

For example, a flowchart of the search process may refer to fig. 9. Specifically, because the parking positions of the charging vehicles have deviation, and the brands, the models, the parking positions and the like of the vehicles to be charged are different, and the view angle of the camera at the tail end of the mechanical arm 802 has limitation, in order to ensure that the charging port is visible, the mechanical arm 802 is controlled to search a designated area. The control robotic arm 802 may complete the path following by generating a search path from knowledge of the general location of the charging port (e.g., vehicle parking position, charging port height, tilt angle, etc.). And synchronously detecting and tracking the target of the charging port in the searching process.

Next, the fan-shaped track search in the embodiment of the present application is described, in general, the height of the charging port of the new energy automobile is distributed in the range of 70 c-100 cm, so that the fan-shaped track search can be performed at this height in cartesian space in combination with the FOV of the camera, the search process is accompanied with a real-time charging port detection algorithm, and when the target is detected, the mechanical arm 802 is driven to move so that the target is located at the center of the field of view, which is beneficial to the visual servoing process in the next stage.

In one possible implementation, fast and high-precision detection and tracking of the charging port targets may be achieved based on deep learning. In order to improve task rapidity and realize charging port searching with high efficiency, an improved YoloV4 deep learning target detection algorithm combined with a template tracking tracking algorithm can be optionally adopted.

The improved YoloV4 algorithm mainly compresses the convolution layer number and the convolution kernel number of the feature extraction network in the original network structure in depth, the convolution layer number of the improved model feature extraction network is reduced (for example, the original 73 layers can be reduced to 22 layers), so that the loading and reasoning process speed of the model is greatly improved, and the accuracy is almost kept unchanged.

Table 1: improved YoloV4 algorithm compared to YoloV4 algorithm

The improved YoloV4 target detection algorithm can accurately detect a charging port region (box coordinates) (x, y, w and h) in an image, the region image is used as a target region to be transferred to a template tracking tracking algorithm, and when the next frame of image arrives, the tracking algorithm outputs the region with the highest correlation coefficient with the target region in the calculated image as a tracking result.

The traditional image detection algorithm cannot cope with unknown and dim underground garage and various vehicle brands and models, but the deep learning is different, is driven by massive data, has extremely strong robustness and generalization capability, but simultaneously, the prediction reasoning process of the deep learning algorithm can carry out a large amount of calculation, and requires the computational power support of hardware equipment (display card). Therefore, neither the single deep learning algorithm nor the conventional image detection algorithm can achieve the desired effect. The scheme utilizes the robustness of the deep learning algorithm to environmental changes and the generalization capability of the deep learning algorithm to strange environments, ensures that the charging port target can be successfully detected in the complex environment, simplifies and compresses the original algorithm model, improves the algorithm running speed, combines the template tracking algorithm in the traditional algorithm, solves the problem that the deep learning algorithm cannot be detected in real time on the portable industrial personal computer due to insufficient calculation force, and realizes the real-time detection and tracking of the charging port target in the complex environment. The detection result may be illustrated by referring to fig. 10, where the square area (box) is the detection result output, and the coordinates are (x, y, w, h) and represent the pixel coordinates in the image. The tracking effect can be obtained by referring to fig. 11, in which the box is a tracking area in each frame image, and the detection result of the charging port in the area is ensured.

Through the above-described search for the charging port, the control mechanism may control the pose of the robotic arm 802 so as to bring the charging port within the field of view of the camera (optionally, into the center area of the field of view of the camera). Thereafter, the control mechanism may control the pose of the robotic arm 802 based on visual servoing so as to guide the charging plug 803 on the body of the robotic arm 802 around the charging port.

The pose (position and orientation) may include a position and a pose, the position may be represented by three coordinate values in a three-dimensional coordinate system, and the pose may be represented by a state defined by rotation about each coordinate axis.

When the charging plug 803 on the main body of the mechanical arm 802 is guided around the charging port, the relative pose of the charging port with respect to the main body of the mechanical arm 802 (specifically, the position of the charging socket on the main body of the mechanical arm 802 may be set), the pose determined based on the information acquired by the vision sensor 804 may be the pose of the vision sensor 804 with respect to the charging port, and since the position of the vision sensor 804 on the mechanical arm 802 is fixed, that is, the relative positional relationship between the vision sensor 804 and the charging plug 803 is fixed, after knowing the pose of the vision sensor 804 with respect to the charging port, the relative pose between the charging plug 803 and the charging port is known.

It should be appreciated that when the relative pose of the charging port relative to the vision sensor 804 is obtained, due to the calibration error, the relative pose of the charging seat relative to the mechanical arm 802 may not be equal, and a system error may exist between the relative poses, and the system error existing between the two coordinate systems may be solved through calibration according to the variation of the relative pose of the base of the mechanical arm 802 and the charging seat.

Specifically, the charging robot is required to complete the insertion and extraction of the charging gun, so that the pose of the charging gun under the coordinate system of the mechanical arm 802 and the pose of the charging gun under the coordinate system of the charging seat represent the same space pose when the charging gun is completely inserted. This first requires knowledge of the pose relationship of the charging dock coordinate system relative to the robotic arm 802 coordinate system. This pose relationship is also dynamically changing as the base of the robotic arm 802 may change over time. The pose of the charging seat relative to the camera coordinate system can be obtained through a visual perception mode, then the pose of the charging seat relative to the mechanical arm 802 coordinate system can be obtained through hand-eye parameters transformed to the mechanical arm 802 terminal coordinate system and then through positive kinematics. Because of the objective existence of systematic errors, hand-eye calibration errors and visual recognition errors of the mechanical arm 802, the charging seat pose obtained by the method and the true value of the charging seat pose are error, and the insertion of the charging gun cannot be guaranteed generally. Because the true value of the charging seat pose cannot be obtained through other methods, the mechanical arm 802 in the embodiment of the application can be provided with the force sensor 801, so that the acquisition of the true value of the charging seat can be realized through the force sense-based self-calibration method by the information acquired by the force sensor 801, and a charging seat pose compensation matrix obtained through vision is constructed to perform observation pose compensation.

In one possible implementation, assuming that there is no gap between the charging plug 803 and the charging port, since the charging connector is a transition fit, assuming that it is true that the charging plug 803 coordinate system coincides with the charging port coordinate system, the world coordinate system is defined as the base W of the charging device, the end coordinate system is defined at the extreme end e of the charging plug 803, the charging port coordinate system is O, and the camera coordinate system is C, then there may be:

in the first step, when the butt joint is successful, the following steps are obtained:

/>

calculation method using positive kinematics:

it can be seen that:

to allow for some tolerance, there may be some but very little error between the charging port and the charging plug 803, and the correct insertion pose may be determined by a force sense self-calibration method

Complete the insertion of the correct true value into the pose +.>

Is a search of (a). The specific method can be exemplified as follows:

(1) The charging plug 803 is used to plug into the charging port.

(2) The pose of the charging plug 803 is flexibly adjusted so that the applied moment is minimized.

(3) Program control adjusts the pose of charging plug 803 so that the other two coordinate directions are stressed and minimized, except for the axial direction of charging plug 803.

(4) Translation in two directions (x, y directions) perpendicular to the axis of charging plug 803 results in minimum and maximum values in the x, y directions

(5) The end charging plug 803 of the mechanical arm 802 is translated to stop near the median in the x, y direction.

Thus, the true value insertion pose can be found through force feedback

In the second step, the end of the arm 802 can be moved to the optimal observation point on the premise that the base of the arm 802 is consistent with the first step.

At the optimal observation point, the end pose of positive kinematic calculation can be obtained:

^W T _e ＝ ^W T ₆ (θ ₁ ) ⁶ T _e

by observing the charging position and the pose at the optimal observation point

Terminal pose obtained by vision:

wherein the method comprises the steps of

Is a vision measurement error, < >>

Error of hand-eye parameter, +.>

The measurement value of the hand-eye calibration is equal to the two formulas, so that the compensation quantity can be obtained:

after the compensation amount is obtained, the target charging port can be observed at the optimal observation point, and the pose of the expected charging plug 803 can be obtained by the following formula:

the formula is combined:

the two formulas are equal to each other, and the following can be obtained:

/>

and then can solve the expected joint angle after insertion

(in the case where the robot arm 802 body is composed of a plurality of joints, the pose of the robot arm 802 body may be controlled by joint angles).

The application provides a force sense calibration method for relative pose errors of a mechanical arm 802 and a charging port, wherein a zero position where a charging plug 803 of the mechanical arm 802 and the charging port are completely inserted is obtained through a force sense self-calibration search technology, and a method for quantitatively compensating errors of the visual observation of the relative pose errors of the charging port and the mechanical arm 802 is provided according to the zero position, so that the problem that the charging plug 803 cannot be inserted due to the absolute pose errors of the charging port, which are collected by deformation errors, visual identification errors, hand-eye calibration errors and the like of the mechanical arm 802, is solved.

In one possible implementation, the robotic arm 802 may include a base station (or base), and a robotic arm 802 body, which may be referred to as an arm. The arm may include a plurality of joints that are connected in sequence and that have rotational degrees of freedom with respect to each other. On the end of the body of the robotic arm 802 (the end remote from the base), a force sensor 801 and a charging socket may be mounted (e.g., may be mounted sequentially). In the embodiment of the application, the control mechanism may control the pose of the main body of the mechanical arm 802 based on a visual servo manner, so as to guide the charging plug 803 on the main body of the mechanical arm 802 around the charging port.

In one possible implementation, the control mechanism may acquire image information acquired by the vision sensor 804 for the region of the charging port in real time; obtaining the real-time pose of the main body of the mechanical arm 802 according to the position information of the charging port in the image information; and controlling the pose of the main body of the mechanical arm 802 according to the real-time pose so as to guide a charging plug 803 on the main body of the mechanical arm 802 to the periphery of the charging port.

Because the image information acquired in the visual servo is acquired in real time, the real-time pose of the mechanical arm 802 main body can be obtained by solving the image information acquired in real time (the real-time pose can be the relative pose between the tail end of the mechanical arm 802 main body and the charging port of the vehicle to be charged), and because the conditions of base movement, obstruction and the like possibly occur when the pose of the mechanical arm 802 main body is controlled, errors are caused in the pose of the mechanical arm 802 main body, the pose of the mechanical arm 802 main body can be corrected based on the real-time pose, the pose recognition precision of the mechanical arm 802 main body can be improved, and further the charging plug 803 on the mechanical arm 802 main body can be guided to the periphery of the charging port more accurately.

The mechanical arm 802 is controlled to perform quick visual servo, so that quick guidance of the charging seat is realized, a stable control law needs to be designed to realize accurate and quick closed-loop control, and the mechanical arm 802 control and visual recognition results need to be highly fused to design a control algorithm from a system angle, so that the mechanical arm 802 moves to further ensure that the terminal charging gun can quickly approach a charging port, and preparation is made for next-step force control plug.

In one possible implementation, pose recognition (e.g., 6D pose recognition) of the charging port is the most important link for implementing the charging plug function, and precise position recognition is a precondition and basis for performing plug. At the same time, the algorithm is required to have high robustness, such as a dimming condition for garage scenes. The 6D position mainly refers to the position (x, y, z) and pose angle (Roll, pitch, yaw) of the charging port in the camera coordinate system at the end of the mechanical arm 802; optionally, RGBD depth camera can be combined with neural network algorithm to realize 6D pose recognition of the charging port, the recognition precision requirement reaches +/-1-2 mm, and the angle precision is within +/-3 degrees (the conversion relation between coordinate systems can be shown by referring to FIG. 16).

In one possible implementation, a method of combining three-dimensional key point detection with a two-dimensional key point detection network is adopted to obtain a high-precision identification result in a low-contrast charging port detection environment.

Wherein visual servoing is a closed-loop control method performed by visual combination with motion. The main function is to realize the guiding and movement of the charging gun at the tail end of the mechanical arm 802 to the charging port. The general flow is to take the pose result of the visual identification of the charging gun/charging seat as input, calculate the driving error by adopting the visual servo feedback control idea, and realize the accurate guidance from the tail end of the mechanical arm 802 to the target pose by adopting the visual servo based on the pose. The self-adaptive gain method is adopted to accelerate the adjustment speed in small errors and realize quick visual servo. At the end of this step the robotic arm 802 will be at the optimal observation point where the most accurate charging port pose has been obtained. The robotic arm 802 will now prepare for the next force-controlled insertion based on guiding the charging jack into proximity with the charging port.

More specifically, the accuracy of the target observation pose can be ensured only by ensuring certain stability of the optimal observation point. However, the pose of the mobile robot base relative to the charging stand may be constantly changing. In order to ensure the consistent pose of the optimal observation point relative to the charging seat under the change, the aim is fulfilled by adopting a quick visual servo of the pose. In one possible implementation, the visual servoing concept of pose feedback may be employed to design a joint velocity control law based on a li-stability criterion, to achieve accurate guidance of the distal end of the robotic arm 802 to the target location.

The specific implementation method is that when the system finishes searching the charging seat, the charging seat is in the field of view of the hand-eye camera, so that the pose matrix from the charging seat to the camera can be obtained, and an error difference value is obtained by comparing the pose matrix with a pose matrix of an expected set optimal observation point. Assuming the current pose vector

Wherein->

The translation vector of the charging stand coordinate system relative to the camera coordinate system is represented, thetau represents the direction vector of the charging coordinate system relative to the camera coordinate system, and the expected pose vector is +.>

0 indicates that the direction of the camera and the direction of the charging coordinate system will remain the same upon convergence. Then define an error value: e=s ^* -s。

By constructing an equation of the proportional relation of the differentiation of the error and the error in-lambda, the error is ensured to decay in an exponential form. This is a simpler visual servo control law:

this control law usually converges too slowly, and when the error is small, a servo control law of adaptive gain can be set:

wherein lambda (|e|) is error i e i. As shown in Table 2, a normal type of fuzzy can be constructedAnd the membership function A (e) performs fuzzy control, and the parameters of the fuzzy control law are adjusted through experiments to perform self-adaptive gain control. According to the established fuzzy aggregation, the fuzzy rule value satisfies a from minimum, small, medium, large and maximum change ₁ >a ₂ >a ₃ >a ₄ >a ₅ Number a ₁ ～a ₅ Representing situation scoring of error e, wherein the smaller the error is, the higher the score is. And substituting the error input values into membership functions to solve membership degrees respectively, and performing defuzzification according to a membership degree maximum principle. If the membership degree of the error input value in a certain set is maximum, a corresponding rule value is selected as a control output quantity, so that the attenuation value becomes larger when the error is small, and the attenuation value becomes smaller when the error is large.

TABLE 2 fuzzy membership functions and fuzzy rules

In each servo period, the value of the pose error differential can be obtained according to the pose error of visual perception through an exponential decay equation, and the pose error differential can be expressed as the product of the interaction matrix and the speed of a camera:

the error relationship of camera speed and pose vector can be expressed as: wherein the method comprises the steps of

For camera speed +.>

For camera angular velocity: />

The camera speed can be represented by the product of the jacobian matrix and the joint speed of the mechanical arm 802, so that the servo speed of the joint can be solved according to the pose error

Alternatively, the obstacle detection may be performed around the mechanical arm 802, and the reaction force obtained according to the distance from the obstacle by the potential field method is used as the deceleration Vo, and when there is an obstacle, the potential energy generated by the obstacle will push the mechanical arm 802 away. Definition:

Where d is the shortest distance between the end of the robotic arm 802 to the environment, the smaller the distance, the greater the potential energy, and μ is a constant. Definition:

the above formula shows that the acceleration generated by the potential field method will be directed in a direction of decreasing potential energy of the potential field. The total joint speed of the mechanical arm 802 can be obtained as follows:

the angular velocity can be used as a control input to the mechanical arm 802 to send a control command, so that closed-loop control is formed, and the mechanical arm 802 is driven to operate for a visual servo control period.

The visual servoing described above continues to run until convergence, at which time the robotic arm 802 moves the camera to the optimal point of view.

Alternatively, the visual servoing procedure may be as shown in fig. 12.

The present application may employ desired pose compensation and jacobian calculated compensation in visual servoing.

(1) Pose compensation is desired. When the visual servo converges, the expected observation pose of the actual pose of the camera is obtained. Namely:

here, the

Representing the measured pose of the camera, and the ideal camera c pose has a compensation amount +.>

The setting of the desired value is thus compensated according to the compensation amount and then used as the actual desired value.

(2) Compensation of jacobian calculation. Because the actual camera pose is adopted as a control parameter, the jacobian matrix can be:

The observation compensation after reaching the optimal observation point is described in the above embodiments, and is not described here again.

Optionally, in one possible implementation, the vision sensor 804 is disposed on the body of the mechanical arm 802.

Alternatively, in one possible implementation, the vision sensor 804 may be disposed not on the body of the robotic arm 802 but at a location corresponding to the third party perspective.

In one possible implementation, in making the above-described determination of the relative pose, it is desirable that the vision sensor 804 acquire image information for the charging port and determine the relative pose between the charging port with respect to the vision sensor 804 based on the image information.

In one possible implementation, the image information may include a 2D image and a 3D point cloud, and the control mechanism may further extract a first 3D position corresponding to the charging port according to the 2D image and a preset charging port 3D model; extracting a second 3D position corresponding to the charging port according to the 3D point cloud; and fusing the first 3D position and the second 3D position to obtain the pose of the charging port. The pose of the charging port may be understood as the relative pose of the charging port with respect to the vision sensor 804.

In one possible implementation, the visual sensor 804 on the charging device may be a depth sensor for acquiring depth images. The depth sensor may be described, for example, but not limited to, as follows: a depth camera, a time of flight (TOF) camera, or a LiDAR, a photographic scanner, or a LiDAR detection and measurement (light detection and ranging, liDAR) sensor. The control mechanism collects the 3D point cloud by, but not limited to, the following:

A. and obtaining a 3D point cloud based on the acquired RGB image and depth image.

In this embodiment of the present application, the control mechanism may collect an RGB image and a depth image through a sensor disposed on the main body of the mechanical arm 802, where the RGB image and the depth image refer to an RGB image and a depth image formed by a surrounding environment that can be perceived by the sensor at a current time and a current position. For example, the RGB image and the corresponding depth image may be acquired by a depth camera disposed on the charging device, or the depth image may be acquired by a depth sensor disposed on the charging device, and the corresponding RGB image may be acquired by a camera module (e.g., a camera) disposed on the charging device.

After the sensor acquires the RGB image and the depth image of the charging equipment at the current moment, firstly, carrying out instance segmentation on the RGB image, for example, carrying out instance segmentation on the RGB image by adopting SOLOv2, blendmask and other algorithms, outputting an instance segmentation result, wherein the instance segmentation result comprises a classification category of each region on the RGB image, a confidence coefficient (not more than 1) belonging to the classification category and a segmented mask (mask), and defining the instance segmentation result as Mi. Then, the data belonging to the charging port may be selected, and the RGB image (may be referred to as a processed RGB image) in which the example division and redefining of the pixel values are implemented is superimposed with the depth image, each partial region in the processed RGB image has depth information, specifically, since the mask corresponding to each partial region of the processed RGB image occupies a region defining pixel 1 on the RGB image and the remaining regions define pixel 0, the processed RGB image is multiplied with the depth image, and since the region with the pixel value of 0 is still 0 after multiplication, the multiplied result only retains the region corresponding to each region in the RGB image and the depth information corresponding to each region. And then, recovering the 3D point cloud of the charging port through the camera internal parameters.

It should be noted that, in some embodiments of the present application, the filtering operation may be performed on the instance segmentation result and the depth image, for example, the outlier of the instance segmentation result is reduced by morphological filtering, the cavity portion of the depth image is eliminated by depth smoothing, and so on, where the purpose of the filtering is to remove noise in summary.

B. A plurality of 3D point clouds are obtained based on the acquired RGB image and the original laser point cloud data.

In this embodiment of the present application, the charging device may collect, by using a sensor disposed on the charging device, an RGB image and original laser point cloud data, for example, the charging device may be disposed with a sensor such as a laser radar and a common camera, and collect the RGB image by using the camera, and collect the original laser point cloud data by using the laser radar, in this case, it is not necessary to obtain a depth image, and only needs to divide an example of the RGB image according to the above similar process, and then corresponds the result of division of the example to the original laser point cloud data obtained at the same time, so as to obtain a 3D point cloud corresponding to the charging port.

In one possible implementation, the visual sensor 804 may be used to obtain two-dimensional image and three-dimensional depth point cloud data, where the two-dimensional image may be used to obtain the 6D pose of the target by detecting the keypoints and by corresponding to the same name keypoints of the three-dimensional model, but this relies on high-precision detection of the two-dimensional keypoints, while introducing reconstruction errors when recovering the 6D information by 2D-3D correspondence. The garage scene has weak light condition, different charging port depths are also different, and certain chromatic aberration exists. Meanwhile, the three-dimensional point cloud can be matched with the three-dimensional model of the target object to obtain the 6D pose of the target, but the accuracy of the point cloud is slightly poor, and the reconstruction information of the two-dimensional image and the three-dimensional point cloud is fused, so that the positioning with higher accuracy is realized. In order to accurately insert the charging port, an accurate pose of the charging port in a coordinate system of the mechanical arm 802 needs to be obtained, the pose of the charging port relative to the sensor can be observed through the sensor, the pose of the sensor relative to the tail end of the mechanical arm 802 is converted to the tail end of the mechanical arm 802 according to the installation position of the sensor, and the relative pose relation of the tail end of the mechanical arm 802 relative to the base of the mechanical arm 802 is converted through positive kinematics. A schematic of the convolutional neural network structure may be shown with reference to fig. 14.

Table 3 2D key point and sequence number detection convolutional neural network structure

/>

For example, a feature map set (feature maps) with high characterizability can be extracted on an image by a feature extractor consisting of layers 1 to 18, and then converted into a vector with a dimension of 14 by using three fully connected layers. And calculating a Mean Square Error (MSE) between the vector and a vector formed by 2D key point true values on the image as a loss function of the network model, and continuously inputting data and iterating the model until loss converges. Finally, a high-precision key point detection algorithm model can be obtained.

By way of example, three-dimensional coordinates of seven key points in the charging port coordinate system can be obtained as shown in the following table:

table 4 three-dimensional coordinates in charging port coordinate system for seven key points of charging port

For a 3D point cloud key point detection network, in an actually observed three-dimensional point cloud, an expected key point may not be observed, the direction vector from each point to the key point is detected through the key point detection network, and the accurate coordinate of the key point is indirectly obtained through voting. Firstly, selecting a plurality of standard key points from a three-dimensional point cloud model, and obtaining the standard key points through a furthest point sampling method. The method comprises the steps of obtaining a first point through a random sampling mode for the first time, selecting a point with the farthest distance from a selected point set from the rest points every time, and obtaining N points as standard key points. And secondly, after the key points are obtained, generating a training data set by using the standard data key points and the point cloud after the manual generation transformation for training. The input point cloud is firstly subjected to feature extraction through a PointNet++ network, a multi-layer fully connected neural network is arranged for each point to calculate the deviation from each point to K key points, then a 3D Hough voting network is adopted to perform voting clustering according to the deviations, and the clustered point set is used for solving to obtain accurate three-dimensional key points (the specific flow can be shown by referring to FIG. 17). When model training is performed, the two functions can be weighted by alpha and beta according to errors of two-dimensional detection and three-dimensional detection, and the total loss function is obtained as follows:

Loss＝∑ _R,t (α||P _2d -proj(RP _3dgt +t)|| ₂ +β||P _p3d -(RP _p3dgt +t)|| ₂ )；

Wherein P is _2d Is the key point of image detection, P _3dgt Is the three-dimensional point coordinate under the charging port coordinate system, R, t is the transfer matrix from the charging port coordinate system to the camera coordinate system, and P _p3d Is the three-dimensional coordinates of the key points detected in the three-dimensional point cloud under the camera coordinate system, P _p3dgt Three-dimensional coordinates of key points in the three-dimensional model. The algorithm can calculate the initial value of R and t through the 2D-3D correspondence of the image, and then pick upThe solution of the optimal pose matrix is obtained by gradient descent by the L-M method (for example, see fig. 13).

In one possible implementation, the 6D post (position and posture) identification of the charging seat is a key link of an unmanned charging task, so that accurate guiding of the mechanical arm can be performed only by ensuring the 6D pose of the charging port to be accurate, and further, the compliant plugging and unplugging is performed by adopting a force control technology. Because the standard slow charging port consists of seven holes, the circle centers of the seven holes of the charging port are selected as key points, and the key point detection is carried out on the target area through an artificial intelligence algorithm. This scheme converts the 7 keypoint detection problem on a two-dimensional image into a 14-dimensional regression problem of 14 values, namely a 14-dimensional vector of 7 keypoint coordinates (x, y). Based on the VGG16 network structure, the parameters of the last three full-connection layers in the network structure are modified to enable the output dimension of the last layer to be 14, and the original softmax loss is changed into the mean square error loss. The input of the final network is a color image, and the input is a vector with the dimension of 14, namely the coordinate values of 7 key points on the image.

The method comprises the steps that a point cloud is downsampled by adopting a furthest point sampling method, except for first random sampling, points with the furthest distance from a selected point set are selected from the rest points every time, the sampled points are used as downsampled versions of dense point clouds, three-dimensional registration is carried out on the downsampled versions of the dense point clouds and standard point clouds, a three-dimensional point corresponding sequence is obtained, and the distance between a standard three-dimensional key point and a target coordinate system key point after R and t transformation is used as a three-dimensional target function;

obtaining two-dimensional pixel coordinates of a center point of a round hole of a charging seat by adopting an improved VGG two-dimensional key point detection network, and obtaining the number of each point through the mutual position relation of the center points (for example, refer to FIG. 15); and obtaining the three-dimensional physical coordinates of the center point of the round hole under the target coordinate system according to the charging seat standard CAD model. The two-dimensional key pixel points (under the pixel coordinate system) and the three-dimensional physical points (under the target coordinate system) are in one-to-one correspondence through the numbering sequence, the corresponding back projection pixel points are obtained by back projection of the three-dimensional physical points under the pixel coordinate system, and the distances between the two-dimensional key points and the back projection points are compared to form a two-dimensional loss function item;

the two functions are weighted by alpha and beta according to the two-dimensional detection and the three-dimensional detection errors, and the total loss function is obtained.

The charging port pose accurate identification method based on 2D image-3D point cloud information can accurately detect two-dimensional key points of a charging port in a sub-pixel level by adopting a 2D depth image detection network, reliably detect offset of the three-dimensional key points of the charging port based on a 3D Hough network by adopting a three-dimensional key point detection neural network, and calculate the accurate pose of the charging port by adopting a technology of minimizing projection errors of the two-dimensional key points and the three-dimensional key points through nonlinear optimization, so that the problem of pose high-accuracy identification of the charging port under the conditions of large illumination change and low target contrast is solved.

By the visual servo method, the charging plug 803 can be guided to the vicinity of the charging port, for example, the accuracy range is within the error range of 1 to 2mm and 2 °, and since the previous planning has a certain error based on vision and the collision safety is considered, the control mechanical arm 802 cannot contact with the charging port at this time, so that a movement is required to be performed for a certain period according to the visual information, the force sensor mounted on the mechanical arm 802 main body can sense that the force sensor contacts with the charging port, and then the adaptive adjustment stage based on the force sensing can be performed (for example, refer to fig. 18).

The application provides a visual servo guiding method with self-adaptive error gain, and provides visual servo speed instruction generation combined with potential field obstacle avoidance. The visual servo needs to interact with the environment in practical application, but cannot collide with the environment to cause damage, and a new rejection speed factor is introduced by adopting a method of combining potential field obstacle avoidance, so that the safety of the visual servo is effectively ensured. Secondly, a visual servo control law of self-adaptive gain is provided, and a large attenuation coefficient under a small error is realized by setting an inverse fuzzy relation between an attenuation index and an error amplitude, so that the convergence speed of visual servo is effectively accelerated. Thirdly, the visual servo based on the pose is adopted, and because the position and pose information are used as feature vectors, the visual servo based on the image contains more accurate information than the visual servo based on the image, the mechanical arm 802 can be effectively guided to the expected optimal observation point, and the problem that a collision-free guiding camera stays at the optimal observation point with the pose relatively fixed under the condition that the pose of the mechanical arm 802 and the charging port is possibly changed continuously is solved.

After the charging plug 803 contacts the charging port, the control mechanism may acquire force information collected by the force sensor 801, where the force information may indicate a force (or moment) of the charging port on the charging plug 803, and control a pose of the main body of the mechanical arm 802 according to the force information, so as to insert the charging plug 803 into the charging port.

In the existing scheme for performing pose control only based on vision, because of errors of the vision, and the vision camera cannot continuously observe the charging port when the charging plug 803 is near the charging port, if the charging plug 803 is plugged and unplugged only by adopting the vision pose observed at the optimal observation point, the situation that the charging plug 803 cannot be plugged to the bottom and is wedged can possibly occur, the pose control of the mechanical arm 802 is performed by acquiring the acting force information acquired by the force sensor 801, and the charging plug 803 can be adjusted to adapt to the plugging direction of the charging port according to the force sense under the situation that the charging plug 803 is partially plugged and the vision information cannot acquire errors, so that the accuracy and the reliability of plugging and unplugging tasks are realized.

Next, the force sensor 801 in the embodiment of the present application is described.

In one possible implementation, the main body of the mechanical arm 802 is provided with a force sensor 801, where the force sensor 801 is used to collect force information of a contact surface between the charging plug and the charging port, and the contact surface includes a side surface of at least one charging head of the plurality of charging heads, and the side surface is a surface parallel to an axial direction of the charging head.

In one possible implementation, the force information includes a force in a radial direction of the charging head. For example, the effort information may include 6-dimensional effort information.

In one possible implementation, the force sensor 801 is fixed to the charging plug 803 (with a side that may be on at least one charging head); alternatively, the mechanical arm 802 includes a joint having a rotational degree of freedom, the control structure is configured to control a pose of a body of the mechanical arm 802 by controlling rotation of the joint, and the force sensor 801 is fixed to the joint; or, the force sensor is fixed between the charging plug and the mechanical arm main body.

In one possible implementation, the force sensor 801 may have one or more force detection units, A-D conversion circuitry, and signal processing circuitry. The force detection units each include a force detection element and an amplification circuit that amplifies an output signal of the force detection element. The amplifying circuit supplies the amplified signal Va to the a-D conversion circuit. The a-D conversion circuit supplies the digitally converted signal Vd to the signal processing circuit. The number of force detection units is set according to the number of detection axes detected by the force sensor 801 and the kind of force about each detection axis. For example, the force sensor 801 uses twelve force detection units when detecting the magnitude of forces parallel to three detection axes orthogonal to each other and the magnitude of moments about the three detection axes.

In one possible implementation, various elements such as piezoelectric, strain gauge, or optical elements may be used as the force detection element.

In one possible implementation, the amplifying circuit is configured as an integrating circuit that integrates and converts the electric charge output from the force detecting element into the voltage signal Va. Specifically, the amplifying circuit has an operational amplifier, a capacitor, and a switching element. The negative input terminal of the operational amplifier is connected to the electrode of the force detection element, and the positive input terminal of the operational amplifier is grounded to ground (reference potential point). The output terminal of the operational amplifier is connected to the a-D conversion circuit. The capacitor is connected between the negative input terminal and the output terminal of the operational amplifier. The switching element is connected in parallel with the capacitor. The switching element performs a switching operation in accordance with an on/off signal supplied from the signal processing circuit.

In one possible implementation, when the switching element is off, the charge output from the force detecting element is stored in a capacitor, and its voltage Va is output to the a-D conversion circuit. On the other hand, when the switching element is turned on, both terminals of the capacitor are short-circuited. As a result, the charge stored in the capacitor is discharged and becomes zero, and the voltage Va output to the a-D conversion circuit becomes 0 volt. The operation of turning on the switching element corresponds to the reset of the force sensor 801.

In one possible implementation, the force conversion section may convert the outputs of the plurality of force detection units into F signals (i) representing forces about the plurality of detection axes and output. The force signal F (i) is also simply referred to as "force F (i)". Alternatively, these forces F (i) comprise forces Fx, fy, fz parallel to three detection axes orthogonal to each other and moments Tx, ty, tz about the three detection axes. The force F (i) also includes a magnitude Fmag of a force vector having forces Fx, fy, fz parallel to the three detection axes as three-axis direction components and a magnitude Tmag of a moment vector having moments Tx, ty, tz about the three detection axes as three-axis direction components. The magnitude Fmag of the force vector is the square root of the quadratic sum of the three forces Fx, fy, fz and the magnitude Tmag of the moment vector is the square root of the quadratic sum of the three moments Tx, ty, tz (the schematic of the x, y, z axes can be seen with reference to fig. 19).

In one possible implementation, the charging plug 803 includes a plurality of charging heads, the charging port includes a plurality of charging holes, and the plurality of charging heads are matched with the plurality of charging holes one by one; the control structure may determine pose adjustment parameters according to the acting force information, and correct the pose of the main body of the mechanical arm 802 from a first pose to a second pose according to the pose adjustment parameters; the first pose and the second pose are different, and the second pose is a pose of the main body of the mechanical arm 802 when each charging head is aligned with a matched charging hole.

In one possible implementation, based on the effort information, it may be determined whether the body of the robotic arm 802 is in the correct pose (e.g., the second pose may be the correct pose). For example, when the force information indicates that there is a non-axial pressure of the charging port against the charging plug 803, it may be determined that the robotic arm 802 body is not in the correct pose, in which case at least one of the plurality of charging heads is not aligned with the matching charging hole, i.e., the central axes of the charging heads do not coincide with the central axes of the matching charging holes or the deviation is greater than a threshold value. For example, reference may be made to a, b, c, d in fig. 22.

In one possible implementation, the first pose is a pose of the body of the robotic arm 802 when a portion of the charging plug 803 has been inserted into the charging port and at least one of the plurality of charging heads is not aligned with a matching charging hole.

In one possible implementation, the aligning includes: the projection range of the charging head to the charging hole direction is in the range of the matched charging hole, and the central axis of the charging head coincides with the central axis of the matched charging hole or the deviation is smaller than a threshold value.

In this embodiment, when the main body of the mechanical arm 802 is not in the correct pose, the pose of the main body of the mechanical arm 802 may be adjusted based on the acting force information, so that the main body of the mechanical arm 802 is in the correct pose.

In one possible implementation, the control mechanism may correct the pose of the robotic arm 802 body (to a second pose, that is, a correct pose) according to the force information when the robotic arm 802 body is in the first pose, where the robotic arm 802 body may insert the charging plug 803 into the bottom of the charging port in a correct manner with less resistance.

For example, referring to fig. 22, a, b, c, d in fig. 22 may be considered a first pose and e may be considered a second pose.

Referring to fig. 23, three curves in fig. 23 are resultant forces applied to the charging plug 803x, y, z in three directions, and as can be seen from the graph of the contact force, the contact force in the x and y directions is gradually reduced after the force control algorithm is used, and as the insertion process is reduced to the vicinity of the zero point, it is illustrated that the self-adaptive pose adjustment greatly reduces the pressure generated by the charging port in the x and y axis direction on the gun body, and after the force control algorithm is used, the insertion contact force in the z direction is gradually reduced from 80N to 60N at the initial moment, and the insertion force is gradually reduced and kept stable after the self-adaptive adjustment stage until reliable insertion is illustrated. From the stress curve, the charging plug 803 is quickly and adaptively adjusted according to force sensing in the whole insertion process, and the resultant force in all directions is in a convergence state, so that good compliance is ensured.

In one possible implementation, the control mechanism may determine pose adjustment parameters based on the force information, which may be used to control the direction and/or rate of movement (e.g., direction, rate, or both) of the charging plug 803.

Next, how the control mechanism determines the pose adjustment parameters based on the force information and controls the pose of the body of the robot arm 802 based on the pose adjustment parameters will be described in connection with a specific example.

In one possible implementation, the direct force feedback combined with the improved admittance control technology can be adopted to perform compliant parameter adjustment by referring to the principle of the far compliant center equipment, so that six-degree-of-freedom compliant characteristics of charging plugs 803 of different types in the inserting and extracting processes are realized, and reliable, safe and rapid plugging operation of charging ports is ensured. Wherein, can divide into two stages with inserting the process, the free motion of berth to charging mouth port, charging mouth port to the compliant insertion motion of bottom. In the free movement stage, the plug is accurately sent into the charging port by utilizing position servo control; then, the admittance control is used to generate compliant motion of the end of the charging plug 803, and the correction of the tool end speed of the mechanical arm 802 is obtained through force/moment feedback and a designed admittance matrix, so that the ideal insertion speed is changed to generate compliant insertion behavior. The whole inserting process integrates the visual perceived pose information, force, moment, position, speed and other information, so that reliable inserting and extracting can be realized.

In one possible implementation, the plugging link is a core link of intelligent charging, and because of the physical contact between the charging plug 803 and the charging port, there is a force change, although the vision of the preamble process has given a more accurate target pose of the charging port, a certain error still exists to cause a larger contact force, and meanwhile, force interaction generated by different processing precision, fatigue degree, wear degree and the like in the process of insertion is more complex, so that in order to ensure the safety and reliability of the final insertion process, active compliance control based on force sensing is necessary. The whole force control plug flow is required to receive the preamble visual information, contact state judgment is required to be carried out by combining the motion state and force sensing, and the self-adaptive adjustment of the charging plug 803 head can be realized by adopting an active compliance control algorithm after the plug is inserted.

For example, the active compliant plug coordinate system may be defined as having a coordinate origin located at the geometric center of the end face of the charging plug 803, a z-axis perpendicular to the end face and pointing in the direction of insertion of the charging plug 803, a y-axis located upward within the end face, and an x-axis determined by the right hand rule.

The basic principle of admittance control is to set the ideal insertion speed of the charging plug 803 to be V _ideal ＝[0,0,v _z ,0,0,0] ^T I.e. at speed v _z And moves linearly along the z-axis. If the contact force/moment location, the actual charging plug 803 reference speed is calculated in Cartesian space as follows:

V _actual ＝V _ideal +A _admittance W _contact ；

wherein V is _actual Is the actual charging plug 803 reference speed, W _contact Is the contact force/moment, A _admittance Is an admittance matrix.

The principle of the charging device implementing the admittance algorithm is shown in fig. 20. Selected admittance matrix A _admittance The actual reference speed of the charging device in the Cartesian space is obtained by adding the flexible speed correction of the charging plug 803 in the Cartesian space to the ideal insertion speed according to the contact force/moment measurement given by the force/moment sensor. The inverse Jacobi matrix is then multiplied to obtain a reference velocity for each joint, thereby controlling the charging device to produce compliant insertion motion on the velocity profile.

When the contact force/moment exists, the RCC device can flexibly adjust the plug pose by itself to realize the alignment with the jack, so as to avoid the generation of excessive contact force/moment. By selecting A _admittance Admittance with RCC deviceThe form of the matrix, i.e.,

where k is the stiffness of the three symmetrical springs of a standard RCC device, α is the angle between the central axis of the regular tetrahedron formed by the three symmetrical springs and the edge of the spring, l is the distance between the compliant far center and the end face of the charging plug 803,

The rotational stiffness of the RCC device coordinate system is respectively x-axis, y-axis and z-axis which are positioned on the flexible far center and are parallel to the active flexible plug coordinate system. Fig. 21 is a schematic diagram of a standard RCC device.

Therefore, by adopting the improved admittance control algorithm, the charging plug 803 generates a compliant behavior similar to that of the RCC device, so as to realize a compliant insertion process and avoid the phenomena of overlarge contact force, jamming, blocking and the like.

In the embodiment of the application, the binding force perception model and the adaptive adjustment of admittance control parameters realize the adaptive adjustment of the position and the speed of the tail end of the TCP according to the friction force of the plugging force (force and moment), realize the high tolerance characteristic of position errors, improve the plugging success rate, realize the minimum plugging force by the flexible characteristic, improve the plugging reliability and the safety, have stronger parameter adaptive capacity and can effectively ensure the plugging completion time; the visual end position sensing and the plugging force sensing are fused, a 'full-insertion' mathematical characterization model is solved, whether full insertion is performed or not is effectively judged, and the task reliability is improved; the reliable pulling out is realized by adopting the compliant force control technology in the pulling out process in the same way, and the reliability of the whole plugging task is ensured.

Fig. 22 shows the relationship between the gun head and the charging port of the charging plug 803 in the process of self-adaptive active pose adjustment and insertion, the a state is the stop position of the charging plug 803 given by visual servo, it can be seen that certain position and pose errors (schematic diagram, smaller actual error value) exist, then the b state is reached by free motion (a certain search path design can be performed here to enhance the contact range and speed), namely contact detection, the self-adaptive adjustment stage of the pose to be entered after contact state detection is shown as a graph c, finally the complete reliable insertion state of e is reached, and the processes c and d continuously and circularly run in the process until the reliable insertion condition is met to complete the task. Therefore, the charging plug 803 is adaptively adjusted along with the contact state and the contact in the whole insertion process, so that the continuous friction force reduction in the insertion process is ensured, and the reliability and the safety are ensured by reducing the lateral resistance.

In one possible implementation, the control structure may control the pose of the mechanical arm 802 body after correcting the pose of the mechanical arm 802 body from the first pose to the second pose, so that the charging plug 803 moves toward the bottom of the charging hole and in a direction along a central axis of the charging head.

After correcting the pose of the mechanical arm 802 main body from the first pose to the second pose, the charging plug 803 is already in an aligned pose with the charging port, in which case the charging plug 803 may be moved towards the bottom of the charging hole and along the direction of the central axis of the charging head until the bottom, and in one possible implementation, the control structure may control the mechanical arm 802 main body to maintain a stationary state based on the force information indicating that the charging port has only resistance to the charging head along the axial direction of the charging plug 803, thereby completing the task of inserting the mechanical arm into the charging port.

After the charging is completed, the reverse movement of the insertion process can be performed, and the pose of the charging plug 803 in the pulling process can be adaptively adjusted by adopting an active flexible technology. After the mechanical arm 802 is pulled out, the mechanical arm 802 is reset according to the setting, so that the mechanical arm 802 is ensured to be contracted to a smaller space, and collision to other objects such as vehicles in the chassis moving process is avoided. After the mechanical arm 802 is retracted, the plugging task is completed, and the charging and charging equipment is moved to a designated place according to the system task setting. Waiting for the next task instruction.

In one possible implementation, the target interface is connected with an initial charging plug before the control structure controls the pose of the mechanical arm main body according to the acting force information; the control structure is also used for controlling the pose of the mechanical arm main body and the connection state of the target interface, and the initial charging plug is detached from the target interface and replaced by the charging plug; the target charging plug is a plug supporting quick charging, and the charging plug is a plug supporting slow charging; or the target charging plug is a plug supporting slow charging, and the charging plug is a plug supporting fast charging.

In one possible implementation, the charging device further includes: a plug receiving area including a target receiving location for setting the charging plug; the control mechanism can control the connection state of the target interface and detach the initial charging plug from the target interface; and controlling the pose of the mechanical arm main body, moving the target interface to the target storage position, controlling the connection state of the target interface, and connecting the target interface with the charging plug.

According to the embodiment of the application, through vision and force sense fusion, the requirement on the visual positioning precision can be effectively reduced, and the cost is reduced. The method is characterized in that the force compliance control can adjust the pose of the charging gun according to the force feedback condition, even if errors exist between the charging gun and the charging seat, the charging gun can be identified from the force feedback, and adverse effects caused by the identification errors of the charging seat are eliminated by adjusting the pose of the charging gun. Thus, the fusion of visual forces avoids the use of expensive structured light cameras, reducing costs while reducing the bulk and weight of the tip mechanism. In addition, the embodiment of the application also overcomes the influence of dragging the cable. Because the dragging cable can influence the pose of the charging gun, the method of binding force sense can quantitatively observe the influence of the dragging cable force through identification and apply force compensation when in control, and effectively overcomes the influence of the dragging cable. The charging port can be quickly found by adopting a visual method, the high-precision identification of the charging seat pose is realized by fusing 2D-3D information, the whole process time from the mechanical arm 802 to the charging port is reduced, and meanwhile, the time for correctly inserting and extracting the pose by force control searching is reduced due to higher visual precision, so that the whole flow speed is greatly improved.

In addition, the embodiment of the application further provides a method for controlling the charging of the mechanical arm, which can be applied to the control structure of the mechanical arm 802 described in the foregoing embodiment, wherein a force sensor 801 and a charging plug 803 are arranged on a main body of the mechanical arm 802, the charging plug comprises a plurality of charging heads, the force sensor is used for collecting force information of a contact surface between the charging plug and the charging port, the contact surface comprises a side surface of at least one charging head in the plurality of charging heads, and the side surface is a surface parallel to an axial direction of the charging head;

the method comprises the following steps:

and according to the acting force information, controlling the pose of the main body of the mechanical arm 802 so as to insert the charging plug 803 into the charging port.

In one possible implementation, the method further comprises:

extracting a 2D position corresponding to the charging port according to the 2D image;

extracting a 3D position corresponding to the charging port according to the 3D point cloud;

and fusing the 2D position and the 3D position to obtain the position information of the charging port.

For the description of the control mechanism, reference may be made to the description of the control structure of the mechanical arm 802 in the above embodiment, which is not repeated here.

The embodiment of the application also provides a device for controlling the charging of the mechanical arm, the device is applied to a control structure of the mechanical arm 802, a force sensor and a charging plug are arranged on the mechanical arm main body, the charging plug comprises a plurality of charging heads, the force sensor is used for collecting acting force information of a contact surface between the charging plug and the charging port, the contact surface comprises a side surface of at least one charging head in the plurality of charging heads, and the side surface is a surface parallel to the axial direction of the charging head;

the device comprises:

and the control module is used for controlling the pose of the main body of the mechanical arm 802 according to the acting force information so as to insert the charging plug 803 into the charging port.

In one possible implementation, the charging port includes a plurality of charging holes, and the plurality of charging heads are matched with the plurality of charging holes one by one; the control module is specifically configured to:

correcting the pose of the main body of the mechanical arm 802 from a first pose to a second pose according to the pose adjustment parameters; the first pose and the second pose are different, and the second pose is a pose of the main body of the mechanical arm 802 when each charging head is aligned with a matched charging hole.

In one possible implementation, after correcting the pose of the body of the mechanical arm 802 from the first pose to the second pose, the control module is further configured to:

the pose of the body of the mechanical arm 802 is controlled so that the charging plug 803 moves toward the bottom of the charging hole and in the direction of the central axis of the charging head.

In one possible implementation, during the process of correcting from the first pose to the second pose, the resistance of the charging port to the charging head in a target direction is reduced, where the target direction is an axial direction or a radial direction of the charging plug 803.

In one possible implementation, the control module is further configured to:

the mechanical arm 802 body is controlled to be kept in a stationary state based on the force information indicating that the charging port has only resistance to the charging head in the axial direction of the charging plug 803.

In one possible implementation, the pose adjustment parameters are specifically used to control the movement direction and movement rate of the charging plug 803.

In one possible implementation, the apparatus further includes:

the acquiring module is configured to acquire image information acquired by the vision sensor 804 for the area of the charging port in real time before controlling the pose of the main body of the mechanical arm 802 according to the acting force information;

obtaining the real-time pose of the main body of the mechanical arm 802 according to the position information of the charging port in the image information;

the control module is further configured to:

and controlling the pose of the main body of the mechanical arm 802 according to the real-time pose so as to guide a charging plug 803 on the main body of the mechanical arm 802 to the periphery of the charging port.

In one possible implementation, the vision sensor 804 is disposed on the body of the robotic arm 802.

In one possible implementation, the image information includes a 2D image and a 3D point cloud, and the acquiring module is specifically configured to:

Next, referring to fig. 24, fig. 24 is a schematic structural diagram of a device for controlling charging of the mechanical arm 802 according to an embodiment of the present application. Specifically, the device 2400 for controlling charging of the mechanical arm 802 includes: receiver 2401, transmitter 2402, processor 2403 and memory 2404 (where the number of processors 2403 in device 2400 controlling the charging of robotic arm 802 may be one or more, one processor is exemplified in fig. 24), where processor 2403 may include application processor 24031 and communication processor 24032. In some embodiments of the present application, receiver 2401, transmitter 2402, processor 2403, and memory 2404 may be connected by a bus or other means.

Memory 2404 may include read only memory and random access memory and provide instructions and data to processor 2403. A portion of memory 2404 may also include non-volatile random access memory (non-volatile random access memory, NVRAM). The memory 2404 stores a processor and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, where the operating instructions may include various operating instructions for implementing various operations.

Processor 2403 controls the operation of the radar system (including antenna, receiver 2401, and transmitter 2402). In a specific application, the various components of the radar system are coupled together by a bus system, which may include a power bus, a control bus, a status signal bus, and the like, in addition to a data bus. For clarity of illustration, however, the various buses are referred to in the figures as bus systems.

The method for controlling the charging of the mechanical arm 802 (shown in fig. 7) disclosed in the embodiment of the present application may be applied to the processor 2403, or implemented by the processor 2403. Processor 2403 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the methods described above may be performed by integrated logic circuitry in hardware in processor 2403 or by instructions in software. The processor 2403 may be a general purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor, or a microcontroller, and may further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The processor 2403 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 2404, and the processor 2403 reads the information in the memory 2404, and combines the hardware thereof to complete the steps of the method for controlling the charging of the mechanical arm 802 provided in the above embodiment.

The receiver 2401 may be used to receive input numeric or character information and to generate signal inputs related to the relevant settings and function control of the radar system. The transmitter 2402 may be used to output numeric or character information through a first interface; the transmitter 2402 may also be used to send instructions to the disk group through the first interface to modify data in the disk group.

Embodiments of the present application also provide a method comprising a computer program product, which when run on a computer, causes the computer to perform the method of controlling the charging of the robotic arm 802 described in the above embodiments.

There is also provided in the embodiments of the present application a computer-readable storage medium in which a program for performing signal processing is stored, which when run on a computer, causes the computer to perform the method of controlling the charging of the robot arm 802 as described in the above embodiments.

The device for controlling the mechanical arm 802 to charge provided in this embodiment may specifically be a chip, where the chip includes: a processing unit, which may be, for example, a processor, and a communication unit, which may be, for example, an input/output interface, pins or circuitry, etc. The processing unit may execute the computer-executable instructions stored in the storage unit to cause the chip in the execution device to perform the image enhancement method described in the above embodiment, or to cause the chip in the training device to perform the image enhancement method described in the above embodiment. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM), etc.

Specifically, referring to fig. 25, fig. 25 is a schematic structural diagram of a chip provided in an embodiment of the present application, where the chip may be represented as a neural network processor NPU250, and the NPU250 is mounted as a coprocessor on a main CPU (Host CPU), and the Host CPU distributes tasks. The NPU has a core part of an arithmetic circuit 2503, and the arithmetic circuit 2503 is controlled by a controller 2504 to extract matrix data in a memory and perform multiplication.

In some implementations, the arithmetic circuit 2503 includes a plurality of processing units (PEs) inside. In some implementations, the operational circuit 2503 is a two-dimensional systolic array. The arithmetic circuit 2503 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2503 is a general-purpose matrix processor.

For example, assume that there is an input matrix a, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the data corresponding to the matrix B from the weight memory 2502 and buffers each PE in the arithmetic circuit. The arithmetic circuit takes matrix a data from the input memory 2501 and performs matrix operation with matrix B, and a partial result or a final result of the matrix obtained is stored in an accumulator (accumulator) 2508.

The unified memory 2506 is used for storing input data and output data. The weight data is directly transferred to the weight memory 2502 through the memory cell access controller (direct memory access controller, DMAC) 2505. The input data is also carried into the unified memory 2506 through the DMAC.

BIU is Bus Interface Unit, bus interface unit 2510, for the AXI bus to interact with the DMAC and finger memory (Instruction Fetch Buffer, IFB) 2509.

The bus interface unit 2510 (Bus Interface Unit, abbreviated as BIU) is used for the instruction fetch memory 2509 to fetch instructions from an external memory, and is also used for the memory unit access controller 2505 to fetch raw data of the input matrix a or the weight matrix B from the external memory.

The DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2506 or to transfer weight data to the weight memory 2502 or to transfer input data to the input memory 2501.

The vector calculation unit 2507 includes a plurality of operation processing units, and performs further processing on the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and the like, as necessary. The method is mainly used for non-convolution/full-connection layer network calculation in the neural network, such as Batch Normalization (batch normalization), pixel-level summation, up-sampling of a characteristic plane and the like.

In some implementations, the vector computation unit 2507 can store the vector of processed outputs to the unified memory 2506. For example, the vector calculation unit 2507 may apply a linear function and/or a nonlinear function to the output of the arithmetic circuit 2503, for example, linearly interpolate the feature plane extracted by the convolution layer, and further, for example, accumulate a vector of values to generate an activation value. In some implementations, vector calculation unit 2507 generates normalized values, pixel-level summed values, or both. In some implementations, the vector of processed outputs can be used as an activation input to the arithmetic circuit 2503, e.g., for use in subsequent layers in a neural network.

An instruction fetch memory (instruction fetch buffer) 2509 connected to the controller 2504 for storing instructions used by the controller 2504;

the unified memory 2506, the input memory 2501, the weight memory 2502, and the finger memory 2509 are all On-Chip memories. The external memory is proprietary to the NPU hardware architecture.

The processor mentioned in any of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the relevant steps of the method for controlling the charging of the robot 802 described in the above embodiment.

It should be further noted that the above-described apparatus embodiments are merely illustrative, and that the units described as separate units may or may not be physically separate, and that units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the embodiment of the device provided by the application, the connection relation between the modules represents that the modules have communication connection therebetween, and can be specifically implemented as one or more communication buses or signal lines.

From the above description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented by means of software plus necessary general purpose hardware, or of course may be implemented by dedicated hardware including application specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components and the like. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions can be varied, such as analog circuits, digital circuits, or dedicated circuits. However, a software program implementation is a preferred embodiment in many cases for the present application. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk of a computer, etc., including several instructions for causing a computer device (which may be a personal computer, a training device, or a network device, etc.) to perform the method according to the embodiments of the present application.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces, in whole or in part, a flow or function consistent with embodiments of the present application. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via a wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, or the like that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), or the like.

Claims

1. A charging apparatus, characterized by comprising: mechanical arm, force sensor and charging plug;

2. The charging device of claim 1, wherein the effort information includes moment information of the charging port to the charging plug.

3. The charging apparatus according to claim 1 or 2, wherein,

the force sensor is fixed on the side face of the at least one charging head; or alternatively, the process may be performed,

4. A charging apparatus according to any one of claims 1 to 3, wherein the charging port includes a plurality of charging holes, the plurality of charging heads being in one-to-one correspondence with the plurality of charging holes; the control structure is specifically used for:

5. The charging apparatus according to claim 4, wherein the first pose is a pose when bottom end surfaces of the plurality of charging heads are not in contact with bottoms of the charging holes, the bottom end surfaces being surfaces perpendicular to an axial direction of the charging heads.

6. The charging device of claim 4 or 5, wherein the aligning comprises: the projection of the charging head along the direction of the matched charging hole falls in the range of the matched charging hole, and the central axis of the charging head coincides with the central axis of the matched charging hole or the deviation is smaller than a threshold value.

7. The charging apparatus according to any one of claims 4 to 6, wherein the control structure is further configured to:

8. The charging apparatus according to any one of claims 4 to 7, wherein there is a tendency for the moment between the charging ports and the charging plug to decrease during the correction from the first posture to the second posture.

9. The charging device according to any one of claims 4 to 8, wherein the pose adjustment parameter is specifically used for controlling a movement direction and a movement rate of the charging plug.

10. Charging apparatus according to any one of claims 1 to 9, wherein the control structure is specifically configured to:

11. The charging apparatus according to any one of claims 1 to 10, wherein the control mechanism is further configured to:

12. The charging apparatus according to claim 11, wherein the vision sensor is provided on the robot arm main body.

13. The charging device according to claim 11 or 12, wherein the image information includes a 2D image and a 3D point cloud, the control mechanism further being configured to:

14. The charging device of claim 13, wherein the first 3D location comprises 3D locations of M first feature points on the charging port; the second 3D position includes 3D positions of M second feature points on the charging port; the M first feature points and the M second feature points are in one-to-one correspondence;

The fusing the first 3D position and the second 3D position includes:

15. The charging device of any one of claims 1 to 14, wherein the robotic arm body further comprises a target interface for detachably connecting the charging plug.

16. The charging device of claim 15, wherein the target interface is connected with an initial charging plug prior to the control structure controlling the pose of the robotic arm body according to the effort information;

17. The charging device of claim 16, wherein the charging device further comprises: a plug receiving area including a target receiving location for setting the charging plug;

18. The charging device according to any one of claims 1 to 17, wherein the charging device is a stationary charging pile, a mobile robot or a charging vehicle.

19. The method is applied to a control structure of the mechanical arm, a force sensor and a charging plug are arranged on a mechanical arm main body, the charging plug comprises a plurality of charging heads, the force sensor is used for collecting acting force information of a contact surface between the charging plug and the charging port, the contact surface comprises a side surface of at least one charging head in the plurality of charging heads, and the side surface is a surface parallel to the axial direction of the charging head;

The method comprises the following steps:

20. The method of claim 19, wherein the effort information includes moment information of the charging port to the charging plug.

21. The method of claim 19 or 20, wherein the charging port comprises a plurality of charging holes, the plurality of charging heads being in one-to-one correspondence with the plurality of charging holes; according to the acting force information, controlling the pose of the mechanical arm main body comprises the following steps:

22. The method of claim 21, wherein the first pose is a pose when bottom end surfaces of the plurality of charging heads are not in contact with bottoms of the charging holes, the bottom end surfaces being surfaces perpendicular to an axial direction of the charging heads.

23. The method of claim 21 or 22, wherein after correcting the pose of the robotic arm body from the first pose to the second pose, the method further comprises:

24. The method according to any one of claims 19 to 23, further comprising:

25. The method according to any one of claims 19 to 24, further comprising:

26. The method of claim 25, wherein the image information comprises a 2D image and a 3D point cloud, the method further comprising:

27. The method of claim 26, wherein the first 3D locations comprise 3D locations of M first feature points on the charging port; the second 3D position includes 3D positions of M second feature points on the charging port; the M first feature points and the M second feature points are in one-to-one correspondence;

the fusing the first 3D position and the second 3D position includes:

28. A computer readable storage medium comprising computer readable instructions which, when run on a computer device, cause the computer device to perform the method of any of claims 19 to 27.

29. A computer program product comprising computer readable instructions which, when run on a computer device, cause the computer device to perform the method of any of claims 19 to 27.