CN117207204B

CN117207204B - Control method and control device of playing robot

Info

Publication number: CN117207204B
Application number: CN202311489049.3A
Authority: CN
Inventors: 孟森森; 宛敏红; 严敏东; 钟灵; 顾建军
Original assignee: Zhejiang Lab
Current assignee: Zhejiang Lab
Priority date: 2023-11-09
Filing date: 2023-11-09
Publication date: 2024-01-30
Anticipated expiration: 2043-11-09
Also published as: CN117207204A

Abstract

The specification discloses a control method and a control device of a musical instrument playing robot, after a piano playing area image is acquired through a camera, coordinates of each piano key in a camera coordinate system are determined according to image feature points in the image, and coordinates of each piano key in a piano key template in the camera coordinate system are determined according to the coordinates of each image feature point in the camera coordinate system and the piano key template. The relative position between the playing robot and the keys is determined through the pose of the playing robot in the playing process and the coordinates of each key under a camera coordinate system, and the error compensation is carried out on the relative position between the playing robot and the keys through a predetermined error fitting function, so that the relative position after the mechanical arm on the playing robot plays compensation is controlled. And the relative positions between the playing robot and the keys are subjected to error compensation through a predetermined error fitting function, so that the error of the playing robot in real-time positioning of the keys is reduced.

Description

Control method and control device of playing robot

Technical Field

The present disclosure relates to the field of artificial intelligence and visual recognition, and in particular, to a control method and a control device for a musical instrument playing robot.

Background

With the rapid development of the artificial intelligence field, a large number of robots are present in a number of practical application scenarios, such as intelligent educational robots, emotion communication robots, cooking robots, performance robots, etc. With the increasing progress of the precision of robots, the degree of personification that can be achieved by robots is increasing, and some robots with higher requirements for personification have been developed, such as playing robots capable of playing pianos.

However, the existing playing robot cannot accurately position the keys in the anthropomorphic playing process, because the anthropomorphic playing piano often needs to act on the head and neck parts of the playing robot to cause the change of the visual angle of the camera of the playing robot, but the existing positioning mode cannot accurately position the positions of the keys while the camera moves, so that the playing robot has larger error in real-time positioning of the keys.

Therefore, how to effectively reduce the error of the playing robot in positioning the keys in real time is a problem to be solved urgently.

Disclosure of Invention

The present disclosure provides a control method and a control device for a playing robot, so as to partially solve the above problems in the prior art.

The technical scheme adopted in the specification is as follows:

the specification provides a control method of a playing robot, comprising the following steps:

the robot for playing the piano performs image acquisition on the piano key area through the camera, and acquired images are obtained;

identifying each image characteristic point from the image, and determining the coordinates of each image characteristic point under a camera coordinate system according to depth information corresponding to each pixel point contained in the image and the coordinates of each image characteristic point in the image;

determining the coordinates of each key in the key template under the camera coordinate system according to the coordinates of each image characteristic point under the camera coordinate system and the key template, wherein the key template is used for representing the coordinates of each key under the world coordinate system;

determining the relative position between the playing robot and each key under the pose of the playing robot in the playing process by the pose of the playing robot in the playing process and the coordinates of each key under a camera coordinate system;

performing error compensation on the relative position through a predetermined error fitting function, and obtaining a compensated relative position, wherein the error fitting function is used for representing visual positioning errors of the piano playing robot to keys under all poses, and is fitted on the basis of visual positioning error values of the piano playing robot under all poses in advance;

And controlling a mechanical arm on the playing robot to play according to the compensated relative position.

Optionally, each image feature point is identified from the image, and the coordinates of each image feature point under a camera coordinate system are determined according to depth information corresponding to each pixel point contained in the image and coordinates of each image feature point in the image, which specifically includes:

determining a binary image corresponding to the image;

determining a straight line which meets a preset length range and a preset slope range according to the change of pixel values in the binary image, wherein the straight line is used as a separation line of a black key area and a white key area, and determining an edge line of each black key in the binary image;

determining each image characteristic point and coordinates of each image characteristic point in the binary image according to the determined separation line and the edge line of each black key, wherein for each black key, one of two corner points on the separation line in the black key, which is closest to the central point of the binary image, is selected as the image characteristic point;

determining the coordinates of the image feature points in the image according to the coordinates of the image feature points in the binary image;

And determining the coordinates of each image feature point under a camera coordinate system according to the depth information corresponding to each pixel point contained in the image and the coordinates of each image feature point in the image.

Optionally, determining the coordinates of each image feature point under a camera coordinate system according to depth information corresponding to each pixel point included in the image and coordinates of each image feature point in the image specifically includes:

determining a key area point cloud according to depth information corresponding to each pixel point contained in the image;

performing plane fitting on point cloud data contained in the point cloud of the key area, and determining ideal depth information corresponding to each pixel point of the white key surface in the image;

and determining the coordinates of each image feature point under a camera coordinate system through the ideal depth information and the coordinates of each image feature point in the image.

Optionally, determining the coordinates of each key in the key template under the camera coordinate system according to the coordinates of each image feature point under the camera coordinate system and the key template, specifically includes:

according to the image feature points, determining point clouds formed by the image feature points under a camera coordinate system, matching the point clouds formed by the image feature points under the camera coordinate system with point clouds corresponding to the feature points in a key template to obtain a matching result, and acquiring coordinates of the image feature points under the world coordinate system according to the matching result;

Establishing a first transformation matrix according to the coordinates of each image characteristic point under the world coordinate system and the coordinates of each image characteristic point under the camera coordinate system, wherein the first transformation matrix is used for representing the coordinate transformation relation between the camera coordinate system and the world coordinate system;

and determining the coordinates of each key in the key template under the camera coordinate system according to the first conversion matrix.

Optionally, determining, by the pose of the playing robot in the playing process and the coordinates of each key under the camera coordinate system, the relative position between the playing robot and each key in the pose of the playing robot in the playing process specifically includes:

converting the coordinates of each key under a camera coordinate system into coordinates under a robot head end point coordinate system through a predetermined second conversion matrix, wherein the second conversion matrix is used for representing the coordinate conversion relation between the camera coordinate system and the robot head end point coordinate system, and the origin of the robot head end point coordinate system is the center point of the robot head;

converting the coordinates of each key under the coordinate system of the tail end of the robot into the coordinates under the coordinate system of the robot by a third conversion matrix determined by the robot according to the pose in the process of playing the musical instrument, wherein the third conversion matrix is used for representing the coordinate conversion relation between the coordinate system of the tail end of the robot and the coordinate system of the robot, and the origin of the coordinate system of the robot is positioned at the waist position of the robot;

And determining the relative positions between the playing robot and each key under the pose of the playing robot in the playing process according to the coordinates of each key under the robot coordinate system.

Optionally, the error fitting function is predetermined, specifically including:

before the playing robot executes playing of the musical instrument, acquiring images acquired by the playing robot under each preset pose as calibration images;

for each pose, determining coordinates of each pre-marked key point under the robot coordinate system according to a calibration image acquired under the pose;

moving the mechanical arm to the position of each key point in the real world by the playing robot, and recording the coordinates of the mechanical arm in the robot coordinate system after the mechanical arm stays to the position of each key point in the real world;

according to the actual coordinates of the key points in the robot coordinate system and the coordinate deviation between the coordinates of the key points in the robot coordinate system determined based on the calibration image, determining a visual positioning error value in the pose;

and fitting an error fitting function according to the vision positioning error value under each pose.

Optionally, for each pose, determining coordinates of each pre-marked key point in the robot coordinate system according to the calibration image acquired under the pose specifically includes:

aiming at each pose, the robot for playing the piano performs image acquisition on the piano key area through the set camera, and acquires the acquired calibration image under the pose;

identifying each image characteristic point from the calibration image under the pose, and determining the coordinate of each image characteristic point under the camera coordinate system according to the coordinate of each image characteristic point under the pose in the calibration image under the pose and the depth information corresponding to each pixel point in the calibration image under the pose;

matching the coordinates of each image feature point in the camera coordinate system under the pose with the coordinates of each feature point in the key template under the world coordinate system, establishing a coordinate conversion relation between the camera coordinate system under the pose and the world coordinate system, and determining the coordinates of each key point under the camera coordinate system under the pose according to the coordinate conversion relation between the camera coordinate system under the pose and the world coordinate system;

and determining the coordinates of the key points in the robot coordinate system in the pose of the robot playing the musical instrument according to the coordinates of the key points in the camera coordinate system in the pose of the robot playing the musical instrument, a second transformation matrix which is determined in advance and a third transformation matrix which is determined according to the pose of the robot playing the musical instrument in the playing process.

The present specification provides a control device of a musical instrument playing robot, comprising:

the acquisition module is used for acquiring images of the piano playing robot through the set camera to obtain acquired images;

the identification module is used for identifying each image characteristic point from the image, and determining the coordinates of each image characteristic point under a camera coordinate system according to the depth information corresponding to each pixel point contained in the image and the coordinates of each image characteristic point in the image;

the first determining module is used for determining the coordinates of each key in the key template under the camera coordinate system according to the coordinates of each image characteristic point under the camera coordinate system and the key template;

the second determining module is used for determining the relative positions between the playing robot and each key under the pose of the playing robot in the playing process according to the pose of the playing robot in the playing process and the coordinates of each key under the camera coordinate system;

the error compensation module is used for carrying out error compensation on the relative position through a predetermined error fitting function to obtain a compensated relative position;

And the control module is used for controlling the mechanical arm on the playing robot to play according to the compensated relative position.

Optionally, the identification module is specifically configured to,

determining a binary image corresponding to the image; determining a straight line which meets a preset length range and a preset slope range according to the change of pixel values in the binary image, wherein the straight line is used as a separation line of a black key area and a white key area, and determining an edge line of each black key in the binary image; determining each image characteristic point and coordinates of each image characteristic point in the binary image according to the determined separation line and the edge line of each black key, wherein for each black key, one of two corner points on the separation line in the black key, which is closest to the central point of the binary image, is selected as the image characteristic point; determining the coordinates of the image feature points in the image according to the coordinates of the image feature points in the binary image; and determining the coordinates of each image feature point under a camera coordinate system according to the depth information corresponding to each pixel point contained in the image and the coordinates of each image feature point in the image.

Optionally, the identification module is specifically configured to,

determining a key area point cloud according to depth information corresponding to each pixel point contained in the image; performing plane fitting on point cloud data contained in the point cloud of the key area, and determining ideal depth information corresponding to each pixel point of the white key surface in the image; and determining the coordinates of each image feature point under a camera coordinate system through the ideal depth information and the coordinates of each image feature point in the image.

Optionally, the first determining module is specifically configured to,

according to the image feature points, determining point clouds formed by the image feature points under a camera coordinate system, matching the point clouds formed by the image feature points under the camera coordinate system with point clouds corresponding to the feature points in a key template to obtain a matching result, and acquiring coordinates of the image feature points under the world coordinate system according to the matching result; establishing a first transformation matrix according to the coordinates of each image characteristic point under the world coordinate system and the coordinates of each image characteristic point under the camera coordinate system, wherein the first transformation matrix is used for representing the coordinate transformation relation between the camera coordinate system and the world coordinate system; and determining the coordinates of each key in the key template under the camera coordinate system according to the first conversion matrix.

Optionally, the second determining module is specifically configured to,

converting the coordinates of each key under a camera coordinate system into coordinates under a robot head end point coordinate system through a predetermined second conversion matrix, wherein the second conversion matrix is used for representing the coordinate conversion relation between the camera coordinate system and the robot head end point coordinate system, and the origin of the robot head end point coordinate system is the center point of the robot head; converting the coordinates of each key under the coordinate system of the tail end of the robot into the coordinates under the coordinate system of the robot by a third conversion matrix determined by the robot according to the pose in the process of playing the musical instrument, wherein the third conversion matrix is used for representing the coordinate conversion relation between the coordinate system of the tail end of the robot and the coordinate system of the robot, and the origin of the coordinate system of the robot is positioned at the waist position of the robot; and determining the relative positions between the playing robot and each key under the pose of the playing robot in the playing process according to the coordinates of each key under the robot coordinate system.

Optionally, the second determining module is specifically configured to,

Before the playing robot executes playing of the musical instrument, acquiring images acquired by the playing robot under each preset pose as calibration images; for each pose, determining coordinates of each pre-marked key point under the robot coordinate system according to a calibration image acquired under the pose; moving the mechanical arm to the position of each key point in the real world by the playing robot, and recording the coordinates of the mechanical arm in the robot coordinate system after the mechanical arm stays to the position of each key point in the real world; according to the actual coordinates of the key points in the robot coordinate system and the coordinate deviation between the coordinates of the key points in the robot coordinate system determined based on the calibration image, determining a visual positioning error value in the pose; and fitting an error fitting function according to the vision positioning error value under each pose.

Optionally, the second determining module is specifically configured to,

aiming at each pose, the robot for playing the piano performs image acquisition on the piano key area through the set camera, and acquires the acquired calibration image under the pose; identifying each image characteristic point from the calibration image under the pose, and determining the coordinate of each image characteristic point under the camera coordinate system according to the coordinate of each image characteristic point under the pose in the calibration image under the pose and the depth information corresponding to each pixel point in the calibration image under the pose; matching the coordinates of each image feature point in the camera coordinate system under the pose with the coordinates of each feature point in the key template under the world coordinate system, establishing a coordinate conversion relation between the camera coordinate system under the pose and the world coordinate system, and determining the coordinates of each key point under the camera coordinate system under the pose according to the coordinate conversion relation between the camera coordinate system under the pose and the world coordinate system; and determining the coordinates of the key points in the robot coordinate system in the pose of the robot playing the musical instrument according to the coordinates of the key points in the camera coordinate system in the pose of the robot playing the musical instrument, a second transformation matrix which is determined in advance and a third transformation matrix which is determined according to the pose of the robot playing the musical instrument in the playing process.

The present specification provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements a method of controlling a robot for playing a musical instrument.

The present specification provides an electronic device including a processor and a computer program stored on a memory and executable on the processor, the processor implementing the control method of the above-mentioned robot for playing musical instrument when executing the program.

The above-mentioned at least one technical scheme that this specification adopted can reach following beneficial effect:

in the control method of the playing robot provided by the specification, the playing robot performs image acquisition on a key area through a set camera, identifies image feature points from acquired images, determines coordinates of each image feature point under a camera coordinate system according to information corresponding to each pixel point contained in the acquired images and coordinates of each image feature point in the acquired images, and determines coordinates of each key in a key template under the camera coordinate system according to the coordinates of each image feature point under the camera coordinate system and the key template. The relative position between the playing robot and the keys is determined through the pose of the playing robot in the playing process and the coordinates of each key under the camera coordinate system, and the relative position between the playing robot and the keys is subjected to error compensation through a predetermined error fitting function, so that the relative position after the mechanical arm on the playing robot plays compensation can be controlled.

According to the method, the relative position between the piano playing robot and the keys is subjected to error compensation through the predetermined error fitting function, so that the compensated relative position can be obtained, accurate positioning of the keys is realized, the relative position between the piano playing robot and the keys is subjected to error compensation through the error fitting function, and the error of the piano playing robot to the real-time positioning of the keys is reduced.

Drawings

The accompanying drawings, which are included to provide a further understanding of the specification, illustrate and explain the exemplary embodiments of the present specification and their description, are not intended to limit the specification unduly. In the drawings:

fig. 1 is a schematic flow chart of a control method of a playing robot provided in the present specification;

FIG. 2 is a schematic view of an image feature point provided in the present specification;

fig. 3 is a schematic structural view of a control device of a playing robot provided in the present specification;

fig. 4 is a schematic structural view of an electronic device corresponding to fig. 1 provided in the present specification.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the technical solutions of the present specification will be clearly and completely described below with reference to specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present specification. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings.

Fig. 1 is a schematic flow chart of a control method based on a constructed playing robot provided in the present specification, including:

s101: the playing robot performs image acquisition on the key area through the set camera to obtain an acquired image.

In the existing key positioning field, the current mainstream method is often limited to acquiring an image of a key area through a camera, and then identifying coordinates of the key in the image, so as to obtain a relative position of the key and the robot camera. The identification of the image of the key area can be performed in various ways, but the coordinates of the identified keys in the image have certain deviation, so that the obtained relative positions of the keys and the robot camera for playing the musical instrument have errors.

For example, the playing robot may acquire a key area image through the RGBD camera, and then recognize coordinates of the key in the key area image and depth information corresponding to each pixel point in the key area image, so as to obtain a relative position of the key and the playing robot camera.

The existing playing robot can directly acquire the coordinates of the keys under the camera coordinate system through the camera in the playing process, and in the playing process, the playing robot moves to cause errors in real-time positioning of the playing robot on the keys.

In addition, the playing robot often determines the coordinates of the key under the camera coordinate system through the obtained relative positions of the key and the camera of the playing robot, then, the coordinates of the key and the mechanical arm required for playing the piano need to be unified into the same coordinate system, for example, the key coordinates are converted from the camera coordinate system to the coordinate system where the mechanical arm of the playing robot is located, so as to obtain the relative positions of the key and the playing robot, and in the conversion process, certain errors may occur, because the conversion matrix used for converting the key coordinates may be interfered by factors, such as measurement errors, mechanical vibration, etc., in the determination process, and the factors may cause errors to occur in the determined conversion matrix, and errors occur after the key coordinates are converted to the coordinate system where the mechanical arm of the playing robot is located, so that the playing robot plays the key may be affected.

As described above, the existing playing robot has an error in calculating the relative positions between the keys and the robot during the playing process, thereby causing the playing robot to fail to accurately play the piano. While in the real-time playing process of a piano, higher accuracy is required for calculation of the relative positions between the keys and the robot.

Based on this, the present specification provides a control method of a playing robot, where after the playing robot obtains the relative positions of the keys and the playing robot through the images obtained by the camera in the playing process, the obtained relative positions of the keys and the playing robot can be subjected to error compensation according to a predetermined error fitting function, so as to accurately determine the relative positions of the keys and the playing robot.

When the playing robot moves in a personification mode, the head and neck part where the camera is located can move up and down and move left and right, and keys required to be played by the playing robot are not in a visual field range in the process, so that the playing robot can acquire real-time images of key areas in the playing process, and acquired images are obtained. And then acquiring coordinates of the keys under a camera coordinate system of the playing robot through the acquired images, and further acquiring the relative positions of the keys and the playing robot, so that the relative positions of the keys and the playing robot after accurate playing error compensation are realized. The camera used by the robot for playing the musical instrument is an RGBD camera, and the acquired image can comprise depth information corresponding to pixel points in the image.

S102: and identifying each image characteristic point from the image, and determining the coordinates of each image characteristic point under a camera coordinate system according to the depth information corresponding to each pixel point contained in the image and the coordinates of each image characteristic point in the image.

After the robot for playing the piano acquires the acquired image, the coordinates of each piano key under the camera coordinate system are required to be determined through the acquired image. The coordinates of the key under the camera coordinate system are directly obtained through the coordinates of the key in the acquired image and the depth information of the coordinates, so that the obtained coordinates of the key under the camera coordinate system have errors. Accordingly, the present specification provides a method of determining the coordinates of a key in a camera coordinate system from an image acquired by an RGBD camera.

The robot for playing the piano firstly identifies each image characteristic point from the acquired image, determines the coordinates of each image characteristic point in the acquired image, and then obtains ideal depth information corresponding to each image characteristic point according to the depth information corresponding to each pixel point contained in the acquired image and the coordinates of each image characteristic point in the acquired image, so as to determine the coordinates of each image characteristic point under a camera coordinate system. Wherein, select several points as the image characteristic point from the angular point of black key.

Specifically, the robot for playing the musical instrument can binarize the acquired image first, so as to obtain a binary image corresponding to the acquired image. In the binary image, since the keys to be positioned are regularly arranged by black and white keys, the pixel values of the separation lines of the black key area and the white key area in the binary image, namely the separation line where the lower edge line of the black key is located, can be changed greatly.

The direct determination of the separation line of the black key region and the white key region by the degree of variation of the pixel value may cause a case of erroneous discrimination, for example, the discriminated separation line is only a part of the separation line of the black key region and the white key region. To avoid this possibility, the separation lines of the black key region and the white key region are more accurately determined by predefining the separation lines, including predefining the length range and the slope range of the separation lines.

After the separation line of the black key and the white key is determined, the position of the edge line of the black key in the binary image can be obtained according to the gap between the black key and the white key through an edge detection algorithm.

After the robot gets the edge line of the black key and the separation line of the black key area and the white key area, the coordinates of the two corner points of the lower half part of each black key in the binary image can be determined according to the edge line of the black key and the separation line of the black key area and the white key area. Wherein, in order to make the coordinates of the obtained image feature points more accurate, for each black key, one closer to the center of the binary image is selected as the image feature point from the two corner points of the lower half of the black key. The robot can determine the coordinates of the image feature points in the acquired image according to the coordinates of the image feature points in the binary image.

Fig. 2 is a schematic diagram of an image feature point provided in the present specification. The dashed line in fig. 2 is a straight line where the center of the visual field is located, where for a black key on the left side of the center of the visual field, the corner point at the lower right corner is selected as an image feature point, such as image feature point one, and for a black key on the right side of the center of the visual field, the corner point at the lower left corner is selected as image feature points, such as image feature points two and three.

It should be noted that, since the collected image mainly includes the key area and the black-white pixels in the collected image are arranged regularly, the coordinates of the feature points of each image in the binary image corresponding to the collected image can directly correspond to the coordinates in the collected image.

In addition, the coordinates of the upper left point or the coordinates of the upper right point of the key area of the whole piano contained in the acquired image can be determined by the method, and if one or two coordinates exist in the acquired image, one or two coordinates can be used as supplementary image feature points.

After the coordinates of the image feature points in the acquired image are determined, the robot for playing the musical instrument needs to determine the coordinates of the image feature points under the camera coordinate system according to the coordinates of the image feature points in the acquired image and the depth information corresponding to the pixel points contained in the acquired image. However, the robot for playing the musical instrument may have an error according to the depth information corresponding to each pixel point in the acquired image obtained by the RGBD camera, for example, the depth information corresponding to each pixel point in the acquired image is not located on the same plane on the white key surface in the acquired image, so that the depth information corresponding to each pixel point in the acquired image needs to be processed to obtain the ideal depth information corresponding to each usable pixel point. The depth information corresponding to the pixels on the surface of the black key acquired by the camera has larger error, so that the depth information corresponding to the pixels on the surface of the white key in the acquired image can be selected and processed, and further the ideal depth information corresponding to the pixels on the surface of the white key in the acquired image is determined.

Specifically, the musical instrument playing robot determines the point cloud of the white key area according to the depth information corresponding to the pixel points of each white key area contained in the acquired image, performs plane fitting on the point cloud data of the white key area, and determines ideal depth information corresponding to each pixel point of the white key surface in the acquired image.

The robot can determine the coordinates of each image characteristic point under a camera coordinate system according to the ideal depth information of each pixel point on the white key surface in the acquired image and the coordinates of each image characteristic point in the acquired image, wherein the camera coordinate system is determined based on a camera of the robot.

S103: and determining the coordinates of each key in the key template under the camera coordinate system according to the coordinates of each image characteristic point under the camera coordinate system and the key template, wherein the key template is used for representing the coordinates of each key under the world coordinate system.

After the coordinates of each image feature point under the camera coordinate system are determined, the robot for playing the piano establishes a first transformation matrix through the coordinates of each image feature point in the piano key template, namely, the coordinates of each image feature point under the world coordinate system, wherein the first transformation matrix is used for representing the coordinate transformation relation between the camera coordinate system and the world coordinate system. And then, converting the coordinates of each key in the key template into a camera coordinate system according to the first conversion matrix, and determining the coordinates of each key under the camera coordinate system. The key templates are used for representing coordinates of each key in a world coordinate system.

The first transformation matrix is established by determining the coordinates of each image feature point in the world coordinate system, that is, by matching each image feature point in the camera coordinate system with each feature point in the key template, and then acquiring the coordinates in the key template corresponding to each image feature point, that is, the coordinates in the world coordinate system according to the matching result.

Specifically, a point cloud matching algorithm is used to match the point cloud formed by the feature points of each image in the camera coordinate system with the point cloud formed by the feature points in the key template.

For example, in the process of using the iterative closest point algorithm (Iterative Closest Point, ICP), a point cloud formed by each image feature point in a camera coordinate system is matched with a point cloud formed by each feature point in a key template to obtain a matching result, and then coordinates in the key template corresponding to each image feature point, that is, coordinates in a world coordinate system, are obtained according to the matching result.

After the robot for playing the musical instrument obtains the coordinates of each image characteristic point under the world coordinate system through the point cloud matching algorithm, a first transformation matrix is determined according to the coordinates of each image characteristic point under the world coordinate system and the coordinates of each image characteristic point under the camera coordinate system. And then, according to the first transformation matrix, the coordinates of each key in the key template under the world coordinate system can be transformed into the coordinates of each key under the camera coordinate system.

In the process of determining the coordinates of each key in the camera coordinate system, the coordinates of all keys may be determined, or the coordinates of the key to be played by the playing robot in the pose may be determined, which is not particularly limited in this specification.

S104: and determining the relative position between the playing robot and each key under the pose of the playing robot in the playing process by the pose of the playing robot in the playing process and the coordinates of each key under the camera coordinate system.

After the robot for playing the piano determines the coordinates of each piano key in the camera coordinate system, the coordinates of each piano key in the camera coordinate system and the mechanical arm are unified into the same coordinate system so as to play the piano keys.

For example, the robot for playing a musical instrument may unify the coordinates of each key in the camera coordinate system and the arm into a robot coordinate system including the coordinates of the arm. Wherein, the origin of the robot coordinate system is positioned at the waist position of the playing robot.

Because the neck part of the playing robot can make anthropomorphic motion, the relative relation between the camera of the playing robot and the waist position of the playing robot is complex, and if the coordinates of each key under the camera coordinate system are directly converted into the robot coordinate system, certain errors can occur. Therefore, the coordinates of the keys under the camera coordinate system can be converted into the robot head end point coordinate system, and then the coordinates of the keys under the robot head end point coordinate system can be converted into the robot coordinate system. The origin of the coordinate system of the tail end point of the robot head is located at the tail end point of the head of the playing robot. Because the relative position of the camera of the robot arranged at the head and the tail end point of the robot is unchanged in the head and neck moving process of the robot, the conversion error value of each key coordinate between the tail end point coordinate system of the robot and the camera coordinate system is smaller.

Since the relative position of the camera of the robot and the end point of the robot head is unchanged in the head and neck moving process of the robot, the coordinate transformation relation between the camera coordinate system and the end point coordinate system of the robot head, namely the second transformation matrix, is fixed. Wherein the second transformation matrix can be predetermined by any computing device according to a fixed relative position between the end point of the head of the playing robot and the camera of the playing robot.

The robot for playing the piano converts the coordinates of each key under the camera coordinate system into the coordinates under the coordinate system of the tail end point of the head of the robot through a second conversion matrix which is determined in advance. And then the robot for playing the musical instrument obtains the relative positions of the tail end point of the head of the robot for playing the musical instrument and the origin of the robot coordinate system under the current pose of the robot for playing the musical instrument according to the relative position data of each component of the robot fed back by the sensor, and further determines the coordinate conversion relation between the tail end point coordinate system of the head of the robot and the robot coordinate system, namely, the third conversion matrix.

After the robot for playing the piano determines the coordinates of each key under the robot coordinate system according to the third transformation matrix, the relative positions between the robot for playing the piano under the robot coordinate system and each key under the robot coordinate system can be determined according to the positions of each key in the robot coordinate system.

S105: and carrying out error compensation on the relative position through a predetermined error fitting function to obtain a compensated relative position, wherein the error fitting function is used for representing the visual positioning error of the piano playing robot to the keys under each pose, and the relative position of the piano playing robot and the keys can be compensated through the error fitting function to obtain the compensated relative position.

The relative positions between the playing robot and the keys determined by the playing robot may have errors due to possible errors in the steps in the determining process, so that the finally determined relative positions between the playing robot and the keys have errors. Therefore, in the specification, the playing robot can perform error compensation on the determined relative positions between the playing robot and each key according to the predetermined error fitting function, and the compensated relative positions are obtained, so that the error of the determined relative positions between the playing robot and each key is reduced.

Wherein the error fitting function needs to be predetermined before use.

When the error fitting function is determined, a plurality of preset poses can be set, namely, the playing robot needs to make actions adaptive to all poses in the function determining process. And then the robot playing the musical instrument acquires images acquired under all preset poses as calibration images, and further determines an error fitting function through the acquired calibration images.

In the process, a plurality of fixed test points can be selected from the key template to serve as key points, then the coordinate of each key point under each pose is determined by the robot playing the piano and is compared with the real coordinate of each key point, so that an error value under each pose is determined, and then an error fitting function is determined according to the error value under each pose. The more preset poses are set, the more key points are selected, the more accurate the error value under each pose is, and the more accurate the obtained error fitting function is.

It should be noted that, the preset poses can be set according to the pitch angle and the horizontal rotation angle of the robot, specifically, the horizontal rotation angle is 0 ° and faces the center of the piano, the pitch angle is 18 °, the zero point is set, in the present specification, the pitch angle range is set to 0-30 °, and the horizontal rotation angle is set to-30 ° for example, without specific limitation.

Aiming at each pose, the robot determines the coordinates of each image feature point contained in the calibration image according to the calibration image acquired under the pose, and then determines the coordinates of each image feature point under a camera coordinate system according to the depth information corresponding to the pixel point where each image feature point is located.

And matching each image characteristic point in the camera coordinate system with each characteristic point in the key template, establishing a coordinate conversion relation between the camera coordinate system in the pose and the world coordinate system according to the matching result, and determining the coordinate of each key point in the camera coordinate system according to the coordinate conversion relation between the camera coordinate system in the pose and the world coordinate system.

And converting the coordinates of each key point in the pose into the coordinates of the tail end point of the robot head by the robot playing the musical instrument through the coordinates of each key point in the camera coordinate system in the pose and a second conversion matrix determined in advance. And then, acquiring a third transformation matrix through the robot playing the musical instrument under the pose according to the positions of the components fed back by the sensors, and transforming the coordinates of each key point under the coordinate system of the tail end point of the head of the robot into the coordinates under the coordinate system of the robot, and taking the coordinates as visual positioning coordinates.

The robot for playing the musical instrument moves the mechanical arm to the position of each key point in the real world, records the coordinates of the mechanical arm in the robot coordinate system after the mechanical arm stays to the position of each key point in the real world, wherein the coordinates of the position of each key point on the mechanical arm in the robot coordinate system are the actual coordinates of each key point in the robot coordinate system.

And determining the distance between the actual coordinates of each key point in the robot coordinate system and the visual positioning coordinates according to the actual coordinates of each key point in the robot coordinate system and the coordinate deviation between the coordinates of each key point in the robot coordinate system determined based on the calibration image.

Then, the average value of the distances between the actual coordinates and the visual positioning coordinates of each key point in the robot coordinate system under the pose can be obtained, so that the visual positioning error value under the pose can be obtained.

And performing surface fitting on the visual positioning error values under each pose to obtain an error fitting function. The surface fitting may be performed by a least square method, which is only a preferred method suitable for the present solution, and of course, the surface fitting may also be performed by a method such as an implicit surface algorithm, a deep learning algorithm, or the like, and the present specification is not particularly limited.

S106: and controlling a mechanical arm on the playing robot to play according to the compensated relative position.

After the relative positions of the keys and the playing robot are acquired, the playing robot compensates the relative positions of the keys and the playing robot according to the error compensation function, so that the compensated relative positions of the keys and the playing robot are obtained, and the correction of the relative positions of the keys and the playing robot is realized.

And the piano playing robot controls the mechanical arm to play each key according to the compensated relative positions of each key and the piano playing robot, so that the real-time playing of the piano is realized.

After the error fitting function is determined in advance, the relative position acquired by the playing robot can be compensated. Taking a piano playing robot for playing a piano in real time as an example, the piano playing robot can compensate the relative positions of the keys and the piano playing robot through an error fitting function after determining the relative positions of the keys and the piano playing robot, and obtain the corrected relative positions of the keys and the piano playing robot, so that accurate playing of the piano is realized through the corrected relative positions of the keys and the piano playing robot.

The control method of the one or more playing robots in the specification is based on the same thought, and the specification also provides a corresponding device, a storage medium and electronic equipment.

Fig. 3 is a schematic structural diagram of a control device based on a constructed playing robot according to an embodiment of the present disclosure, where the device includes:

the acquisition module 301 is configured to perform image acquisition on the key area by using the robot for playing the piano through the set camera, so as to obtain an acquired image;

The identifying module 302 is configured to identify each image feature point from the image, and determine coordinates of each image feature point under a camera coordinate system according to depth information corresponding to each pixel point included in the image and coordinates of each image feature point in the image;

a first determining module 303, configured to determine, according to the coordinates of the feature points of the images in the camera coordinate system and the key template, the coordinates of the keys in the key template in the camera coordinate system;

the second determining module 304 is configured to determine, according to a pose of the playing robot during playing of the piano and coordinates of each key under a camera coordinate system, a relative position between the playing robot and each key under the pose of the playing robot during playing of the piano;

the error compensation module 305 is configured to perform error compensation on the relative position through a predetermined error fitting function, so as to obtain a compensated relative position;

and the control module 306 is used for controlling the mechanical arm on the playing robot to play according to the compensated relative position.

Optionally, the identification module 302 is specifically configured to,

Optionally, the first determining module 303 is specifically configured to,

Optionally, the second determining module 304 is specifically configured to,

Optionally, the second determining module 304 is specifically configured to perform image acquisition on the key area by using the robot for playing the piano through the set camera for each pose, and acquire the acquired calibration image under the pose; identifying each image characteristic point from the calibration image under the pose, and determining the coordinate of each image characteristic point under the camera coordinate system according to the coordinate of each image characteristic point under the pose in the calibration image under the pose and the depth information corresponding to each pixel point in the calibration image under the pose; matching the coordinates of each image feature point in the camera coordinate system under the pose with the coordinates of each feature point in the key template under the world coordinate system, establishing a coordinate conversion relation between the camera coordinate system under the pose and the world coordinate system, and determining the coordinates of each key point under the camera coordinate system under the pose according to the coordinate conversion relation between the camera coordinate system under the pose and the world coordinate system; and determining the coordinates of the key points in the robot coordinate system in the pose of the robot playing the musical instrument according to the coordinates of the key points in the camera coordinate system in the pose of the robot playing the musical instrument, a second transformation matrix which is determined in advance and a third transformation matrix which is determined according to the pose of the robot playing the musical instrument in the playing process.

The present specification also provides a computer-readable storage medium storing a computer program which, when executed by a processor, is operable to perform the control method of the robot for playing a musical instrument provided in fig. 1 described above.

Based on the control method of the playing robot shown in fig. 1, the embodiment of the present disclosure further provides a schematic structural diagram of the electronic device shown in fig. 4. At the hardware level, as in fig. 4, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile storage, although it may include hardware required for other services. The processor reads the corresponding computer program from the nonvolatile memory to the memory and then runs the computer program to realize the control method of the robot playing the musical instrument, which is shown in the figure 1.

Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present description, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.

Claims

1. A control method of a musical instrument playing robot, comprising:

performing error compensation on the relative position through a predetermined error fitting function, and acquiring a compensated relative position, wherein the error fitting function is determined before the playing robot executes the playing, an image acquired by the playing robot under each preset pose is acquired before the playing robot executes the playing, the image is used as a calibration image, for each pose, the coordinates of each key point marked in advance under a robot coordinate system are determined according to the calibration image acquired under the pose, a mechanical arm on the playing robot is moved to the position of each key point in the real world through the playing robot, the coordinates of the mechanical arm in the robot coordinate system after the mechanical arm stays to the position of each key point in the real world are recorded, visual positioning difference values under the pose are determined according to the actual coordinates of each key point in the robot coordinate system and the coordinate deviation of each key point under the robot coordinate system determined based on the calibration image, and the visual positioning difference values under the pose are used for fitting the error of the playing function according to the visual positioning error of the pose;

2. The method according to claim 1, wherein each image feature point is identified from the image, and the determining the coordinates of each image feature point in the camera coordinate system according to the depth information corresponding to each pixel point included in the image and the coordinates of each image feature point in the image specifically includes:

determining a binary image corresponding to the image;

3. The method according to claim 2, wherein determining the coordinates of each image feature point in the camera coordinate system according to the depth information corresponding to each pixel point included in the image and the coordinates of each image feature point in the image specifically includes:

4. The method according to claim 1, wherein determining the coordinates of each key in the key template in the camera coordinate system according to the coordinates of each image feature point in the camera coordinate system and the key template, specifically comprises:

5. The method of claim 1, wherein determining the relative position between the playing robot and each key in the pose of the playing robot during playing of the musical instrument by the pose of the playing robot during playing of the musical instrument and the coordinates of each key in the camera coordinate system specifically comprises:

6. The method according to claim 1, characterized in that for each pose, determining the coordinates of the pre-marked keypoints in the robot coordinate system from the calibration image acquired under that pose, in particular comprises:

7. A control device for a musical instrument playing robot, comprising:

the error compensation module is used for carrying out error compensation on the relative position through a predetermined error fitting function, acquiring a compensated relative position, wherein the error fitting function is determined before the playing robot executes the playing, an image acquired by the playing robot under each preset pose is acquired as a calibration image, for each pose, the coordinates of each preset marked key point under a robot coordinate system are determined according to the calibration image acquired under the pose, a mechanical arm on the playing robot is moved to the position of each key point in the real world through the playing robot, the coordinates of the mechanical arm in the robot coordinate system after the mechanical arm stays to the position of each key point in the real world are recorded, the visual positioning error of the playing robot is determined according to the actual coordinates of each key point in the robot coordinate system and the deviation between the coordinates of each key point under the robot coordinate system determined based on the calibration image, and the visual positioning error is used for fitting the error of the vision positioning function according to the visual positioning error of the playing robot;

8. A computer readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-6.

9. An electronic device comprising a processor and a computer program stored on a memory and executable on the processor, characterized in that the processor implements the method of any of the preceding claims 1-6 when executing the program.