CN113191174B

CN113191174B - Article positioning method and device, robot and computer readable storage medium

Info

Publication number: CN113191174B
Application number: CN202010039632.4A
Authority: CN
Inventors: 刘伟峰; 万保成
Original assignee: Beijing Jingdong Qianshi Technology Co Ltd
Current assignee: Beijing Jingdong Qianshi Technology Co Ltd
Priority date: 2020-01-14
Filing date: 2020-01-14
Publication date: 2024-04-09
Anticipated expiration: 2040-01-14
Also published as: CN113191174A

Abstract

The present disclosure provides an article positioning method, an article positioning device, a robot, and a computer-readable storage medium. The article positioning method comprises the following steps: acquiring a two-dimensional image related to the surface of the article and three-dimensional position information related to the surface of the article; extracting an edge image from the two-dimensional image; extracting corner data from the edge image; processing and analyzing the extracted corner data to obtain a target polygon representing the edge of the surface of the article; determining three-dimensional position information of a target polygon from three-dimensional position information related to the surface of the object; and determining the position of the surface of the object by using the three-dimensional position information of the target polygon.

Description

Article positioning method and device, robot and computer readable storage medium

Technical Field

The present disclosure relates to the field of computer technology, and more particularly, to an article positioning method, an article positioning device, a robot, and a computer-readable storage medium.

Background

With the progress of technology, the degree of mechanization is continuously improved, and the robot is often required to be used for grabbing articles and carrying the articles in the production and logistics processes. For example, during the shipment of goods, a worker picks out goods from a pallet according to a shopping order and gathers the goods onto a tray to form a stack of goods. The trays are then transported to a prescribed location and the robot picks items one by one from the stack of items on a vision-aided basis. At this time, the commodity is in a primary packaging state, namely, the external packaging is provided with a carton, so that the commodity is formed into articles with different sizes.

In the process of realizing the disclosed concept, the inventor finds that under the scene of unstacking objects by using a robot, the type and size information of the objects in the stack are generally unknown, so that the existing vision auxiliary system cannot use the size and the template for auxiliary positioning, thereby bringing great difficulty to the accurate positioning of the objects. If the accurate positioning cannot be realized, the situation that the article cannot be picked up or the article falls off may occur in the following process of grabbing the article and carrying the article.

Disclosure of Invention

In view of this, the present disclosure provides an article positioning method, an article positioning device, a robot, and a computer-readable storage medium.

One aspect of the present disclosure provides a method of article positioning, comprising: acquiring a two-dimensional image related to the surface of an article and three-dimensional position information related to the surface of the article; extracting an edge image from the two-dimensional image; extracting corner data from the edge image; processing and analyzing the extracted corner data to obtain a target polygon representing the edge of the surface of the article; determining three-dimensional position information of the target polygon from the three-dimensional position information related to the object surface; and determining the position of the surface of the object by using the three-dimensional position information of the target polygon.

According to an embodiment of the present disclosure, the article is one article in a stack formed by closely arranging a plurality of articles, and the surfaces of the plurality of articles in at least one direction of the stack are polygonal surfaces having the same number of sides.

According to an embodiment of the present disclosure, the processing and analyzing the extracted corner data to obtain a target polygon characterizing an edge of a surface of the article includes: clustering the corner data to reduce the number of corner points; extracting a plurality of quadrangles formed by connecting angular points; and screening the target rectangles from the quadrangles.

According to an embodiment of the present disclosure, selecting a target rectangle from the plurality of quadrilaterals includes: selecting a quadrangle with the size within a target size range from the quadrangles; screening one or more candidate quadrilaterals according to the coincidence degree between quadrilaterals with the sizes within the target size range; generating a convolution kernel corresponding to the size of the candidate quadrangle for each of the one or more candidate quadrangles, the convolution kernel having a girdle-shaped rectangular border region; and matching the corresponding candidate quadrilaterals by using the girdle-shaped rectangular border areas of the corresponding convolution kernels of the candidate quadrilaterals to obtain the target rectangle.

According to an embodiment of the present disclosure, matching each candidate quadrangle with the girdle-shaped rectangular border area of the corresponding convolution kernel of the candidate quadrangle to obtain the target rectangle includes: assigning 1 to the endless-belt-shaped rectangular frame region and assigning 0 to a region other than the endless-belt-shaped rectangular frame region; convolving the candidate quadrilaterals corresponding to the convolution check by utilizing the convolution check to obtain a convolution value; judging whether the matching degree of the candidate quadrangle and the ring belt-shaped rectangular frame area is larger than a specified value by utilizing the convolution value; and if the matching degree is greater than a predetermined value, determining the candidate quadrangle as the target rectangle.

According to an embodiment of the present disclosure, if there are a plurality of the candidate quadrilaterals having a degree of matching with the corresponding endless-belt-shaped rectangular frame region greater than a prescribed value, the candidate quadrilaterals having the highest degree of matching are determined as the target rectangle.

According to an embodiment of the present disclosure, when extracting a plurality of quadrangles connected by corner points, angles of four corners of the extracted quadrangles are all in a range of 80 ° to 100 °.

According to an embodiment of the present disclosure, after determining the position of the surface of the article, the method further includes: calculating a target picking pose for picking the articles according to the information of the positions of the surfaces of the articles; and sending the target picking pose to a robot so that the robot picks the article according to the target picking pose.

Another aspect of the present disclosure provides an article positioning device, comprising: the acquisition module is used for acquiring a two-dimensional image related to the surface of the article and three-dimensional position information related to the surface of the article; an edge image extraction module for extracting an edge image from the two-dimensional image; the angular point data extraction module is used for extracting angular point data from the edge image; the processing analysis module is used for processing and analyzing the extracted corner data to obtain a target polygon representing the edge of the surface of the article; a target polygon position determining module, configured to determine three-dimensional position information of the target polygon from the three-dimensional position information related to the object surface; and an article surface position determining module for determining a position of the article surface using the three-dimensional position information of the target polygon.

According to an embodiment of the present disclosure, the article is a rectangular parallelepiped article, and the processing analysis module includes: the corner clustering sub-module is used for clustering the corner data so as to reduce the number of the corners; the quadrilateral extraction submodule is used for extracting a plurality of quadrilaterals formed by connecting angular points; and the target rectangle screening submodule is used for screening the target rectangle from the quadrangles.

According to an embodiment of the present disclosure, the target rectangle screening submodule includes: a quadrangle primary selection unit for selecting quadrangles with the size within the target size range from the quadrangles; a candidate quadrilateral screening unit, configured to screen one or more candidate quadrilaterals according to the degree of coincidence between quadrilaterals whose sizes are within a target size range; a convolution kernel generation unit configured to generate, for each of the one or more candidate quadrilaterals, a convolution kernel corresponding to a size of the candidate quadrilaterals, the convolution kernel having a girdle-shaped rectangular frame region; and a convolution kernel matching unit, configured to match the corresponding candidate quadrilaterals by using the girdle-shaped rectangular border area of the corresponding convolution kernel of each candidate quadrilaterals, so as to obtain the target rectangle.

According to an embodiment of the present disclosure, the convolution kernel matching unit includes: a assigning subunit, configured to assign 1 to the ring-belt-shaped rectangular frame area and assign 0 to an area other than the ring-belt-shaped rectangular frame area; a convolution subunit, configured to obtain a convolution value by convolving the candidate quadrilaterals corresponding to the convolution check; a judging subunit configured to judge whether or not a matching degree between the candidate quadrangle and the endless-belt-shaped rectangular frame region is greater than a predetermined value using the convolution value; and a determination subunit configured to determine the candidate quadrangle as the target rectangle if the matching degree is greater than a predetermined value.

According to an embodiment of the present disclosure, if there are a plurality of the candidate quadrilaterals having a degree of matching with the corresponding endless-belt-shaped rectangular frame region greater than a prescribed value, the determination subunit determines the candidate quadrilaterals having the highest degree of matching as the target rectangle.

According to an embodiment of the present disclosure, angles of four corners of the quadrangle extracted by the quadrangle extraction submodule are all in a range of 80 ° to 100 °.

According to an embodiment of the present disclosure, the article positioning device further includes: the target picking pose calculation module is used for calculating a target picking pose for picking the articles according to the information of the positions of the surfaces of the articles; and the sending module is used for sending the target picking pose to a robot so that the robot can pick the article according to the target picking pose.

Another aspect of the present disclosure provides a robot including: one or more processors; and a memory for storing one or more instructions that, when executed by the one or more processors, cause the one or more processors to implement the method described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing executable instructions that, when executed by a processor, cause the processor to implement the above-described method.

According to the embodiment of the disclosure, according to the obtained two-dimensional image related to the surface of the article and the three-dimensional position information related to the surface of the article, the article can be positioned more accurately by performing a series of processing on the two-dimensional image related to the surface of the article without using special equipment, the problem that the conventional vision assistance system needs to use the size and the template for assisting in positioning is solved at least to a certain extent, and the effect of simply realizing the more accurate positioning of the article is achieved.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:

FIG. 1 schematically illustrates an application scenario of an article positioning method and apparatus according to an embodiment of the disclosure;

FIG. 2 schematically illustrates a flow chart of a method of article positioning according to an embodiment of the disclosure;

FIG. 3 schematically illustrates a flow chart of a method of article positioning according to an embodiment of the disclosure;

FIG. 4 schematically illustrates a flow chart of a method of article positioning according to an embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of a method of article positioning according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a diagram of a manner of assigning values to a girdle-like rectangular border area of a convolution kernel in accordance with an embodiment of the present disclosure;

fig. 7-16 schematically illustrate one particular embodiment of an article positioning method according to an embodiment of the present disclosure;

FIG. 17 schematically illustrates a block diagram of an article positioning apparatus according to an embodiment of the disclosure; and

fig. 18 schematically shows a block diagram of a robot according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," means having at least one of A, B and C "shall include, but not be limited to, means having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "at least one of A, B or C, etc." is used, in general such a convention should be used in the sense one having skill in the art would understand the convention (e.g., "a device having at least one of A, B or C" would include but not be limited to devices having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Embodiments of the present disclosure provide an article positioning method. The article positioning method comprises the following steps: acquiring a two-dimensional image related to the surface of the article and three-dimensional position information related to the surface of the article; extracting an edge image from the two-dimensional image; extracting corner data from the edge image; processing and analyzing the extracted corner data to obtain a target polygon representing the edge of the surface of the article; determining three-dimensional position information of a target polygon from three-dimensional position information related to the surface of the object; and determining the position of the surface of the object by using the three-dimensional position information of the target polygon.

Fig. 1 schematically illustrates an application scenario in which an article positioning method and apparatus may be applied according to an embodiment of the present disclosure. It should be noted that fig. 1 illustrates only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios.

As shown in fig. 1, articles of different sizes in the initial packaged state are collected on a tray 140 to form a stack, wherein the stack includes articles 130. According to embodiments of the present disclosure, the plurality of articles may be in a rectangular parallelepiped shape, it being noted that in embodiments of the present disclosure, the rectangular parallelepiped shape may include a square shape. The tray 140 is disposed at a predetermined position, and the imaging device 110 is disposed above the predetermined position. Image information of the article 130 is photographed by the photographing device 110 and transmitted to the robot 120, and the robot 120 positions the article 130 based on the image information and then grips and transports the article 130. Robot 120 may, for example, grasp and transport article 130 by sucking the upper surface of article 130 through pick mechanism 121. The action of robot 120 to grasp items 130 from a stack is also referred to as unstacking.

In this application, for example, the articles 130 in the stack have a rectangular surface shape in a plan view, but are of various sizes because of a large variety of types. In other words, the article 130 is one article in a stack in which a plurality of articles are closely arranged, and the surfaces of the plurality of articles in at least one direction of the stack are polygonal surfaces having the same number of sides, but the size of the surfaces of the article 130 is unknown.

In this application scenario, if the article 130 cannot be positioned more accurately, in the following process of capturing the article and carrying the article, the article may not be picked up or the article may fall off. In view of the specific requirements of destacking, in locating article 130, it is necessary to accurately know shape and size information of the upper surface of article 130 to calculate the center of gravity of article 130, while it is necessary to accurately know corresponding depth information of the upper surface of article 130 to determine the lowering height of pickup mechanism 121.

The current object positioning techniques mainly include a template matching technique, a point cloud segmentation technique, a technique of directly predicting a bounding box of an object through deep learning, and the like, but these object positioning techniques are not suitable for the above application scenarios for the following reasons.

(1) The template matching technology is used for locating the object by pre-storing templates of the possibly occurring objects, finding a target template matched with the object from the pre-stored templates when locating the object, and utilizing the information of the target template. If the type and size of the article are unknown, the template of the article cannot be stored in advance. Thus, the template matching technique is not suitable for the above application scenario.

(2) The point cloud segmentation technique divides the point cloud according to the characteristics of space, geometry, texture, and the like, and locates the point cloud having similar characteristics within the same division as one item, the technique requires the use of a 3D camera having a sufficiently high resolution, and is computationally intensive. In a case where one article in a stack in which a plurality of articles are closely arranged, particularly in a case where the heights of a plurality of articles in a stack are also close, it is difficult to divide directly by a point cloud. Thus, the point cloud segmentation technique is not applicable to the above application scenario.

(3) The technology of directly predicting the bounding boxes of the articles through deep learning needs to collect data of all the articles in the warehouse to perform model training, so that the workload is huge, the articles in the warehouse are frequently updated, the model needs to be frequently updated to adapt to the update of the articles, and the practical application is not facilitated. Therefore, the technique of directly predicting the bounding box of an item by deep learning is not applicable to the above application scenario.

Aiming at the technical pain point of target positioning in the mixed article unstacking application scene, the present disclosure provides an effective article positioning method, which avoids the defect that the traditional template matching technology needs to rely on the template and the point cloud segmentation technology to separate the targets on the same layer; meanwhile, the defect that the model needs to be frequently updated in practical application by the technology of directly predicting the bounding box of the object through deep learning is overcome. The technical scheme of the disclosure does not depend on the size and the template of the article, namely, the article can be positioned more accurately.

Fig. 2 schematically illustrates a flow chart of an article positioning method according to an embodiment of the present disclosure.

As shown in fig. 2, the method includes operations S210 to S260.

In operation S210, a two-dimensional image related to the surface of the article and three-dimensional position information related to the surface of the article are acquired.

This operation S210 is described in detail in connection with the scene example of fig. 1. As shown in fig. 1, the object surface may be, for example, an upper surface of the object 130, and the act of acquiring a two-dimensional image related to the object surface and three-dimensional position information related to the object surface may be implemented, for example, by the camera 110. The photographing device 110 may be provided with one 2D camera and one 3D camera, for example. The 2D camera and the 3D camera are configured by using an external parameter calibration technology, so that plane position information shot by the 2D camera for the same point is consistent with plane position information shot by the 3D camera in a space cartesian coordinate system. That is, since the external parameter calibration based on the same coordinate system is performed for the above-described 2D camera and 3D camera, each point photographed by the 2D camera has one determined three-dimensional position information, and the plane position and depth position thereof can be determined. Thus, the two-dimensional image may be obtained by photographing with a 2D camera, for example, and the three-dimensional position information may be obtained by photographing with a 3D camera, for example. The 2D camera and the 3D camera may be general cameras.

The two-dimensional image related to the surface of the object 130 may be a two-dimensional image including the surface of the object captured by a 2D camera, and the two-dimensional image may be an RGB image or a gray-scale image. In the case where the two-dimensional image is an RGB image, conversion into a grayscale image is required for convenience of processing described later.

For example, the two-dimensional image associated with the surface of article 130 may be a two-dimensional image taken with a 2D camera having 400 ten thousand pixels in an area including 1.5 x 2m of the surface of article 130. That is, the two-dimensional image related to the surface of the object 130 includes not only the two-dimensional image of the surface of the object 130 but also the two-dimensional image of the surface of other objects in the photographing area even the ground, the data amount is large, and there is disturbance data. Thus, the surface of article 130 may not be precisely located from the two-dimensional image associated with the surface of the article.

Three-dimensional positional information associated with the surface of article 130 may include spatial coordinate information of article 130, from which, for example, a particular spatial position of the surface of article 130 may be determined. The three-dimensional position information may be captured using a 3D camera, but may also be obtained in other ways.

Although the conventional 2D camera can acquire a planar image of an object, spatial information of the object cannot be directly acquired. In addition, the conventional 3D camera can acquire three-dimensional position information of an article, but cannot directly generate a planar image which is convenient to recognize. Thus, it is difficult to achieve more accurate positioning of the item using either a conventional 2D camera or 3D camera alone.

In operation S220, an edge image is extracted from the two-dimensional image.

The edge image may be extracted from the two-dimensional image by a deep learning method, for example, a general feature extraction network such as RCF network, deep content network, etc. may be used. The general feature extraction networks do not need to update the network model due to frequent updating of the stored articles, so that the convenience of practical application is improved. The edge image extracted from the two-dimensional image in this way includes not only an image of the edge of the actual article surface but also an image of the edge of some pattern existing on the article surface or the like, the data amount is still relatively large, and interference data exists. Thus, the surface of article 130 may not be precisely located from the edge image.

In operation S230, corner data is extracted from the edge image. The extraction of corner data from edge images may for example use Harris corner detection techniques.

Article 130 has a polygonal surface according to embodiments of the present disclosure. Corner data is extracted from the edge image in order to determine the planar position of the corners of the polygonal surface of the article.

By this operation S230, it is possible to determine roughly tens of thousands of corner points.

In operation S240, the extracted corner data is processed and analyzed to obtain a target polygon representing an edge of the surface of the article.

According to an embodiment of the present disclosure, the article 130 may be one article in a stack of a plurality of articles closely arranged, and the surfaces of the plurality of articles in at least one direction of the stack are polygonal surfaces having the same number of sides. In this way, the polygonal surfaces of the plurality of articles in at least one direction of the stack, although not of the same size, can be determined by connecting the same number of corner points due to the same number of sides. The target polygon characterizing the edges of the article surface obtained in operation S240 can reflect the planar position of the polygonal surface of the article with high accuracy.

In operation S250, three-dimensional position information of the target polygon is determined from the three-dimensional position information related to the surface of the object.

As described above, since the 2D camera and the 3D camera are calibrated based on the external parameters of the same coordinate system, and the target polygon representing the edge of the surface of the object is obtained based on the two-dimensional image captured by the 2D camera, each point of the target polygon has a certain three-dimensional position information. Thus, after the target polygon is determined, the three-dimensional position information of the target polygon can be determined from the three-dimensional position information related to the surface of the object.

In operation S260, the position of the surface of the object is determined using the three-dimensional position information of the target polygon. In practice, the three-dimensional position information of the target polygon can be regarded as the position of the object surface, directly for the subsequent processing.

Through the series of processes of operations S210 to S260, according to the obtained two-dimensional image related to the surface of the object and the three-dimensional position information related to the surface of the object, the object can be positioned more accurately without using special equipment, the problem that the existing vision assistance system needs to use the size and the template for assistance positioning is solved at least to a certain extent, and the effect of simply realizing the more accurate positioning of the object is achieved. Thus, accurate positioning of the article can be achieved independent of the size and template of the article.

Fig. 3 schematically illustrates a flow chart of an article positioning method according to an embodiment of the present disclosure.

After determining the location of the surface of the article, operations S270 to S280 may be further included according to an embodiment of the present disclosure as follows.

In operation S270, a target picking pose for picking an item is calculated from information of the position of the item surface. After the position of the surface of the article is determined, the size, the height, the gravity center and the like of the surface of the article can be obtained. Thus, for example, the translational position, the lowering height, and the like of the pickup mechanism 121 of the robot 120 can be calculated.

In operation S280, the target picking pose is transmitted to the robot so that the robot picks the item according to the target picking pose.

Through the operation, the object picking can be realized by using the object positioning method disclosed by the embodiment of the disclosure to assist the robot.

Fig. 4 schematically illustrates a flow chart of an article positioning method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, article 130 may be a cuboid article, where article 130 has a rectangular surface. In this case, as shown in fig. 4, operation S240 may further include the following operations S410 to S430.

In operation S410, the corner points are clustered to reduce the number of corner points. Many neighbor corner points are collected near the corners of the article, and if each corner point participates in calculation, the calculation amount is too large and no substantial benefit exists. Therefore, the number of corner points can be reduced by clustering the corner points, and the operation speed of the processing described later can be improved. Corner clustering uses, for example, a DBSCAN clustering algorithm. For example, by this operation S410, about one hundred corner points can be selected from the tens of thousands of corner points.

In operation S420, a plurality of quadrangles connected by corner points are extracted. Specifically, four corner points are selected each time, and the four corner points are connected in pairs to form a quadrangle. In order to reduce the operational interference, only convex quadrangles formed by connecting corner points can be extracted.

According to the embodiments of the present disclosure, when extracting a plurality of quadrangles connected by corner points, angles of four corners of the extracted quadrangles may be all in a range of 80 ° to 100 °. This is because, in the case where the article 130 has a rectangular surface, angles of four corners of the extracted quadrangle are in a range close to right angles, only the quadrangle close to the rectangle can be extracted, and the operation interference can be reduced. In addition, the angles of the four corners are each in the range of 80 ° to 100 °, and the angles of the four corners are each in the range of 90 ° ± 10%, but the range may be, of course, a range of 90 ° ± 5% or the like, depending on the actual situation.

In operation S430, a target rectangle is selected from among a plurality of quadrangles. The target rectangle is the target polygon representing the edge of the surface of the article, and can reflect the plane position of the rectangular surface of the article with high precision.

Through the series of processes of operations S410 to S430 described above, the data amount can be effectively reduced, and the target rectangle for article positioning can be quickly obtained without requiring knowledge of the size of the article or storage of templates corresponding to the article in advance.

Fig. 5 schematically illustrates a flow chart of an article positioning method according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, as shown in fig. 5, operation S430 may further include the following operations S510 to S540.

In operation S510, a quadrilateral having a size within a target size range is selected from among a plurality of quadrilaterals.

Typically, the size of the items in the stack is not determined, but is not too large or too small. Therefore, a target size range is set, and only quadrilaterals with sizes within the target size range are screened out. For example, the target size range may be set to 20cm by 30cm to 80cm by 100cm. If the quadrangle with the four sides of 50cm, 60cm, 50cm and 60cm is found, screening the quadrangle and carrying out the next treatment; if a quadrangle with four sides of 5cm, 6cm, 5cm and 6cm is found, the quadrangle is directly abandoned to reduce data interference. The target size range may be appropriately set according to an actual usage scenario. For example, the target size range may also be set according to the size of the smallest item that the robot can pick.

In operation S520, one or more candidate quadrilaterals are screened out according to the degree of coincidence between quadrilaterals having a size within the target size range.

In actual processing, a plurality of highly coincident quadrilaterals may occur, which often correspond to only one real article surface. According to this concept, one representative quadrangle may be selected from a plurality of highly overlapping quadrangles for processing described later. For example, if the overlap ratio of two quadrangles with each other is 80% or more, one of the two quadrangles is reserved. The criteria for how to determine which quadrangle to reserve can be set as needed. For example, one quadrilateral that is closer to a rectangle may be retained.

In operation S530, for each of the one or more candidate quadrilaterals, a convolution kernel corresponding to the size of the candidate quadrilaterals is generated, the convolution kernel having an endless-belt-like rectangular border region. The length and width of the convolution kernel may, for example, be consistent with the length and width of the candidate quadrangle. That is, the outer edge of the girdle-shaped rectangular border area of the convolution kernel may coincide with the length and width of the candidate quadrangle, for example, while its inner edge may be several pixels wide from the outer edge.

In operation S540, the respective candidate quadrilaterals are matched with the girdle-shaped rectangular frame region of the corresponding convolution kernel of each candidate quadrilaterals to obtain the target rectangle. For example, the gray value of the portion of the corresponding candidate quadrangle falling within the above-described ring-belt-like rectangular frame region may be multiplied by the value on the convolution kernel.

Fig. 6 schematically illustrates a schematic diagram of a manner of assigning values to a girdle-shaped rectangular border area of a convolution kernel in accordance with an embodiment of the present disclosure.

According to an embodiment of the present disclosure, operation S540 may specifically include the following procedure.

As shown in fig. 6, 1 is assigned to the endless rectangular frame region and 0 is assigned to a region other than the endless rectangular frame region.

And convoluting the corresponding candidate quadrilaterals by using a convolution check to obtain a convolution value.

Specifically, the gray value of the portion of the corresponding candidate quadrangle falling within the above-described endless rectangular border region may be multiplied by the value 1 within the endless rectangular border region, and the resulting products may be summed.

The values of the edge image and the non-edge image obtained in the foregoing operation are greatly different. For example, the gradation value is one of image data, and in the case where the edge of the two-dimensional image is displayed in black, the gradation value of the edge image is lower, and the gradation value of the non-edge image is higher.

If the candidate quadrangle is a true target rectangle, the candidate quadrangle should correspond to the edge area of the two-dimensional image, and the above convolution value is low because the gray value of the edge image is low.

If the candidate quadrangle is not a true target rectangle, at least a part of the candidate quadrangle does not correspond to the edge region of the two-dimensional image, and the above convolution value is high because the gray value of the non-edge image is high. The degree of matching of the candidate quadrilaterals to the girdle-like rectangular border area may be characterized by the convolution value.

Thus, a prescribed value for the convolution value can be set, and it can be determined whether or not the degree of matching of the candidate quadrangle and the endless-belt-like rectangular frame region is greater than the prescribed value using the convolution value.

And if the matching degree is larger than the specified value, determining the candidate quadrangle as a target rectangle.

According to the embodiment of the present disclosure, if there are a plurality of candidate quadrilaterals having a degree of matching with the corresponding endless belt-like rectangular frame region greater than a prescribed value, the candidate quadrilaterals having the highest degree of matching are determined as target rectangles.

Fig. 7 to 16 schematically illustrate one specific embodiment of an article positioning method according to an embodiment of the present disclosure.

Fig. 7 schematically shows a top view of an article 130 and a tray 140 on which the article 130 is placed. The article 130 and the tray 140 in fig. 7 correspond to the article 130 and the tray 140 in fig. 1.

For ease of illustration, only one item 130 on tray 140 is described. The same process is used for other items on the tray 140.

In this embodiment, a photographing device composed of one 2D camera and one 3D camera is provided, but a diagram of the photographing device is omitted here.

As described above, the range shown in fig. 7 is the shooting area of the 2D camera, and the size is 1.5×2m.

For this photographing region, a two-dimensional image of a plane is obtained by a 2D camera having 400 ten thousand pixels, and three-dimensional position information corresponding to the two-dimensional image is obtained by a 3D camera.

The two-dimensional image includes not only a two-dimensional image of the surface of the object 130, but also other objects, trays, the ground, and the like in the photographing region.

Next, an edge image is extracted from the two-dimensional image.

Fig. 8 schematically shows the extracted edge image. In fig. 8, the edge image is displayed in black, and the non-edge image is displayed in white.

The edge image includes not only an image of the edge of the actual article surface, but also an image of the edge of some pattern present on the article surface, etc.

Next, corner data is extracted from the edge image using Harris corner detection techniques.

Fig. 9 schematically shows a part of the corner points extracted.

It should be noted that, at this time, there are about several thousands to ten thousands of extracted corner points, and the main corner point gathering positions are indicated by dots in fig. 9.

Then, clustering the corner data using a DBSCAN clustering algorithm.

Fig. 10 schematically shows a part of corner points after clustering.

The corner points after clustering are about one hundred.

Next, a plurality of quadrangles connected by corner points are extracted.

Fig. 11 schematically shows a part of a quadrilateral connected by corner points.

Since the upper surface of the article 130 in this embodiment is rectangular, only a quadrangle close to the rectangle can be extracted, and the operation interference can be reduced. For example, in fig. 11, only three quadrangles shown by a solid black line may be extracted.

Next, a quadrangle having a size within the target size range is selected from the plurality of quadrangles.

The upper right quadrangle in fig. 11 is discarded due to undersize.

And screening one or more candidate quadrilaterals according to the coincidence degree between quadrilaterals with the size within the target size range.

In actual processing, a plurality of highly coincident quadrilaterals may occur, which often correspond to only one real article surface. If the overlap ratio of the two quadrangles with each other is 80% or more, one of the two quadrangles is reserved. The criteria for how to determine which quadrangle to reserve can be set as needed. For example, one quadrilateral that is closer to a rectangle may be retained. This screening according to the degree of coincidence is not shown in fig. 11.

Fig. 12 schematically illustrates the candidate quadrilaterals screened out.

For the candidate quadrangle on the left, the minimum bounding rectangle of the candidate quadrangle is obtained by using four vertex coordinates (u 0, v 0), (u 1, v 1), (u 2, v 2), (u 3, v 3) of the candidate quadrangle, thereby obtaining the length L and the width W of the minimum bounding rectangle. This operation is implemented by the minAreRect function of OpenCV. In addition, the minimum circumscribed rectangle length L and width W can be obtained through the minAreRect function, and the inclination angle of the minimum circumscribed rectangle can be obtained, so that whether the minimum circumscribed rectangle is an inclined rectangle or not can be judged through the inclination angle.

And designing a convolution kernel with a ring-belt-shaped rectangular frame region according to the length L and the width W of the minimum circumscribed rectangle obtained by the candidate quadrangle. The length and width of the convolution kernel are consistent with those of the minimum circumscribed rectangle, and the width of the annular rectangular frame region can be 10 pixels. The value on the band is 1 and the value on the inside is 0. Then, 1 is assigned to the endless-belt-shaped rectangular frame region and 0 is assigned to a region other than the endless-belt-shaped rectangular frame region. Fig. 13 shows convolution kernels corresponding to the candidate quadrilaterals to the left in fig. 12.

After the minimum circumscribed rectangle is obtained, edge image data corresponding to the candidate quadrangle can be intercepted in the two-dimensional image according to the minimum circumscribed rectangle.

If the minimum circumscribed rectangle is an inclined rectangle, the image can be rotated through a warp Affine function of OpenCV, and after the rectangle is rotated, the edge image data is intercepted in the two-dimensional image.

Fig. 14 shows, in a hatched portion, edge image data taken according to the candidate quadrangle to the left in fig. 12.

And (3) marking the intercepted edge image data as I, marking the convolution kernel designed in the last step as K, and calculating the convolution as follows:

the value of the annular rectangular frame region of the convolution kernel is 1, and the value of the inside is 0. Therefore, by the above calculation, the gradation value data of the edge image of 10 pixels width of the endless-belt-like rectangular frame region is summed up.

As described above, the edge image is displayed in black, and the gray value is low; the non-edge image is displayed as white with a higher gray value.

In the candidate quadrangle on the left in fig. 12, the black part in the border region of the endless belt-like rectangle is large, and therefore the summation value is low, and the candidate quadrangle is determined as the target rectangle.

The candidate quadrangle in the center in fig. 12 is judged by the same method as described above. Fig. 15 shows a convolution kernel corresponding to the central candidate quadrangle in fig. 12.

Fig. 16 shows, in a hatched portion, edge image data taken from the candidate quadrangle in the center in fig. 12.

In the candidate quadrangle in the center in fig. 12, the white portion in the border region of the endless belt-like rectangle is large, and therefore the sum value is high, so that the candidate quadrangle is not judged as the target rectangle.

One specific embodiment of the present disclosure is described above based on fig. 7 to 16, but the present disclosure is of course not limited to the above specific embodiment.

Another aspect of the present disclosure provides an article positioning device.

Fig. 17 schematically illustrates a block diagram of an article positioning device according to an embodiment of the disclosure.

As shown in fig. 17, article positioning apparatus 1700 of an embodiment of the present disclosure may include an acquisition module 1710, an edge image extraction module 1720, a corner data extraction module 1730, a process analysis module 1740, a target polygon position determination module 1750, and an article surface position determination module 1760.

The acquisition module 1710 is configured to acquire a two-dimensional image related to the surface of the article and three-dimensional positional information related to the surface of the article.

The edge image extraction module 1720 is configured to extract an edge image from a two-dimensional image.

The corner data extraction module 1730 is configured to extract corner data from an edge image.

The processing analysis module 1740 is configured to process and analyze the extracted corner data to obtain a target polygon that characterizes an edge of the surface of the object.

The target polygon position determination module 1750 is configured to determine three-dimensional position information of the target polygon from the three-dimensional position information associated with the surface of the article.

The article surface position determination module 1760 is configured to determine a position of the article surface using the three-dimensional position information of the target polygon.

According to an embodiment of the present disclosure, the article is one article in a stack of a plurality of articles closely arranged, and the surfaces of the plurality of articles in at least one direction of the stack are polygonal surfaces having the same number of sides.

According to an embodiment of the present disclosure, the article is a cuboid article. In this case, the processing analysis module 1740 may include a corner clustering sub-module, a quadrilateral extraction sub-module, and a target rectangle screening sub-module.

The corner clustering submodule is used for clustering the corner data so as to reduce the number of the corners.

The quadrilateral extraction submodule is used for extracting a plurality of quadrilaterals formed by connecting angular points.

The target rectangle screening submodule is used for screening target rectangles from the quadrangles.

According to an embodiment of the present disclosure, the target rectangle screening sub-module may include a quadrilateral initial selection unit, a candidate quadrilateral screening unit, a convolution kernel generation unit, and a convolution kernel matching unit.

The quadrilateral initial selection unit is used for screening quadrilateral with the size within the target size range from a plurality of quadrilaterals.

The candidate quadrilateral screening unit is used for screening one or more candidate quadrilaterals according to the coincidence degree between quadrilaterals with the size within the target size range.

The convolution kernel generation unit is configured to generate, for each of one or more candidate quadrilaterals, a convolution kernel corresponding to a size of the candidate quadrilaterals, the convolution kernel having an endless-belt-shaped rectangular border region.

The convolution kernel matching unit is used for matching the corresponding candidate quadrilaterals by utilizing the girdle-shaped rectangular frame area of the corresponding convolution kernel of each candidate quadrilaterals so as to obtain the target rectangle.

According to an embodiment of the present disclosure, the convolution kernel matching unit may include an assignment subunit, a convolution subunit, a judgment subunit, and a determination subunit.

The assignment subunit is configured to assign 1 to the ring-belt-shaped rectangular frame area and assign 0 to an area other than the ring-belt-shaped rectangular frame area.

The convolution subunit is used for carrying out convolution on the corresponding candidate quadrilaterals by utilizing the convolution check to obtain a convolution value.

The judging subunit is used for judging whether the matching degree of the candidate quadrangle and the girdle-shaped rectangular frame area is larger than a specified value or not by utilizing the convolution value.

The determination subunit is configured to determine the candidate quadrangle as the target rectangle if the matching degree is greater than a prescribed value.

According to the embodiment of the present disclosure, if there are a plurality of candidate quadrilaterals having a degree of matching with the corresponding endless belt-like rectangular frame region greater than a prescribed value, the determination subunit determines the candidate quadrilaterals having the highest degree of matching as the target rectangle.

According to the embodiment of the present disclosure, angles of four corners of the quadrangle extracted by the quadrangle extraction submodule are all in a range of 80 ° to 100 °.

According to embodiments of the present disclosure, the item positioning apparatus may further include a target picking pose calculation module and a transmission module.

The target picking pose calculating module is used for calculating the target picking pose for picking the article according to the information of the position of the surface of the article.

The sending module is used for sending the target picking pose to the robot so that the robot can pick the object according to the target picking pose.

According to the embodiment of the disclosure, the article positioning device is independent of the size and the template of the article, i.e. can realize accurate positioning of the article.

Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in any other reasonable manner of hardware or firmware that integrates or encapsulates the circuit, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which when executed, may perform the corresponding functions.

For example, any of the acquisition module 1710, the edge image extraction module 1720, the corner data extraction module 1730, the processing analysis module 1740, the target polygon position determination module 1750, and the article surface position determination module 1760 may be incorporated in one module/unit/sub-unit or any of them may be split into a plurality of modules/units/sub-units. Alternatively, at least some of the functionality of one or more of these modules/units/sub-units may be combined with at least some of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to embodiments of the present disclosure, at least one of the acquisition module 1710, the edge image extraction module 1720, the corner data extraction module 1730, the processing analysis module 1740, the target polygon position determination module 1750, and the item surface position determination module 1760 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in hardware or firmware in any other reasonable manner of integrating or packaging circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, at least one of the acquisition module 1710, the edge image extraction module 1720, the corner data extraction module 1730, the processing analysis module 1740, the target polygon position determination module 1750, and the item surface position determination module 1760 may be at least partially implemented as a computer program module, which, when executed, may perform the corresponding functions.

It should be noted that, the article positioning device in the embodiment of the present disclosure corresponds to the article positioning method in the embodiment of the present disclosure, and the description of the article positioning device specifically refers to the article positioning method and is not described herein.

The robot of the disclosed embodiments includes one or more processors and a memory for storing one or more instructions that, when executed by the one or more processors, cause the one or more processors to implement the methods described above.

Fig. 18 schematically shows a block diagram of a robot according to an embodiment of the disclosure. The robot shown in fig. 18 is only one example, and should not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.

As shown in fig. 18, a robot 1800 according to an embodiment of the present disclosure includes a processor 1801, which may perform various appropriate actions and processes according to programs stored in a Read Only Memory (ROM) 1802 or programs loaded from a storage section 1808 into a Random Access Memory (RAM) 1803. The processor 1801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special-purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 1801 may also include on-board memory for caching purposes. The processor 1801 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM1803, various programs and data necessary for the operation of the robot 1800 are stored. The processor 1801, ROM1802, and RAM1803 are connected to each other by a bus 1804. The processor 1801 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM1802 and/or the RAM 1803. Note that the program can also be stored in one or more memories other than the ROM1802 and the RAM 1803. The processor 1801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the disclosure, the robot 1800 may also include an input/output (I/O) interface 1805, the input/output (I/O) interface 1805 also being connected to the bus 1804. The robot 1800 may also include one or more of the following components connected to the I/O interface 1805: an input section 1806 including a keyboard, a mouse, and the like; an output portion 1807 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage section 1808 including a hard disk or the like; and a communication section 1809 including a network interface card such as a LAN card, a modem, or the like. The communication section 1809 performs communication processing via a network such as the internet. The drive 1810 is also connected to the I/O interface 1805 as needed. Removable media 1811, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memory, and the like, is installed as needed on drive 1810 so that a computer program read therefrom is installed as needed into storage portion 1808.

According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 1809, and/or installed from the removable medium 1811. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1801. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

The computer-readable storage medium of the embodiments of the present disclosure stores executable instructions that, when executed by a processor, cause the processor to implement the methods described above.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

For example, according to embodiments of the present disclosure, the computer-readable storage medium may include the ROM 1802 and/or the RAM 1803 described above and/or one or more memories other than the ROM 1802 and the RAM 1803.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of methods, apparatus, robots, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. An article positioning method comprising:

acquiring a two-dimensional image related to the surface of an article and three-dimensional position information related to the surface of the article;

extracting an edge image from the two-dimensional image;

extracting corner data from the edge image;

processing and analyzing the extracted corner data to obtain a target polygon representing the edge of the surface of the article;

determining three-dimensional position information of the target polygon from three-dimensional position information related to the object surface; and

determining the position of the surface of the object by utilizing the three-dimensional position information of the target polygon;

wherein the article is a cuboid article,

The processing and analyzing the extracted corner data to obtain a target polygon characterizing an edge of the surface of the object comprises:

clustering the corner data to reduce the number of corner points;

extracting a plurality of quadrangles formed by connecting angular points; and

and screening the target rectangles from the quadrilaterals.

2. The method of claim 1, wherein the article is one of a stack of a plurality of articles arranged in a close relationship, the surfaces of the plurality of articles in at least one direction of the stack being polygonal surfaces having the same number of sides.

3. The method of claim 1, wherein screening out a target rectangle from the plurality of quadrilaterals comprises:

selecting quadrilaterals with the size within a target size range from the quadrilaterals;

screening one or more candidate quadrilaterals according to the coincidence degree between quadrilaterals with the size within the target size range;

generating, for each of the one or more candidate quadrilaterals, a convolution kernel corresponding to a size of the candidate quadrilaterals, the convolution kernel having an annulus-like rectangular border region; and

and matching the corresponding candidate quadrilaterals by using the girdle-shaped rectangular frame area of the corresponding convolution kernel of each candidate quadrilaterals so as to obtain the target rectangle.

4. The method of claim 3, wherein matching each candidate quadrilateral with its corresponding annulus-like rectangular border region of the convolution kernel to obtain the target rectangle comprises:

assigning 1 to the girdle-shaped rectangular frame area and assigning 0 to an area except for the girdle-shaped rectangular frame area;

convolving the corresponding candidate quadrilaterals by utilizing the convolution check to obtain a convolution value;

judging whether the matching degree of the candidate quadrangle and the girdle-shaped rectangular frame area is larger than a specified value or not by utilizing the convolution value; and

and if the matching degree is larger than a specified value, determining the candidate quadrangle as the target rectangle.

5. The method of claim 4, wherein,

and if a plurality of candidate quadrilaterals with the matching degree with the corresponding ring-belt-shaped rectangular frame area being larger than a specified value exist, determining the candidate quadrilaterals with the highest matching degree as the target rectangle.

6. The method according to claim 1, wherein, when extracting a plurality of quadrangles formed by connecting corner points, angles of four corners of the extracted quadrangles are each in a range of 80 ° to 100 °.

7. The method of claim 1, wherein after determining the location of the article surface, further comprising:

calculating a target picking pose for picking the article according to the information of the position of the surface of the article; and

and sending the target picking pose to a robot so that the robot picks the object according to the target picking pose.

8. An article positioning device comprising:

the acquisition module is used for acquiring a two-dimensional image related to the surface of the article and three-dimensional position information related to the surface of the article;

an edge image extraction module for extracting an edge image from the two-dimensional image;

the angular point data extraction module is used for extracting angular point data from the edge image;

the processing analysis module is used for processing and analyzing the extracted corner data to obtain a target polygon representing the edge of the surface of the article;

a target polygon position determining module, configured to determine three-dimensional position information of the target polygon from the three-dimensional position information related to the object surface; and

the object surface position determining module is used for determining the position of the object surface by utilizing the three-dimensional position information of the target polygon;

Wherein the article is a cuboid article,

the process analysis module comprises:

the corner clustering sub-module is used for clustering the corner data to reduce the number of corners;

the quadrilateral extraction submodule is used for extracting a plurality of quadrilaterals formed by connecting angular points; and

and the target rectangle screening sub-module is used for screening the target rectangle from the quadrangles.

9. A robot, comprising:

one or more processors;

a memory for storing one or more instructions,

wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1 to 7.

10. A computer readable storage medium storing executable instructions which, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 7.