CN115997560B

CN115997560B - Fruit picking method, system and device based on mechanical arm

Info

Publication number: CN115997560B
Application number: CN202211729092.8A
Authority: CN
Inventors: 杨顺; 海丹; 韩威; 郑思仪
Original assignee: Beijing Zhongke Power Technology Co ltd
Current assignee: Beijing Zhongke Power Technology Co ltd
Priority date: 2022-12-30
Filing date: 2022-12-30
Publication date: 2023-12-01
Anticipated expiration: 2042-12-30
Also published as: CN115997560A

Abstract

The invention discloses a fruit picking method, system and device based on a mechanical arm, relates to the technical field of agricultural picking machinery, and mainly aims to improve the picking accuracy of a target to be picked. The main technical scheme of the invention is as follows: determining a first position of at least one target to be picked according to the image acquired by the first vision module; determining a picking sequence list of the targets to be picked according to the first position and the current position of the mechanical arm; controlling the mechanical arm to move the second vision module to a first position corresponding to a target to be picked based on the picking sequence list; determining the size information and picking position of the target to be picked according to the image acquired by the second vision module; adjusting the size of a picking opening of the fruit picking module according to the size information; and controlling the fruit picking module to pick the target to be picked according to the picking position. The invention is used for picking fruits.

Description

Fruit picking method, system and device based on mechanical arm

Technical Field

The invention relates to the technical field of agricultural picking machinery, in particular to a fruit picking method, system and device based on a mechanical arm.

Background

Along with the continuous improvement of the social science and technology level, agricultural production gradually develops towards the directions of mechanization, automation and intellectualization, for example, in the past, people need to pick according to the maturity, coloring, size and the like of fruits and vegetables selectively, but the labor consumption is large, the progress is slow, and the picking progress and the follow-up development are influenced, so that the machine is gradually adopted to replace the manual picking, the labor cost is effectively reduced, and the profit of fruits and vegetables is improved.

However, the camera of the vision module adopted by the existing grabbing type picking machine is usually fixed, and the camera view angle is insufficient, so that the situation that the camera is blocked and the situation of specific positioning judgment errors of the target to be picked possibly occur, and finally, the picking accuracy of the target to be picked is low.

Disclosure of Invention

In view of the above problems, the invention provides a fruit picking method, system and device based on a mechanical arm, which mainly aims to improve the picking accuracy of a target to be picked.

In order to solve the technical problems, the invention provides the following scheme:

in a first aspect, the present invention provides a fruit picking method based on a mechanical arm, wherein two vision modules and a fruit picking module are arranged on the mechanical arm, and the method comprises:

Determining a first position of at least one target to be picked according to the image acquired by the first vision module;

determining a picking sequence list of the targets to be picked according to the first position and the current position of the mechanical arm;

controlling the mechanical arm to move the second vision module to a first position corresponding to a target to be picked based on the picking sequence list;

determining the size information and picking position of the target to be picked according to the image acquired by the second vision module;

adjusting the size of a picking opening of the fruit picking module according to the size information;

and controlling the fruit picking module to pick the target to be picked according to the picking position.

In a second aspect, the present invention provides a fruit picking device based on a mechanical arm, wherein two vision modules and a fruit picking module are arranged on the mechanical arm, and the device comprises:

the first determining unit is used for determining a first position of at least one object to be picked according to the image acquired by the first vision module;

the second determining unit is used for determining a picking sequence list of the targets to be picked according to the first position and the current position of the mechanical arm;

the control unit is used for controlling the mechanical arm to move the second vision module to a first position corresponding to a target to be picked based on the picking sequence list;

The third determining unit is used for determining the size information and the picking position of the object to be picked according to the image acquired by the second vision module;

the first adjusting unit is used for adjusting the size of the picking opening of the fruit picking module according to the size information;

the picking unit is used for controlling the fruit picking module to pick the target to be picked according to the picking position.

In order to achieve the above object, according to a third aspect of the present invention, there is provided a storage medium comprising a stored program, wherein the apparatus in which the storage medium is controlled to execute the robot-based fruit picking method of the first aspect when the program is run.

In order to achieve the above object, according to a fourth aspect of the present invention, there is provided a processor for running a program, wherein the program is run to perform the robot-based fruit picking method of the first aspect.

By means of the technical scheme, the fruit picking method, system and device based on the mechanical arm can acquire images through the first visual module, then at least one target to be picked is determined according to the images acquired by the first visual module, and further a first position of the at least one target to be picked is determined, and the first position is a position to be moved by the second visual module, so that the target to be picked is convenient to observe through cooperation between the two visual modules, the subsequent target to be picked is convenient to pick, then the current position of the mechanical arm can be further determined, and further the picking sequence list of the target to be picked can be determined according to the first position and the current position of the mechanical arm, so that picking can be performed according to the picking list later, rather than random picking, and further the mechanical arm can be controlled to move the second visual module to the first position corresponding to the target to be picked based on the picking sequence list, so that the second visual module can be further picked at the first position, the size of the first visual module can be further picked at the first position can be determined, and the size of the first visual module can be picked can be further picked according to the first position, and the size of the first visual module can be picked can be further picked at the first position, and the size can be picked can be further picked according to the first size, the picking device has the advantages that targets to be picked with different sizes can be picked, the picking device is not limited to targets to be picked with uniform sizes, the applicability of picking different targets to be picked is improved, and after the sizes of the picking openings of the fruit picking modules are adjusted, the fruit picking modules can be controlled to pick the targets to be picked according to the picking positions. Thus, the first position of the target to be picked is determined through the image shot by the first vision module, then the second vision module is used for further collecting the target to be picked at the first position, the size information and the picking position of the target to be picked are determined, rough judgment is not carried out on the target to be picked only by using the independent vision module, the recognition and positioning precision of the target to be picked are improved to a certain extent, then the size of the picking opening of the fruit picking module can be adjusted according to the determined size information of the target to be picked, the target to be picked is picked on the basis, and therefore the applicability of different targets to be picked can be improved through adjusting the size of the picking opening according to the size of the target to be picked, and the accuracy of the target to be picked can be improved finally through increasing the recognition and positioning precision of the target to be picked to be different sizes.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:

fig. 1 shows a schematic perspective view of a fruit picking system based on a mechanical arm according to an embodiment of the present invention;

fig. 2 shows a flowchart of a fruit picking method based on a mechanical arm according to an embodiment of the present invention;

FIG. 3 shows a flowchart of another fruit picking method based on a mechanical arm according to an embodiment of the present invention;

fig. 4 shows a block diagram of a fruit picking device based on a mechanical arm according to an embodiment of the present invention;

Fig. 5 shows a block diagram of another fruit picking device based on a mechanical arm according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

In the continuous improvement process of the social science and technology level, the requirements and the efficiency of the agricultural production are continuously improved, the agricultural production gradually develops towards the directions of mechanization, automation and intellectualization, for example, in the past, because the maturity of the fruits is inconsistent, people need to pick selectively according to indexes such as coloring, size and the like, the bottleneck links of the industrial development are the maximum manual consumption, but the agricultural robot which can reduce the labor cost and improve the fruit and vegetable profit is gradually adopted for picking the fruits, but when the existing agricultural robot is used for picking the fruits, the maturity and the positioning of the fruits are roughly judged by utilizing a single camera, then the fruits are picked, but the positioning of the fruits is not accurate enough, and on the other hand, the existing agricultural robot camera is usually fixedly arranged on the robot and cannot move, so that the judgment of the camera on the fruits is not accurate enough, and finally, the clamping force of the existing picking machine is difficult to reach the optimum, the fruits are usually extruded and even scratched, and the storage time is influenced. Therefore, the embodiment of the invention provides a fruit picking method based on a mechanical arm, which can improve the picking accuracy of a target to be picked.

The fruit picking method based on the mechanical arm provided by the embodiment of the invention is described below by taking the fruit picking system based on the mechanical arm as an executing body as an example.

Referring to fig. 1, fig. 1 is a schematic perspective view of a fruit picking system based on a mechanical arm according to an embodiment of the invention. As shown in fig. 1, in a fruit picking system of a practical application scene, the system comprises a four-axis robot arm 3 for picking fruits, two vision modules, a fruit picking module 4 and a control module 12; the first vision module 1 is arranged on the upper surface of a first motion joint of the four-axis mechanical arm 3, the first vision module is driven to move up and down by the movement of the mechanical arm, and the second vision module 2 and the fruit picking module 4 are arranged on a third motion joint of the four-axis mechanical arm 3; in this system, the first vision module 1 may determine the maturity of the picked object in the target field, determine at least one target to be picked in the target field and the first position where the second vision module 2 moves based on the maturity of the picked object, and further the second vision module 2 may determine the picking position and size information of the target to be picked, and finally the fruit picking module 4 may adjust the size of the picking opening according to the size information of the target to be picked to pick the target to be picked; the control module 12 is electrically connected with the four-axis robot arm 3, the two vision modules and the fruit picking module 4 respectively, and executes the fruit picking method based on the robot arm, so that the system finishes picking the target to be picked.

Further, on the basis of the fruit picking system based on the mechanical arm, the system further comprises a picking port 5, a fruit temporary storage box 6, a negative pressure source 9, a photoelectric sensor 10, a fruit storage box 11 and a power supply module 13, wherein the power supply module 13 is used for supplying power to the fruit picking system when in operation, the picking port 5 is connected with the negative pressure source 9 and is used for sucking a target to be picked into the picking port 5 through the negative pressure suction of the negative pressure source 9, the fruit picking module 4 further comprises a reducing assembly and a driving assembly, the reducing assembly is arranged at a cylinder port (not shown) of the picking port 5 and can be contracted or expanded relative to the direction of the picking port 5 so as to expand or reduce the size of the picking port 5, and the driving assembly is movably connected with the reducing assembly (not shown) so as to drive the contraction or expansion of the reducing assembly; the fruit temporary storage box 6 is connected to the lower part of the fruit picking module 4 and is used for temporarily storing the picked objects which fall into the lower part of the fruit temporary storage box 6 after the picked objects are successfully picked, wherein a cover plate is further arranged below the fruit temporary storage box 6, the fruit storage box 11 is positioned below the fruit temporary storage box 6 and is connected with the cover plate below the fruit temporary storage box 6, and the picked objects can successfully fall into the fruit storage box 11 when the cover plate below the fruit temporary storage box 6 is opened so as to ensure the integrity of the picked objects.

When the fruit picking module 4 works, the driving assembly can drive the reducing assembly to shrink or expand according to the size information of the target to be picked so as to adjust the size of the picking opening 5, the target to be picked can be picked at the picking position, after the target to be picked is picked successfully, the picked target can fall into the fruit temporary storage box 6 connected to the lower part of the fruit picking module 4, meanwhile, the control module 12 receives the jump signal of the photoelectric sensor 10 to verify that the target to be picked is picked successfully, meanwhile, the control module 12 can detect the bearing of the fruit temporary storage box 6 through a pressure sensor (not shown) arranged below the fruit temporary storage box 6, and then when the fruit temporary storage box 6 is detected to reach a specified bearing threshold, a cover plate below the fruit temporary storage box 6 is opened, and the picked target is unloaded into the fruit storage box 11.

The control module 12 is a control board with AI computing capability, and CAN be connected with two or more cameras, and CAN communicate with each module through a serial port or CAN to control each module to complete the designated work.

Referring to fig. 2, fig. 2 is a flowchart of a fruit picking method based on a mechanical arm according to an embodiment of the present invention:

201. A first position of at least one target to be picked is determined from the image acquired by the first vision module.

In this step, the mechanical arm may first drive the first vision module to move up and down, so that the first vision module may first collect images of all picked objects in the target field of view, where it is to be noted that, the first position may be a position moved by the second vision module, and the position is used by the second vision module to further collect images of the objects to be picked.

Specifically, when determining the first position of at least one target to be picked, the first vision module may determine the primary positioning information of the target to be picked according to the image with the target to be picked captured by the first vision module, and then may determine the first position only according to the primary positioning information of the target to be picked, or may also determine the first position according to the primary positioning information combined with the position of the second vision module.

The first positions may be multiple, and the specific number is not limited, and one target to be picked may correspond to at least one first position, or there may be cases where part of the first positions between different targets to be picked are the same.

Wherein the first vision module may be a depth (RGBD) camera or a binocular camera with depth perception capabilities; the second machine vision module selects a depth (RGBD) camera or a binocular camera with depth perception capability or a monocular RGB camera without depth perception capability.

202. And determining a picking sequence list of the targets to be picked according to the first position and the current position of the mechanical arm.

In the step, after determining the first position of at least one object to be picked according to the image acquired by the first vision module, the current position of the mechanical arm can be further determined, and then a picking sequence list of the object to be picked is determined according to the first position and the current position of the mechanical arm.

Specifically, when determining the picking sequence list of the targets to be picked by using the first position and the current position of the mechanical arm, the distance between each target to be picked and the current position of the mechanical arm may be determined according to the first position, then the maturity of each target to be picked is determined by using the first vision module, and further the maturity and the distance are weighted and combined, or the picking sequence list of the targets to be picked may be determined by planning the optimal path of each target to be picked by using the mechanical arm according to the first position corresponding to each target to be picked and the current position of the mechanical arm.

For example, when determining the picking order list by using a weighted combination of the maturity and the distance, if the distance between the target a to be picked and the current position of the mechanical arm is 2 meters, the maturity is first-stage maturity, the distance between the target B to be picked and the current position of the mechanical arm is 3 meters, the maturity is third-stage maturity, the distance between the target C to be picked and the current position of the mechanical arm is also 1 meter, and the maturity is second-stage maturity, it should be noted that in this embodiment, the higher the maturity level, the more mature the target to be picked, based on this, the picking order list may be the target C to be picked-the target B to be picked in turn.

203. And controlling the mechanical arm to move the second vision module to a first position corresponding to the target to be picked based on the picking sequence list.

In this step, after the picking order list is determined, a corresponding target to be picked may be determined according to the order in the picking order list, and then, according to the target to be picked, one of the first positions corresponding to the target to be picked is determined, where the determined first position may be the closest to the current position of the mechanical arm, or may be the farthest distance but the best viewing angle, which is not limited herein, and further, the mechanical arm may be controlled to move the second vision module to the first position corresponding to the target to be picked.

For example, if the first object to be picked in the picking sequence list is a, the corresponding first positions of the first object to be picked are respectively A1 position closest to the current position of the mechanical arm and A2 position farthest from the current position of the mechanical arm but with the best viewing angle, in this case, if the object to be picked is to be better observed, the A2 position may be selected, then, the mechanical arm may be further controlled to move the second vision module to the A2 position, if the object to be picked is to be observed faster, the mechanical arm may be controlled to move the second vision module to the A1 position, and further, it is to be noted that the viewing angle of each first position may reach the minimum standard for observing the object to be picked.

204. And determining the size information and picking position of the object to be picked according to the image acquired by the second vision module.

In this step, after the second vision module is moved to the first position, the second vision module may be controlled to collect an image of the object to be picked at the first position, then, by a feature-based method, or a semantic segmentation method based on deep learning, the picking position of the object to be picked and the size information of the object to be picked are determined by combining the image of the object to be picked collected by the second vision module, where it is required to be explained that the image of the object to be picked collected by the second vision module may be a plurality of images, and when the size information of the object to be picked is estimated, the equivalent diameter of the object to be picked may be estimated, for example, a tomato is assumed, and at this time, the tomato is a sphere, mainly, the equivalent diameter of the sphere is estimated when the tomato is a sphere.

The picking positions and the size information of the targets to be picked can be multiple, and the targets to be picked can be in different directions and different angles.

205. And adjusting the size of a picking opening of the fruit picking module according to the size information.

In this step, after determining the size information, the size of the picking opening of the fruit picking module may be further determined, and then the size of the picking opening and the size of the target to be picked may be compared or a difference value between the two may be calculated, to determine whether the picking opening of the fruit picking module needs to be adjusted, if so, further, the size of the picking opening needs to be adjusted according to the difference value, that is, how much the size of the picking opening needs to be reduced or increased based on the original size, where it needs to be described that, if the size of the picking opening does not exceed too much the size of the target to be picked, that is, although the size of the picking opening is larger than the size of the target to be picked, the picking opening may just need to be inhaled, and no adjustment is required.

For example, if the size of the object to be picked is 4cm and the size of the picking opening is 3cm, it may be determined that the size of the picking opening is 1cm smaller than the object to be picked, so it may be determined that the size of the picking opening needs to be increased by 1-2cm at this time to ensure that the object to be picked can be inhaled.

206. And controlling a fruit picking module to pick the target to be picked according to the picking position.

In the step, after the picking position is determined, the mechanical arm can be controlled to drive the fruit picking module to a picking target, and then the fruit picking module is controlled to pick the target to be picked.

When the target to be picked is picked, the target to be picked can be sucked into the picking opening by utilizing suction force, and then the picking opening can be reduced to the minimum after the target to be picked is sucked, so that the target to be picked cannot fall out at will.

Based on the implementation manner of fig. 2, the fruit picking method based on the mechanical arm provided by the invention can acquire images through the first visual module, then determine the first position of at least one target to be picked according to the images acquired by the first visual module, wherein the first position is the position to be moved by the second visual module so as to facilitate further acquisition of the target to be picked, thus, through cooperation between the first visual module and the second visual module, the second visual module can conveniently perform subsequent secondary positioning, increase the judgment precision of the target to be picked, facilitate improvement of the accuracy rate of the target to be picked when the target to be picked is carried out, then further determine the current position of the mechanical arm, further determine the picking sequence list of the target to be picked according to the first position and the current position of the mechanical arm, thus, the picking sequence of the target to be picked can be ensured when the target to be picked is picked according to the subsequent picking list, the speed of the target to be picked can be facilitated to be improved, further control the position to be controlled by the first visual module so as to further acquire the position to be picked when the target to be picked, the first position is further determined by the first visual module, the first position can be further determined based on the first position of the first visual module, and the position to be picked by the first visual module can be further determined when the target to be picked by the first visual module, at least the first position is more accurate when the target to be picked by the first visual module is picked, the second vision module is further utilized to determine the size information and the picking position of the targets to be picked, then the sizes of the picking openings of the fruit picking modules are adjusted according to the size information, the targets to be picked with different sizes can be picked, the targets to be picked are not limited to the targets to be picked with uniform sizes, the applicability of picking different targets to be picked is improved, and after the sizes of the picking openings of the fruit picking modules are adjusted, the fruit picking modules can be controlled to pick the targets to be picked according to the picking positions. Thus, the first position of the target to be picked is determined through the image shot by the first vision module, then the second vision module is used for further collecting the target to be picked at the first position, the size information and the picking position of the target to be picked are determined, the target to be picked is not roughly judged by the aid of the independent vision module, the recognition and the positioning accuracy of the target to be picked are improved to a certain extent, then the size of the picking opening of the fruit picking module can be adjusted according to the determined size information of the target to be picked, the target to be picked is picked on the basis, and therefore the applicability of different targets to be picked can be improved through picking the size of the picking opening according to the target to be picked, and the accuracy of picking the target to be picked can be improved through increasing the recognition and the positioning accuracy of the target to be picked and adjusting the size of the picking opening.

Further, as a refinement and expansion of the embodiment shown in fig. 2, the embodiment of the invention further provides another fruit picking method based on a mechanical arm, as shown in fig. 3, which specifically includes the following steps:

301. a first position of at least one target to be picked is determined from the image acquired by the first vision module.

In this step, a preferred embodiment of determining the first position of at least one object to be picked is provided, which is different from the preferred embodiment in step 201, specifically, the first vision module may be controlled to collect an image of the object to be picked and image depth information of the object to be picked in the field of view, after the image depth information of the object to be picked is determined, the maturity of the object to be picked may be determined by using a deep learning method or a feature-based method on the basis of the image depth information of the object to be picked, then the maturity of the object to be picked may be determined from the objects to be picked, and further, primary positioning information of the object to be picked may be determined based on the image depth information, and at least one first position of the object to be picked may be determined.

Further, when determining at least one first position of the object to be picked according to the primary positioning information of the object to be picked, the parameter information of the second vision module may be determined first, then the picking distance suitable for the second vision module to be picked is determined according to the parameter information of the second vision module, then the distance and the position of the second vision module may be combined, and the first position of the object to be picked and the primary positioning information of the object to be picked may be determined, so that the second vision module may conveniently and secondarily position the object to be picked subsequently, and accuracy of the object to be picked is improved.

The number of picked objects in the target field of view can be multiple, and the number of picked objects can be all observed by driving the first vision module to move up and down by the mechanical arm.

The object to be picked is a picked object which can be picked and is determined from picked objects, and the object to be picked can be picked later.

302. And determining a picking sequence list of the targets to be picked according to the first position and the current position of the mechanical arm.

In this step, a more preferable embodiment, different from the embodiment in step 202, is provided when determining the picking order list of the objects to be picked, specifically, the current position of the fruit picking module located at the third motion joint of the mechanical arm is determined, then an optimal path for picking each picking object by the mechanical arm is planned according to the first position of the objects to be picked and the current position of the fruit picking module, and then the picking order of each object to be picked is determined according to the optimal path, so as to obtain the picking order list.

For example, if the fruit picking module is located at A1, the first position of the target 1 to be picked is located at A2 in the southeast direction 2 m away from the A1 position, the first position of the target 2 to be picked is located at A3 in the northwest direction 2 m away from the A1 position, and the first position of the target 3 to be picked is located at A4 in the southeast direction 1 m away from the A1 position, the picking order list may be the target 3 to be picked-target 1 to be picked-target 2 to be picked, so that by determining the picking order list according to an optimal path, and picking according to the picking order list, the picking efficiency of the target to be picked may be greatly increased.

303. And controlling the mechanical arm to move the second vision module to a first position corresponding to the target to be picked based on the picking sequence list.

In this step, it is provided that when the second vision module is moved to the first position corresponding to the target to be picked, different from the preferred embodiment of step 203, specifically, the first target to be picked in the picking sequence list may be determined first, then a first position of the target to be picked is determined, then the second vision module may be controlled to collect the image corresponding to the target to be picked and the image depth information at the corresponding first position, further, whether the target to be picked is blocked may be determined by using a feature-based method or a semantic segmentation method based on deep learning on the basis of the image corresponding to the target to be picked, if yes, other first positions corresponding to the target to be picked may be replaced, if the target to be picked is already determined to be blocked according to the images photographed at all the first positions corresponding to the target to be picked, then the mechanical arm may be controlled to move the second vision module to the first position corresponding to the target to be picked in the next sequence list, thereby implementing that the blocked target to be picked is blocked, and avoiding failure in efficiency due to avoiding failure in picking is caused by avoidance of the blocked target.

For example, if the first order objects to be picked in the picking order list are the objects to be picked 1, the second order objects to be picked are the objects to be picked 2, all the first positions corresponding to the objects to be picked 1 are A1, A2 and A3, the second vision module captures the image of the objects to be picked 1 at the A1, and further determines that the objects to be picked 1 are blocked by the image captured at the A1, the second vision module can be replaced to capture the image of the objects to be picked 1 at the A2 corresponding to the objects to be picked 1, and if the objects to be picked 1 captured at the A1 and the A2 are all determined that the objects to be picked 1 are blocked, the mechanical arm can be controlled to move the second vision module to the first position corresponding to the objects to be picked 2.

304. And determining the size information and picking position of the object to be picked according to the image acquired by the second vision module.

In this step, a more preferable embodiment is provided, which is different from the embodiment of step 204 when determining the size information and the picking position of the target to be picked, specifically, if the first position corresponding to the target to be picked is determined that the target to be picked is not occluded, the multiple image information and the corresponding size information of the target to be picked may be analyzed according to the multiple images collected by the target to be picked, where the multiple images of the target to be picked may be photographed at different first positions corresponding to the target to be picked, and may be photographed at different angles, and then the multiple size information of the target to be picked may be determined at different angles, and then the target size information of the target to be picked may be determined from the multiple size information, for example, the target size information of the target to be picked may be the largest size information, thus the target size information may be obtained from the multiple size information, and the depth information may be not obtained from the multiple images, and the target size information may be obtained from the multiple images, and the target size information may not be obtained from the depth information.

305. And adjusting the size of a picking opening of the fruit picking module according to the size information.

In this step, a more preferable embodiment, different from the embodiment in step 205, is provided when the size of the picking opening is adjusted, specifically, the size information of the object to be picked and the size information of the picking opening of the fruit picking module can be determined first, the difference between the size information and the size information can be determined, and then the size of the picking opening and the direction of adjustment can be determined according to the difference, for example, the size of the picking opening is adjusted to shrink inwards or expand outwards, and then the driving component in the fruit picking module can be controlled to drive the reducing component in the fruit picking module to shrink or expand according to the size and the direction of adjustment, so as to adjust the size of the picking opening.

306. And controlling a fruit picking module to pick the target to be picked according to the picking position.

In this step, a more preferable embodiment, different from step 2056, is provided when the target to be picked is picked, specifically, when the target to be picked is picked, the target to be picked may be detected in time, whether the target to be picked is picked to the target to be picked within a specified time threshold is detected, where the specified time threshold is the maximum time used for picking the target to be picked, if the target to be picked is not picked within the specified time threshold, the suction force of a negative pressure source for providing suction force to suck the target to be picked may be detected, and then the size of the picking port may be adjusted according to the suction force of the negative pressure source, that is, if the target to be picked is not picked within the specified time threshold, and the suction force of the negative pressure source reaches a first suction threshold, the target to be picked may cause difficulty in picking due to the size of the picking port, the picking port may be increased by the specified suction force, if the suction force of the negative pressure source is less than the maximum time used for picking the target to be picked, if the size of the target to be picked within the specified time threshold is not picked to the target to be picked, the size of the target to be picked may be detected, the size of the negative pressure source may be reduced by the second threshold, the size may be increased by the specified suction force, and the picking port may be reduced due to the size of the opening may be picked due to the size is increased in time threshold.

307. And detecting whether a fruit temporary storage box in the fruit picking module reaches a specified bearing threshold value or not through a pressure sensor.

In this step, after picking the target to be picked is accomplished, the target to be picked can fall into automatically and be located fruit temporary storage box in the fruit picking module, can utilize the pressure sensor that sets up fruit temporary storage box below simultaneously detect in real time whether the weight of fruit temporary storage box reaches appointed bearing threshold, wherein, appointed bearing threshold is the biggest weight that fruit temporary storage box can bear, so that fruit temporary storage box surpasses the condition of appointed bearing threshold and causes subsequent target to be picked can't fall into fruit temporary storage box, causes the picking target to fall on ground, leads to the condition that the picking target damaged appears.

308. And controlling the cover plate below the fruit temporary storage box to be opened, and unloading the picked picking target into the fruit storage box in the fruit picking module.

In the step, if the detection reaches the specified bearing threshold, the cover plate below the fruit temporary storage box can be controlled to be opened, and then the picked picking target can be unloaded into the fruit storage box of the fruit picking module, so that the picked picking target can be safely placed in the fruit storage box, and the integrity of the picked target is ensured.

In some possible embodiments, when the fruit picking module is controlled to pick the target to be picked according to the picking position in step 306, the method further includes:

detecting whether a jump signal of a photoelectric sensor in the fruit picking module is received within a preset time period, wherein the photoelectric sensor is arranged on the inner wall of a picking port;

if yes, determining that the target to be picked is picked successfully, and switching to the next target to be picked in the picking sequence list for picking;

if not, determining that the picking of the targets to be picked fails, and discarding picking the targets to be picked when the picking failure times of the same targets to be picked reach the designated times, and listing the targets to be picked into a difficult-to-pick list.

In this step, when picking the target to be picked, whether a jump signal of the photoelectric sensor on the inner wall of the picking port is received or not may be detected in a preset time period, if the jump signal of the photoelectric sensor is received, it may be determined that the picking of the target to be picked is successful, then the picking may be switched to the next target to be picked in the picking sequence list, if the jump signal of the photoelectric sensor is not detected, then the picking failure of the target to be picked may be determined, and the picking failure number may be recorded, at this time, the mechanical arm may be further controlled to retreat so as to tear the picking target off, when the target to be picked is pulled off, it may be further detected whether the jump signal of the photoelectric sensor is received, if the jump signal is successful, then the picking operation may be switched to the next target to be picked in the picking sequence list, if the picking failure number of times of the same target to be picked reaches a designated number, and the picking of targets to be picked may be abandoned in the picking sequence list, and the situation that the picking efficiency of the same target to be picked is not to be picked may be infinitely is avoided, and the situation of decreasing the picking efficiency may occur.

Under the condition that picking is successful, a target which is successfully picked automatically falls into a fruit temporary storage box below the fruit picking module, so that if picking is successful, the photoelectric sensor can jump to generate a jump signal, and if failure occurs, the photoelectric sensor cannot jump, and therefore the jump signal of the photoelectric sensor cannot be detected.

The number of times of the assignment may be 3 times or 5 times, which may be determined according to specific service conditions, and is not limited in any way.

In some possible embodiments, before adjusting the size of the picking port of the fruit picking module according to the size information in step 305, the method further comprises:

determining whether the size information of the target to be picked is within the size adjustment range of the picking port of the fruit picking module;

if not, the picking is abandoned.

In this step, before picking the target to be picked, whether the size information of the target to be picked is within the size adjustment range of the picking port or not may be determined in advance according to the size information of the target to be picked and the size information of the picking port of the fruit picking module, for example, the size information of the target to be picked is greater than the maximum size of the picking port of the fruit picking module, if the size information is reached, picking of the target to be picked may be abandoned, and thus, occurrence of a situation that picking failure caused by that the size of the target to be picked exceeds the pickable range of the fruit picking module wastes picking time is avoided, and thus picking efficiency may be ensured to a certain extent.

Further, as an implementation of the method shown in fig. 2, the embodiment of the invention further provides a fruit picking device based on the mechanical arm, which is used for implementing the method shown in fig. 2. The embodiment of the device corresponds to the embodiment of the method, and for convenience of reading, details of the embodiment of the method are not repeated one by one, but it should be clear that the device in the embodiment can correspondingly realize all the details of the embodiment of the method. As shown in fig. 4, the apparatus includes:

a first determining unit 401, configured to determine a first position of at least one object to be picked according to the image acquired by the first vision module;

a second determining unit 402, configured to determine a picking order list of the objects to be picked according to the first position determined by the first determining unit 401 and the current position of the mechanical arm;

a control unit 403, configured to control the mechanical arm to move the second vision module to a first position corresponding to the object to be picked based on the picking order list determined by the second determining unit 402;

a third determining unit 404, configured to determine size information and picking positions of the objects to be picked according to the image acquired by the second vision module controlled by the control unit 403;

A first adjusting unit 405, configured to adjust a picking port size of the fruit picking module according to the size information determined by the third determining unit 404;

and a picking unit 406, configured to control the fruit picking module to pick the target to be picked according to the picking position determined by the third determining unit 404.

Further, as an implementation of the method shown in fig. 3, the embodiment of the invention further provides another fruit picking device based on the mechanical arm, which is used for implementing the method shown in fig. 3. The embodiment of the device corresponds to the embodiment of the method, and for convenience of reading, details of the embodiment of the method are not repeated one by one, but it should be clear that the device in the embodiment can correspondingly realize all the details of the embodiment of the method. As shown in fig. 5, the apparatus includes:

In an alternative embodiment, the first determining unit 401 includes:

a first control module 4011, configured to control the first vision module to collect an image of a picked object in a target field of view and image depth information of the picked object, and analyze maturity of the picked object based on the image of the picked object;

a first determining module 4012, configured to determine at least one target to be picked from picked according to the maturity of the picked determined by the first control module 4011, and determine primary positioning information of the at least one target to be picked based on image depth information corresponding to the target to be picked;

A second determining module 4013, configured to determine a first position corresponding to each object to be picked according to the primary positioning information determined by the first determining module 4012.

In an alternative embodiment, the second determining module 4013 is specifically configured to:

determining a distance suitable for shooting the picking target by the second vision module according to the parameter information of the second vision module;

and determining the first position of the object to be picked according to the distance, the position of the second vision module and the primary positioning information of the object to be picked determined by the first vision module.

In an alternative embodiment, the second determining unit 402 includes:

a third determination module 4021 for determining a current position of the fruit picking module;

a planning module 4022, configured to plan an optimal path for the mechanical arm to pick each target to be picked according to the first position corresponding to each target to be picked and the current position of the fruit picking module determined by the third determining module 4021;

a fourth determining module 4023, configured to determine a picking order of each target to be picked based on the optimal path planned by the planning module 4022, so as to obtain a picking order list.

In an alternative embodiment, the control unit 403 includes:

A second control module 4031, configured to control the mechanical arm to move the second vision module to a first position corresponding to the object to be picked according to the picking sequence in the picking sequence list;

the judging module 4032 is configured to use the second control module 4031 to move a second vision module to a corresponding first position to collect an image corresponding to a target to be picked and image depth information, and judge whether the target to be picked is blocked based on the image corresponding to the target to be picked;

and a replacing module 4033, configured to replace the first position corresponding to the target to be picked if the judging module 4032 judges that the target to be picked is blocked, until the second vision module has judged that the target to be picked is blocked at all the first positions corresponding to the target to be picked, and then control the mechanical arm to move the second vision module to the first position corresponding to the next target to be picked in the picking sequence list.

In an alternative embodiment, the third determining unit 404 includes:

the analysis module 4041 is configured to analyze, if the target to be picked is not blocked, a plurality of image depth information and corresponding size information of the target to be picked according to the plurality of images acquired by the second vision module;

A fifth determining module 4042, configured to determine target size information of the target to be picked from the plurality of size information analyzed by the analyzing module 4041;

a sixth determining module 4043, configured to determine a picking position of the object to be picked according to the image depth information corresponding to the object size information determined by the fifth determining module 4042.

In an alternative embodiment, the first adjusting unit 405 includes:

a seventh determining module 4051, configured to determine and adjust the size and the adjustment direction of the picking opening in the fruit picking module according to the size information;

the driving module 4052 is configured to control the driving assembly in the fruit picking module to drive the reducing assembly in the fruit picking module to shrink or expand according to the size and the adjustment direction determined by the seventh determining module 4051, so as to adjust the size of the picking opening.

In an alternative embodiment, when the picking unit 406 controls the fruit picking module to pick the object to be picked according to the picking position, the apparatus further includes a second adjusting unit 407, and the second adjusting unit 407 includes:

a first detection module 4071, configured to detect whether the target to be picked is picked within a specified time threshold;

The adjusting module 4072 is configured to adjust the size of the picking opening according to the suction force of the negative pressure source in the fruit picking module if the detecting module 4071 detects no, wherein if the suction force of the negative pressure source reaches a first suction threshold, the picking opening is increased by a specified size according to the size information of the target to be picked, and if the suction force of the negative pressure source is smaller than a second suction threshold, the picking opening is decreased by the specified size according to the size information of the target to be picked, and the second suction threshold is smaller than the first suction threshold.

In an alternative embodiment, when the picking unit 406 controls the fruit picking module to pick the object to be picked according to the picking position, the apparatus further includes a detecting unit 408, and the detecting unit 408 includes:

the second detection module 4081 is configured to detect whether a jump signal of a photoelectric sensor in the fruit picking module is received within a preset time period, where the photoelectric sensor is disposed on an inner wall of the picking opening;

a switching module 4082, configured to determine that the picking of the target to be picked is successful if the second detection module 4081 detects that the picking is successful, and switch to the next target to be picked in the picking sequence list for picking;

The first discarding module 4083 is configured to determine that the picking of the targets to be picked fails if the second detecting module 4081 detects no, discard picking the targets to be picked when the number of times of picking failures of the same target to be picked reaches a designated number of times, and list the targets to be picked in a difficult-to-grasp list.

In an alternative embodiment, before the first adjusting unit 405 adjusts the size of the picking port of the fruit picking module according to the size information, the apparatus further includes a fourth determining unit 409, and the fourth determining unit 409 includes:

an eighth determining module 4091, configured to determine whether the size information of the object to be picked is within the size adjustment range of the picking port of the fruit picking module;

a second discarding module 4092 is configured to discard picking if the eighth determining module 4091 determines no.

In an alternative embodiment, after the picking unit 406 controls the fruit picking module to pick the object to be picked according to the picking position, the apparatus further includes a discharging unit 410, the discharging unit 410 includes:

a third detection module 4101 for detecting whether a fruit temporary storage box in the fruit picking module reaches a specified bearing threshold value through a pressure sensor, wherein the pressure sensor is arranged below the fruit temporary storage box;

And a discharging module 4102, if the third detecting module 4101 detects that the picking object reaches the picking object, the picking object is discharged into the fruit storage box in the fruit picking module by controlling the cover plate below the fruit temporary storage box to be opened.

Further, an embodiment of the present invention further provides a storage medium, where the storage medium is configured to store a computer program, where the computer program controls a device where the storage medium is located to execute the fruit picking method based on the mechanical arm described in fig. 2-3.

Further, an embodiment of the present invention further provides a processor, where the processor is configured to run a program, where the program runs to perform the mechanical arm based fruit picking method described in fig. 2-3.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

It will be appreciated that the relevant features of the methods and apparatus described above may be referenced to one another. In addition, the "first", "second", and the like in the above embodiments are for distinguishing the embodiments, and do not represent the merits and merits of the embodiments.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, the present application is not directed to any particular programming language. It will be appreciated that the teachings of the present application described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present application.

Furthermore, the memory may include volatile memory, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), in a computer readable medium, the memory including at least one memory chip.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. The fruit picking method based on the mechanical arm is characterized in that the mechanical arm is provided with two vision modules and a fruit picking module, and the method comprises the following steps:

controlling the mechanical arm to move the second vision module to a first position corresponding to the target to be picked based on the picking sequence list comprises the following steps: controlling the mechanical arm to move the second vision module to a first position corresponding to a target to be picked according to the picking sequence in the picking sequence list;

acquiring an image corresponding to a target to be picked and image depth information at a corresponding first position by using the second vision module, and judging whether the target to be picked is blocked or not based on the image corresponding to the target to be picked;

if yes, replacing the first positions corresponding to the targets to be picked until the second vision module judges that the targets to be picked are shielded at all the first positions corresponding to the targets to be picked, and controlling the mechanical arm to move the second vision module to the first position corresponding to the next target to be picked in the picking sequence list;

Determining the size information and picking position of the object to be picked according to the image acquired by the second vision module, including:

if the target to be picked is not shielded, analyzing a plurality of image depth information and corresponding size information of the target to be picked according to a plurality of images acquired by the second vision module;

determining target size information of the target to be picked from a plurality of pieces of size information;

determining picking positions of targets to be picked according to the image depth information corresponding to the target size information;

controlling the fruit picking module to pick the target to be picked according to the picking position, wherein the method further comprises:

detecting whether the target to be picked is picked within a specified time threshold;

if not, the size of the picking opening is adjusted according to the suction force of the negative pressure source in the fruit picking module, wherein if the suction force of the negative pressure source reaches a first suction threshold, the size of the picking opening is increased according to the size information of the target to be picked, if the suction force of the negative pressure source is smaller than a second suction threshold, the size of the picking opening is reduced according to the size information of the target to be picked, and the second suction threshold is smaller than the first suction threshold.

2. The method of claim 1, wherein determining a first location of at least one object to be picked from the image acquired by the first vision module comprises:

controlling the first vision module to acquire images of picked objects in a target field of view and image depth information of the picked objects, and analyzing maturity of the picked objects based on the images of the picked objects;

determining at least one target to be picked from the picked objects according to the maturity of the picked objects, and determining primary positioning information of the at least one target to be picked based on image depth information corresponding to the target to be picked;

and determining a first position corresponding to each target to be picked according to the primary positioning information.

3. The method according to claim 2, wherein determining the first position corresponding to each object to be picked according to the primary positioning information comprises:

4. The method of claim 1, wherein the determining the picking order list of the targets to be picked based on the first position and a current position of a robotic arm comprises:

determining a current position of the fruit picking module;

planning an optimal path of each object to be picked by the mechanical arm according to the first position corresponding to each object to be picked and the current position of the fruit picking module;

and determining the picking sequence of each target to be picked based on the optimal path to obtain a picking sequence list.

5. The method of claim 1, wherein said adjusting the pick mouth size of the fruit picking module based on the size information comprises:

determining and adjusting the size and the adjusting direction of a picking opening in the fruit picking module according to the size information;

and controlling a driving component in the fruit picking module according to the size and the adjusting direction to drive a reducing component in the fruit picking module to shrink or expand so as to adjust the size of the picking opening.

6. The method of claim 1, wherein when the fruit picking module is controlled to pick the object to be picked according to the picking position, the method further comprises:

7. The method of claim 1, wherein prior to said adjusting the size of the picking port of the fruit picking module according to the size information, the method further comprises:

if not, the picking is abandoned.

8. The method of claim 1, wherein after said controlling said fruit picking module to pick said target to be picked in accordance with said picking position, said method further comprises:

detecting whether a fruit temporary storage box in a fruit picking module reaches a specified bearing threshold value or not through a pressure sensor, wherein the pressure sensor is arranged below the fruit temporary storage box;

And if the picking target is reached, controlling the cover plate below the fruit temporary storage box to be opened, and unloading the picked picking target into the fruit storage box in the fruit picking module.

9. Fruit picking device based on arm, its characterized in that is provided with two vision modules and fruit picking module on the arm, the device includes:

the control unit is used for controlling the mechanical arm to move the second vision module to a first position corresponding to a target to be picked based on the picking sequence list, and comprises the following components: the second control module is used for controlling the mechanical arm to move the second vision module to a first position corresponding to a target to be picked according to the picking sequence in the picking sequence list;

the judging module is used for utilizing the second control module to move the second vision module to the corresponding first position to acquire an image corresponding to the target to be picked and image depth information, and judging whether the target to be picked is blocked or not based on the image corresponding to the target to be picked;

The replacing module is used for replacing the first position corresponding to the target to be picked if the judging module judges that the target to be picked is shielded until the second vision module judges that all the first positions corresponding to the target to be picked are shielded, and controlling the mechanical arm to move the second vision module to the first position corresponding to the next target to be picked in the picking sequence list;

the third determining unit is configured to determine size information and picking positions of the objects to be picked according to the image acquired by the second vision module, and includes: the analysis module is used for analyzing a plurality of image depth information and corresponding size information of the target to be picked according to the plurality of images acquired by the second vision module if the target to be picked is not shielded;

a fifth determining module, configured to determine target size information of the target to be picked from the plurality of size information analyzed by the analyzing module;

a sixth determining module, configured to determine a picking position of the object to be picked according to the image depth information corresponding to the object size information determined by the fifth determining module;

The picking unit is used for controlling the fruit picking module to pick the target to be picked according to the picking position, the device further comprises a second adjusting unit, and the second adjusting unit comprises:

the first detection module is used for detecting whether the target to be picked is picked within a specified time threshold;

the adjusting module is used for adjusting the size of the picking opening according to the suction force of the negative pressure source in the fruit picking module if the detecting module detects no, wherein the picking opening is increased by a specified size according to the size information of the target to be picked if the suction force of the negative pressure source reaches a first suction force threshold value, and the picking opening is reduced by the specified size according to the size information of the target to be picked if the suction force of the negative pressure source is smaller than a second suction force threshold value, and the second suction force threshold value is smaller than the first suction force threshold value.

10. A fruit picking system based on a mechanical arm, which is characterized by comprising a four-axis mechanical arm for picking fruits, two vision modules, a fruit picking module and a control module;

the first vision module is arranged on the upper surface of a first motion joint of the four-axis mechanical arm, the first vision module is driven to move up and down by the movement of the mechanical arm, and the second vision module and the fruit picking module are arranged on a third motion joint of the four-axis mechanical arm;

The first vision module is used for determining the maturity of a picked object in a target visual field, and determining at least one target to be picked in the target visual field and a first position where the second vision module moves based on the maturity of the picked object;

the second vision module is used for determining picking position and size information of a target to be picked;

the fruit picking module is used for adjusting the size of the picking opening according to the size information of the target to be picked so as to pick the target to be picked;

the control module is electrically connected with the four-axis robot arm, the two vision modules and the fruit picking module respectively, and executes the fruit picking method based on the robot arm according to any one of claims 1 to 8 so that the system can pick the target to be picked.

11. A storage medium comprising a stored program, wherein the program, when run, controls an apparatus in which the storage medium is located to perform the robot-based fruit picking method of any one of claims 1 to 8.

12. A processor for running a program, wherein the program is run to perform the robot-based fruit picking method of any one of claims 1 to 8.