JP2007200309A - Position specification method and apparatus for plant target part, and working robot using the same apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method of quickly specifying a position of a fruit to be harvested, or the like even under a changing light environment. <P>SOLUTION: The position specification method includes: a peculiar color selection step of selecting a characteristic color from a color image of a target part of all the colors as a peculiar color; a color component binarizing step of obtaining color space data from two color components except for color components relating to light and darkness in color data of the peculiar color; a color pattern registering step of displaying the color space data on a monitor screen, further selecting characteristic colors of a small number, and storing them as color pattern data; a working area color component binarizing step of photographing with a digital camera a circumference where the working target part exists in a real operation and obtaining the color space data of two color components except for color components relating to light and darkness from the photographed digital color image of all the colors; and a target part position displaying step of comparing the color space data with the stored color pattern data and displaying the position of only the part where both the data match each other in the image of the working area. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  In order to automatically perform various operations such as picking target parts such as fruits, vegetables, and flower berries, flowers, leaves, etc., the present invention determines the position of the target part from a wide-area image captured by a camera. The present invention relates to a method for recognizing, a device for recognizing the existence position of a target part by the method, and a working robot for performing various operations such as picking a plant fruit or a flower using the device. .

In order to automatically execute various operations such as picking and harvesting fruits and vegetables such as strawberries, tomatoes, eggplants, and cucumbers with robots and other devices, the location of the target fruits and vegetables is first recognized. However, it is almost impossible to mechanically recognize the fruit from the stems and leaves.
Therefore, since the existence of fruit can be recognized from the presence of light reflected from the fruit, the existence of the fruit, the existence position, the direction of the existence position, and the shape of the fruit can be recognized. A method for recognizing the existence of a fruit portion is conceivable.
Therefore, as a method of grasping fruits etc. from the position and shape of such fruits, a pattern matching method that pre-registers a shape pattern specific to the fruit to be picked and tries to select one that matches this pattern The possibility of
However, in the pattern matching method by shape, the actual parts of crops represented by strawberries etc. have various shapes such as size and external shape, and the actual pattern of shapes suitable for picking up all shapes is assumed. It is extremely difficult to register in advance.
For this reason, the pattern matching method by the shape in the plant field cannot be adopted.

In addition, the time for picking fruit is important, as in the case of shipping as a product, but image processing technology using a computer that only recognizes the “shape” cannot distinguish the degree of maturity. If the color is not identified, it cannot be determined whether it is suitable for picking.
Therefore, automatic operation by a machine such as picking cannot be performed with an apparatus based only on shape recognition.

Therefore, it is possible to confirm the presence of fruits and the like by sensing them with light, but also pay attention to the fact that “color” can be recognized from the wavelength of the light reflected from the fruits and so By capturing the wavelength of light and looking at the captured image, it is possible to grasp the existence of the fruit and its color and shape.
Therefore, when trying to recognize the color of fruit as an element to be identified, register the color of the fruit suitable for picking in advance, and the color captured from the registered color and the fruit in the field work When the two match, it is possible to identify and judge that the fruit having the color is suitable for picking.
This “color” can be recognized using the three elements of lightness, hue, and color, but when trying to identify from these three elements of color, the color data to be identified is considered with three parameters. There must be.
In the current image processing technology using a computer, it is extremely complicated in terms of data processing to identify and judge with three parameters for color.
In addition, it is difficult to accurately capture the color information of agricultural products that contain subtle colors including intermediate colors from green to white to red, such as fruits such as strawberries, It is not easy to quantify and handle this as color data that is the basis of judgment materials.

For example, in Patent Document 1 which is a conventional technique dealing with color, “in a plucking hand attached to the tip of a wrist of a fruit vegetable harvesting robot system, a pair of upper and lower surfaces of a pair of fingers that are driven to open and close are paired. A plucking blade and a preparation blade were provided, respectively, and an air ejection hole for ejecting compressed air for removing the handle attached when the fingers were opened from the inner surface of the finger was formed on the opposing surfaces of the pair of fingers. A featured fruit vegetable harvesting hand ”has been proposed, and in this document 1,“ color image of L, a, b, and color system a is calculated by processing a color original image taken with a camera. In addition, there is a description of "a method for obtaining a coordinate for the position of a fruit by calculating the center of gravity of a strawberry fruit from the figure by binarizing the grayscale image with an optimum threshold value to obtain a figure for extracting the position of the fruit".
JP 2001-95348 A

However, since the technique of “color” proposed in Patent Document 1 tries to grasp the center of gravity of strawberry fruits using the three elements of color, all three elements of color composed of lightness, hue, and color are used. Color data is required, and information on brightness, in particular, is greatly affected by changes in the usage environment, so the amount of data to be used becomes extremely large even with information processing related to brightness. Because the equipment for this becomes very complicated, the equipment is not only very expensive, but it takes a long time to process the data. It takes a lot of time to make a decision, and the work becomes too slow, so actual operation at the work site is extremely difficult.
Even if it is used in a special light environment such as indoor cultivation with minimal changes in lightness, color data related to the lightness is necessary, so it will not take long to process the data. It is difficult to do.

On the other hand, picking fruits with stems such as strawberries and tomatoes can easily damage the fruits if they are mechanically gripped and peeled off with a robot hand. There is a research report that cutting the stem directly and harvesting the fruit does not reduce the quality of the product.
However, recognizing the stem by image processing with a computer is much more difficult than accurately grasping the actual part because most stems are thin and small.

In such a situation, the present inventor conducted various studies on the automation of various operations using an apparatus for picking fruits and the like. As a result, in the field of plants, unlike the industrial field of machinery and the like, fruits that are products Judgment of maturity must be done not only by shape but by color, and when dealing with that color, lightness, which is one of the three elements of color, changes greatly due to the influence of natural environmental conditions around the plant. I noticed that there are important issues to do.
And at work sites that handle plants, time efficiency is required, and in order to work quickly, it is indispensable to speed up mechanical processing as well as speeding up data processing by computers, and the computational speed of computers is limited. If it is determined that the amount of data to be processed will be decisive, it will be decisive to distinguish fruits and the like by color. The idea was that if the number was reduced, the total amount of data could be easily reduced.
Therefore, it is only necessary to know the location of the fruit, etc., so the amount of data related to the “shape” of the target fruit or vegetable must be minimized, and the light environment at the work site can vary greatly. In view of this, the possibility of omitting color elements related to lightness, which increases the amount of data, was considered.

  In view of this, the present invention distinguishes and harvests mature fruits from the color image including the surroundings of the fruits and the like that appear on the monitor screen photographed by the camera even in the natural environment where the light environment changes. Provided is a method for accurately recognizing the presence and position of a target fruit and the like and facilitating the location of a work target part such as a fruit of a plant, and a device for performing the method. It is an object of the present invention to provide a working robot that can automatically perform various operations such as a work of picking a specified fruit.

In order to achieve the above object, in the invention of claim 1 of the method for specifying the position of the target part of the plant of the present invention, an input device for data and commands including a digital camera, an output device including a display device, A computer system including a CPU and a computer main body having a memory and a computer program for data processing incorporated in the computer main body are used.
Then, one or a plurality of parts of one or many kinds of plants are targeted, and the digitized color image data of the target part is input and stored in the computer main body, which is stored in the display device. A unique color selection step of reproducing on a monitor screen and selecting a characteristic color of the target portion as a unique color in the reproduced image, and color elements related to light and dark from the color data of all the selected unique colors A color element binarization process for obtaining color space data that can be displayed in color on a biaxial coordinate plane by converting to color data of two excluded color elements, and a color space obtained by the color element binarization process A color pattern registration step of displaying data on the monitor screen, further selecting one or a few characteristic colors from the color image, and storing them in a computer as color pattern data for identification Perform a pre-preparation made from.
Then, in the actual work, using the computer system provided with the computer program, the digital camera captures an area including the periphery where the target portion to be operated exists, and the captured digital color image of all colors , A work area color element binarization process for obtaining color space data that can be displayed in color on a biaxial coordinate plane by converting to color data of two color elements excluding color elements relating to light and dark, and the work area color element The color space data of the work area obtained in the two-dimensional processing step is compared with the color pattern data stored in the advance preparation, and both are displayed in the image of the work area displayed on the monitor screen. And a target part position display step of displaying only the matched parts while distinguishing the positions.
The present invention is characterized in that the position of the target portion can be accurately displayed from the area including the periphery in the monitor screen regardless of the brightness of the work environment.

  According to the invention of claim 2, in the method for specifying the position of the target part of the plant according to claim 1, the target part whose presence position is displayed in the image of the monitor screen is displayed after the target part position display step. On the other hand, the enclosure range in the screen is determined according to the size of the target portion in the image to be recognized by the computer, and the position of the target portion displayed in the monitor screen image in the target portion position display step The present invention is characterized in that there is an object partial marking step of automatically identifying the position of data existing in the enclosed range and marking the data of the identified position.

  In the invention of claim 3, in the target part marking step of claim 2, a program in which priority conditions are set by a computer for each of a plurality of target parts displayed in the image of the monitor screen Therefore, it is characterized by having a marking priority order determining step for automatically displaying markings in the priority order.

  In the invention of claim 4, in the color pattern registration step of claim 1, the computer stores a plurality of pieces of color pattern data of the target part from different locations for one target part. The color pattern data of the plurality of target portions is labeled, and each labeled color pattern data is set so that it can be simultaneously displayed in the image of the monitor screen in actual work, and in the target portion position display step, the monitor screen It has a composite selection step of extracting a part where the enclosed ranges of different parts of the target part of the plant whose existence position is displayed in the image partially overlap, and automatically determining that the target part exists only in the extracted part It is characterized by that.

In the invention of claim 5, the color data of all the color systems described in claim 1 can be converted into color data of two color elements excluding color elements relating to light and dark, and color display can be performed on a biaxial coordinate plane. A method for obtaining biaxial coordinates (x) and (y), which are simple color space data,
From the xy chromaticity diagram calculated from the CIE-XYZ color system,

の数式（ａ）、（ｂ）にＲ、Ｇ、Ｂの輝度値を入力して色度座標（ｘ）、（ｙ）値を求めることを特徴とする。

The luminance values of R, G, and B are input to the mathematical formulas (a) and (b), and the chromaticity coordinates (x) and (y) values are obtained.

  The invention according to claim 6 is the device for specifying the position of the target portion of the plant by the method for specifying the position of the target portion of the plant according to any one of the first to fifth aspects.

  According to a seventh aspect of the present invention, there is provided the plant target part position specifying device according to the plant target part position specifying method according to any one of the first to fifth aspects, the CPU 1 and the memory 2. And a computer system C1 having a digital image input device 3 for color images and a display device 4 for a monitor for viewing an image, wherein the computer system is claimed by the computer system. A program P1 is provided, which executes processing of data of color and position of a digital color image in accordance with a processing step in the method for specifying a position of a target portion of a plant according to any one of 1 to 5.

  According to an eighth aspect of the present invention, there is provided a working robot using the plant target part position specifying device according to the sixth or seventh aspect.

In invention of Claim 9, it is a working robot using the position specifying apparatus of the target part of the plant of Claim 6 or 7, Comprising: For apparatus mounting provided with the drive means 6 for traveling movement A digital color image input device 7 for plant photography, a work hand 8 for performing the work, a manipulator 9 for driving the work hand, and a sensor 11 for checking the body position. The body 5, the manipulator 9, the work hand 8, the digital color image input device 7, and the body position confirmation sensor 11 are each provided with a drive control unit that executes operations according to computer commands.
Then, on the body 5 or at another place, the position of the target part of the plant is specified from the driving program P2 for instructing the operation of each part constituting the apparatus and the digital image inputted by the digital color image input device 7 A computer main body C2 having a data processing program P1 for performing the method for specifying the position of the target part of the plant according to any one of claims 1 to 5 is provided.
And the command signal and the data processing signal are transmitted and received between the drive control unit and the computer C2, and the manipulator 9 is operated while the position of the target part of the plant is specified by the position specifying device. It is characterized in that the hand 8 is automatically moved to the position of the specified target part, and the work on the target part of the plant can be executed by the work hand 8.

Since the present invention is configured as described above, the following effects can be obtained.
In the present invention, color space data that can be displayed in color on a biaxial coordinate plane by two color elements obtained by removing color elements related to light and dark from color features such as mature fruits to be harvested in the color element binarization processing step. Further, from this data, a more characteristic color is selected in the color pattern registration step, and the color pattern data of the target portion, which is the basis for identifying the fruit to be harvested, is registered.
In the actual work, the work area obtained by removing the color elements related to light and dark in the work area color element binarization processing step from the color pattern data of the target part registered in advance and the color data photographed with the actual work. By collating with the color space data, the existence of the target part such as mature fruit to be harvested from the color matching part in the photographed monitor screen can be automatically confirmed.

The color space data and the color pattern data used for color matching are data from which color elements related to light and darkness such as lightness are excluded, so light due to changes in the natural environment such as day and night and weather, and changes in lighting conditions, etc. Even if there is a change in the amount of light (darkness), the change is not reflected in the color data. As a result, stable colors that are hardly affected even in the light environment where the lightness and darkness changes rapidly such as outdoors Data can be obtained.
In other words, even in working places where plants grow where the brightness changes from time to time depending on the weather on bright sunny days, dark rainy days, etc. Even when there are times when it is dark behind the scenes, even if the brightness of the color of the fruit changes due to the influence of the surrounding light, the target part such as the fruit is left behind. Can be distinguished from the surroundings, and its position can be specified reliably.

In the data processing by the computer at that time, this color space data is only the color data of the two color elements excluding the color elements related to light and dark, among the three elements of color composed of brightness, hue and saturation. Compared with data of all colors, the amount of information is extremely small, so that the data processing does not consume the storage capacity in the memory, and the time required for arithmetic processing by the CPU can be reduced in a very short time. become.
Then, from the color space data of the two color elements, the process of extracting the “color pattern” of the target portion with a smaller amount of data, the data to be collated is reduced as much as possible, and the time required for the data processing is reduced. In addition, it has become possible to reduce the size of the target part more accurately from the characteristic parts such as fruits.

  In addition, the position confirmation of the presence of the target part by the color of the target part is much smaller than the conventional confirmation of the presence based on the shape, and particularly in plants, leaves, fruits, stems, roots, etc. Since the specific color of each part of the plant is determined by the species, there is a unique characteristic in each part of the fruit, such as leaves, fruits, stems, etc. The confirmation of existence by color has the advantage that misidentification is less likely to occur.

As a result of actually confirming the position identification device on the outdoor strawberry cultivation farm, the color is a color that can be displayed in color on the biaxial coordinate plane by two color elements excluding the color elements related to light and dark as described above. Since data is processed based on spatial data and the data is collated by a computer and a program to make a judgment, stable identification is possible based on the color characteristics of the target part. An amazing result that the actual misperception rate is "0" was obtained.
Conventionally, even if an identification device that is activated by recognition only by shape or recognition by both shape and color is used, almost half of the recognition errors occur, and such accuracy cannot be used for actual work. However, the present invention has opened the way for practical application to various work related to plants such as harvesting work.

  Furthermore, conventional computers and programs required for shape recognition are not so complicated, and there are almost no failures in actual operation, so handling is very easy and equipment to be used can be provided at low cost. It becomes.

According to the second aspect of the present invention, the target portion existing in the enclosed range from the position data of the target portion is not recognized by the person viewing the image by the target portion marking step performed after the target portion position display step. Can be automatically identified.
As a result, a fully automatic work robot for a plant that performs work without human attendance is faster and more efficient than human work by implementing the position identification device for the target portion of the plant according to the method of the present invention. The performance can be improved to the extent that the work can be performed.

  Further, according to the invention of claim 3, in the operation, the object part marking process includes a marking priority order determining step, so that a computer for each of the plurality of object parts displayed in the image of the monitor screen is obtained. Can be automatically marked in order of priority according to the program for which priority conditions have been set. When there are many target parts, this marking priority is displayed when a large number of positions are displayed in the image on the monitor screen. It became possible to perform efficient work in the order according to the order without the work robot getting lost.

Further, in the invention of claim 4, the color pattern registration step includes a composite selection step of labeling and registering a plurality of one target portion, so that one target portion is called a stem and a fruit. If there is a plurality of features, even if there is a target part of the plant in the image of the monitor screen, it is automatically determined to be a target part only when a part of the plurality of enclosed ranges overlaps, otherwise Can be unmarked.
As a result, for example, it is possible to associate an image of a real part and an image of a stem part, and it can be determined that the existence of both is a fruit to be harvested when a part of the overlapping position is shared. For example, even if an error occurs in the identification of the real part, the probability of the identification error is greatly reduced by taking the identification of the other stem part into account. It became possible to identify the fruits with stems.
In addition, by accurately grasping the fruit and stem parts separately, for example, in the case of a strawberry to be harvested by cutting the stem, the existence and position of the stem can be grasped quickly and accurately, It is possible to avoid the parts from being scratched and accurately pick the stems, reduce the possible scratches during harvesting, and obtain a fruit with high commercial value that does not cause quality degradation due to scratches. Became.

According to the invention of claim 5, the color data of all colors is converted into color data of two color elements excluding the color elements related to light and dark from the xy chromaticity diagram calculated from CIE-XYZ excluding the lightness. Thus, the two-axis coordinates x and y, which are color space data that can be displayed in color on the two-axis coordinate plane, can be obtained by the calculation using the equation (1).
The calculation of this method allows the computer to easily process the color data in accordance with the coordinate formula program, and the color pattern of the target portion can be easily obtained.
Furthermore, color matching based on the color pattern data of the target part and identification of the target part can be performed quickly, and as a result, work can be performed on individual target parts such as strawberries with a very large number of individuals. However, it has become possible to produce a high-performance work robot that can be implemented quickly and repeatedly.

  Further, the invention of claim 6 and 7 is a device for specifying the target part of the plant by the method for specifying the target part of the plant according to any one of claims 1 to 5, wherein the device of the invention of claim 7 A computer program for executing the method for specifying the position of the target portion of the plant is incorporated in a computer system including a digital input device such as a digital camera for capturing digital color image data having color and position data. Therefore, the position of the target part can be specified by the method for specifying the position of the target part of the plant according to claims 1 to 5.

And in invention of the working robot of Claims 8 and 9, since the position specifying device for the target part of the plant of Claim 6 and Claim 7 is mounted, while capturing an image with the provided digital camera, The manipulator is automatically controlled by the drive program built in the computer while the position of the target part is specified, and the position of the work hand is moved to perform various operations by the work hand quickly while moving the body. It became possible.
At that time, the work hand is mounted with a shape, structure and function according to work such as fruit picking, flower pollination, flower picking, etc., or by replacing the device with a structure suitable for the work, These various operations can be performed.

The present invention is an invention of a position specifying method of a target portion of a plant, a position specifying device of the target portion by the specifying method, and a working robot using the position specifying device, and embodiments thereof will be described in order below. .
First, a method for specifying the position of the target portion of the plant will be described.

In the method for specifying the position of the target portion of the plant of the present invention, the specific work requires a device for specifying the position of the target portion and a program for processing data obtained by operating the device. It is.
As shown in FIGS. 8A and 8B, this apparatus is a keyboard for inputting data and commands including a color image input device 3 such as a CCD camera to a computer body having a CPU 1 and a memory 2. 10. A computer system C1 for position specification having an output device for displaying an image including an input device such as a mouse, a touch panel and a light pen and a display device 4 for monitoring is used. A computer program P1 for data processing is incorporated.
The computer program P1 is a computer program for processing input, storage, processing, determination, and storage of data such as position information with a focus on image processing functions.

In the present invention, the target plant is one or many kinds of plants that can be a work target such as strawberries and tomatoes, and the desired target part of the plant is one part that is a target for work such as fruits and leaves, or Multiple parts such as fruit and stem.
Such target parts, for example in the case of tomatoes, are harvested with red ripe fruit parts, but the work depends on the stage of growth, from seedlings to sampling and disposal of seedlings. There are various kinds of work such as pollination and thinning, and all of the parts to be worked are desired parts.

Next, the method of the present invention will be described by taking as an example a case where the target plant is a tomato and the target portion is a red tomato fruit portion to be harvested.
In the present invention, preparatory work is required before actual work on site.
As this preparatory work, as shown in FIG. 1, a unique color selection process, a color element binarization process, and a color pattern registration process are performed.
In the unique color selection step, since the desired target portion is tomato fruit, the actual color photograph, color video photograph, and actual object are converted into digitized color image data once digitized. Work to fix.
This digital image is used to acquire digital information of color, not the color itself of the color image. Therefore, if the portion to be selected as the target portion is a ripe fruit, it can be quickly done by photographing the actual fruit that has been ripe with a digital camera.
However, if it is stored for a while after harvesting, and matured by storage and shipped, the tomatoes that are fully ripe and still have a little light green part are harvested early, and there are immature light green parts The actual object is photographed with a digital camera as the target portion.
Then, the digitized color image data is reproduced on the monitor screen, and if it is a ripe tomato in the reproduced image, the red part of the tomato, which is a characteristic color of that part, is used as the intrinsic color. Make a selection.

Then, in the next color element binary processing step, based on the color data of the digitized image of the red color selected above, that is, the color data of the selected unique color of the tomato. In addition, color space data that can be converted into color data of two color elements excluding color elements related to light and dark by a computer program and can be displayed in color on a biaxial coordinate plane (FIG. 4 is an example of a strawberry, ) Is displayed in the same way as).
This color space data does not display the shape of a strawberry or the like, but is shown in the shape of a biaxial coordinate plane based on coordinates with two color elements as vertical and horizontal axes, as shown in FIG. It is data.

Here, a general method of conventional color technical processing will be described supplementarily.
All colors exist in nature, and CIE (International Commission on Illumination) -XYZ, L.a.b, L.u.v, Munsell color system as a method for expressing the colors of all the colors. Various color systems such as the Ostwald color system and the NCS color system have been proposed, and these colors are often used for coloration of industrial products, color adjustment of image equipment, and the like.
Particularly in recent years, color systems such as L · a · b and L · u · v are often used for visual recognition.
What is common to all the color systems is that colors must be determined by all three parameters.
Normally, the existence of an object can be grasped by three-dimensional coordinates with three parameters of the same quality called “distance” of only position information like a physical space, but the existence of an object can be grasped by color. This is difficult to grasp unless three-dimensional color replacement is performed with three different parameters such as hue, saturation, and brightness.
Therefore, specialized knowledge is required to grasp the presence of an object from the color, and data processing by a computer is extremely troublesome.

  The three color elements of hue, saturation, and lightness, which are the parameters of the color, are digitized to express a specific color. Based on these three color elements, tristimulus is used in color data processing by a computer. As a value (each value of CIE-XYZ is calculated by distribution of numerical values of R (red), G (green), and B (blue) which are the three primary colors of light, which color is determined.

Therefore, for example, when looking at "brightness and darkness" which is one of the color parameters, in the case of the plant to be handled in the present invention, it grows and matures by photosynthesis, so it exists closely with the light environment, and the location is As described above, it usually exists in a place where there is always a change in the amount of light (brightness and darkness) due to changes in the natural environment such as day and night and weather, and fluctuations in lighting conditions. When handling large elements as color data, the data processing requires enormous capacity and time.
Therefore, if an attempt is made to confirm the location of an object with a color that includes light and dark elements, a delay in data processing is unavoidable, and as a result, it should be put to practical use at work sites that require efficiency such as speed. Is extremely difficult.

The present inventor who faced the above situation has that the plant to be handled in the present invention has a color inherent to each part such as a leaf stem, and the color does not change depending on the brightness of light. For objects that are noticed and have color characteristics, we infer that it is possible to judge colors that are stable in nature by positively removing color elements related to light and dark from the data used as judgment materials. If so, we came to the conclusion that the problem that had caused a large delay in data processing by using light and dark color elements as parameters until now could be solved at once by reducing the amount of data drastically. .
Various methods are tried to realize this conclusion, and it is possible to perform color data processing that converts color element data of all colors into the remaining two color element data excluding the light and dark color elements. I understood.
In the present invention, since the method described in claim 5 is the optimum method for the data processing, the method will be described below.

The method described in claim 5 is color space data that can be displayed in color on a biaxial coordinate plane by converting color data of all colors into color data of two color elements excluding color elements relating to light and dark. This is a method for obtaining the coordinates of an axis.
In this method, the above-described equation (1) is used from the xy chromaticity diagram calculated from the CIE-XYZ color system.
In equations (a) and (b), the luminance values of R, G, and B are input to obtain chromaticity coordinate values x and y.
As described above, when the handling of the color data is simplified, the final determination of the position of the object on the computer can be performed instantaneously.

In this method, a computer program is automatically processed by a dedicated computer program, and two color elements can be converted into biaxial coordinates. Further, only the colors that are based on the color area distributed on the coordinates are extracted. Is done.
For this purpose, first, an xy chromaticity diagram represented by biaxial coordinates calculated from the CIE-XYZ color system of the international color standard shown in FIG. 6A is used as the biaxial coordinates.
In the xy chromaticity diagram shown in FIG. 6A, colors such as red, blue, green, yellow, orange, and purple can be represented by coordinates of the x-axis and y-axis. It can be grasped numerically.
From this xy chromaticity diagram, the chromaticity coordinate x value and the y value are obtained.

In the present invention, R, G, and B are converted into the above formula (a) and formula (b) from the image data of the colors of all the images (shown in FIG. 4A) of the monitor screen including the target portion. Are input, and the color coordinate x value and y value are calculated from each value in Equation (1).
In Equation 1, R is red, G is green, and B is blue. This indicates the three primary colors of light, and the respective luminance values are input to obtain the x and y values of the color coordinates. is there.
Then, the x value and the y value are obtained from the RGB values as one point for all the images on the monitor screen including the target portion and all the portions on the image. At that time, the color data of the portion is on the section designated by continuous numerical values (0 to 1) so that it can be expressed by plane coordinates divided into 30 in each of the x-axis direction and the y-axis direction. The data is converted to the coordinates divided by the specified number of.
Thus, the mosaic block-like color figures shown in FIGS. 4 (b), 5 (a) and 6 (b), that is, the above chromaticity and brightness excluding color elements relating to light and dark. It is possible to display on the monitor screen a color figure of a block-like color that is a biaxial coordinate plane in which the color space data of the two color elements is expressed.
A process performed to obtain color data (color space data) that can be expressed in the space of the graphic is the color element binarization process.

  The color space data can be represented on the monitor screen as an icon when visually grasped, or mathematically represented by a matrix. In addition, when the number of pixels to be expressed is input to each of the sections, a histogram of color information is obtained with the xy chromaticity diagram divided by the number designated by the two axes of the x axis and the y axis as a plane. .

The xy chromaticity diagram calculated from the CIE-XYZ color system is such that, instead of RGB, when color (F) is equalized with a mixed color of the three primary colors RGB and F = RR + GG + BB, r = R / (R + G + B), g = G / (R + G + B), and b = B / (R + G + B), r + g + b = 1 holds, and two of them are expressed.
In addition, in the application of the above formula (a), it is also possible to change the coefficient ratio or adjust the number of significant digits.
As a calculation method for obtaining the x value and the y value of such color space data, the coefficient of the conversion formula (a) is 4 digits after the decimal point. If the coefficient is shifted by 4 digits and regarded as an integer, the coefficient is 32 bits long. Since this is a numerical value that can be expressed by data, and the value does not become negative in the course of the calculation, this program can express and calculate the coefficient with integer values shifted by 4 digits without applying floating point arithmetic.
In the present invention, the conversion to the color space data can be easily speeded up by performing the data processing as described above.

  The color space data obtained in the color element binary processing step is characteristic color data of the target portion to be processed. In the present invention, the color space data is further subjected to secondary processing, and the next step. The comparison and narrowing can be performed with the color pattern data (1) shown in FIG. 6 (c).

Therefore, in the next color pattern registration step, the color space data obtained in the color element binary processing step is displayed on the monitor screen, and a more characteristic one is displayed from the color image (shown in FIG. 4B). One or a small number of colors are selected and registered as color data of the color pattern of the target portion (shown in FIG. 4D).
At this time, the color space data can be handled mathematically as a matrix as shown in FIG.
That is, as a method of expressing a color pattern in the color space, the color can be expressed by a row example in the color space data shown in (b) of FIG. 4 and (a) of FIG. In (b) of FIG. 5, color space data is expressed by a matrix M divided by 30 × 30, and each element of a ₁ , _{1 to} a ₃₀ , ₃₀ has a color arranged there. Is entered.
The 30 × 30 division is an experimental example, and the number of divisions can be appropriately set according to the target and the work site.
4 (D) and 5 (C) is a “color pattern” in which data is imaged and displayed on the monitor screen, and this is also a matrix similar to the color space data. Can be expressed as
FIG. 5 (2) expresses the color pattern (c) of FIG.
By registering the color pattern data of the target part in advance, it is possible to compare and match with the color space data that does not include the data regarding the lightness and darkness of the color of the target part obtained at the next actual work site. It becomes possible to narrow down and specify the location.

  In this way, according to the method of the invention of claim 5, by the method of easily obtaining the coordinates on the two axes composed of two color elements, the mathematical expression of the “Expression 1” is obtained from the xy chromaticity diagram calculated from CIE-XYZ excluding the lightness. Based on the xy chromaticity diagram, it is possible to easily process the color data of a target part of a plant such as a target fruit by a computer according to a program of coordinate formulas regardless of the brightness of the work environment, By quickly identifying the target portion, it is possible to put to practical use an apparatus that performs a work that is repeatedly performed on individual target portions such as tomatoes and strawberries that have an extremely large number of individuals.

As described above, color space data is obtained, and if an xy chromaticity diagram is represented by a block-shaped image or matrix in a designated section, then, from the image or matrix, the color of the real part of the tomato A certain portion is selected from the features, and color pattern data is obtained from the selected portion.
That is, the color pattern data is selected from the color space data, which further characterizes the tomato fruit.
Therefore, since tomato has a characteristic of red color, it is converted to dark red color pattern data that develops color during tomato maturity from its hue and saturation.
In other words, the color pattern is a graphic representation of the color representing the color pattern data in the image, and the block-shaped portion characterizing the target tomato is removed from the block-shaped color graphic in the graphic representation of the color space data. A combination shape formed by an assembly of the selected block-like portions selected.

Although the color space data and the color pattern data of the target portion have been described in detail above, when returning to the next description in the case of the tomato again, the tomato displays the color space data obtained in the color element binary processing step on the monitor screen. Then, one more characteristic dark red portion is selected from the color image, and this dark red is stored in the computer as color pattern data of the target portion of tomato and registered.
This completes the preparatory work process before entering the actual field work by registering the color pattern data.

After performing the above-described preparation work, the actual work of specifying the position of the desired tomato at the cultivation site is performed.
In this actual work, a work area color element binary processing process and a target part position display process are performed.
In the work area color element binary processing step, in an actual work, an area including the periphery of the tomato seedling to be worked is photographed with a digital camera as a digital color image of all colors, and using a computer system, The color data of all color systems is converted into color data of two color elements excluding color elements related to light and dark by a computer program to obtain color space data that can be displayed in color on a biaxial coordinate plane.
This color space data is data for all colors in the work area including tomatoes.
Then, in the next target portion position display step, dark red that is the color space data of the site wide area including tomato seedlings obtained in the work area color element two-dimensionalization processing step and the color pattern data of the registered tomatoes In the image of the monitor screen only in a portion where the color data of both the color space data of the work area and the color pattern data of tomato that can be displayed in color on the biaxial coordinate plane match Display the location.
That is, in the above steps, the portion where the color data matches is the position where the tomato exists, and that portion is displayed in the image of the monitor screen, and the presence position of the tomato is confirmed.

The relative position of the tomato in the surrounding area can be confirmed by displaying an image displaying the extracted portion on the original image leaving the digital color before processing as it is.
Moreover, the image by which the position was displayed can be obtained by making colors other than tomato into another color.
Furthermore, the brightness value of the selected portion of the tomato color pattern data is changed in the tomato color data obtained in the unique color selection process, which is color space data that can be displayed in color on the biaxial coordinate plane. By programming, only the part where the color space data and the color pattern data of tomato match in the entire color display image is brightened and the other parts are expressed darkly, so the location is more clearly surrounding. It is also possible to make it possible to clearly distinguish from the inside.

In addition, when displaying the existence position in the image of the monitor screen of only the portion of the tomato that matches the color pattern data in the color space data, on the display image positioned in the color pattern data of the tomato registered in advance Are left as they are (if any), the pixel portion on the display image that is not located in the color pattern is erased (other pixels are black, R = 0, G = 0, B = 0). You can also.
Then, in the image displayed on the monitor screen, only the pixel portion having the dark red color specified by the color pattern data of the tomato remains, and the number of the cluster of the non-black pixel portions is collected. Labeling can be performed, and only one desired tomato fruit can be selected from the masses indicating the presence of a plurality of tomatoes in the image on the monitor screen.

  In the present invention, as described above, data in which color elements related to light and darkness such as lightness are excluded from tomato color data is used. Therefore, the amount of light (lightness and darkness) can be reduced due to changes in the natural environment such as weather and changes in lighting conditions. Even if there is a change, the change is not reflected in the color data, and as a result, characteristic color data of tomatoes that are not affected by the light environment that changes drastically at the work place can be stably obtained, and cultivation with large changes in light and darkness Even in the place, it is possible to accurately specify the position of the desired ripe tomato from the area including the periphery.

Next, the invention of claim 2 will be described.
This invention is a target part marking process for marking a target part performed after the target part position display process.
Up to the target partial position display step, a specific tomato can be displayed in the image on the monitor screen, and a person who sees the image can recognize the location of the tomato, but is not in a stage until the computer makes a decision.
For this reason, in this step, the computer can automatically identify the tomatoes in the target portion existing in the enclosed range from the position data of the target portion.
This makes it possible to mount a plant target part position specifying device or a working robot for a plant using the position specifying device.

According to a third aspect of the present invention, a priority condition is set by a computer for each of the plurality of target portions displayed in the monitor screen image by incorporating a marking priority order determination step into the target portion marking step. It becomes possible to automatically mark in priority order according to the program.
When there are many target portions, when a large number of positions are displayed in the image on the monitor screen, the work robot performs efficient work in the order according to the marking priority order without being lost. It becomes possible.
As a method of assigning priorities, various orders such as right order and top order can be set.

Further, in the invention of claim 4, the color pattern registration step includes a composite selection step of registering a plurality of color pattern data for one target portion and registering it, so that one target portion is a stem and When there are multiple features such as fruits, even if there is a plant target part in the image on the monitor screen, it is automatically determined to be the target part only when part of the multiple enclosures overlaps. Other than that, it becomes possible to eliminate.
In this case, multiple labeled color pattern data needs to be parallelized on each of the color pattern data and multiplexed on the monitor image.To achieve this, the color pattern data is stored in a shared memory, Independent image processing for each color pattern data can be executed on the shared memory.

Further, in obtaining the color pattern data, it is possible to obtain a plurality of color pattern data, label them and register them.In this case, it is possible to collate with a plurality of registered color pattern data. The display of the chunks showing the target portion having the color labeled on the monitor screen may spread over a wide range.
Then, as it is, it is not possible to narrow down a region where tomatoes, strawberry fruits, etc. exist, and further, from there, it is not possible to extract a pair of fruits and stems for one target portion.
Therefore, in such a case, the color pattern data is related separately and parallelized to be recognized separately in a plurality of enclosed ranges, and only when the recognized plurality of enclosed ranges partially overlap, the target portion is automatically Therefore, it is necessary to make it possible to process each labeled data in parallel.
In this way, not only the combination of the real and stem, but also when you want to specify from multiple different types of target parts such as real, stem, stem etc. by parallel processing and more complicated association, you want to specify from the target parts of very close colors Accurate identification is possible.

Therefore, when the image of the actual part and the image of the stem part are associated with each other, and an error that the target part is identified in one of the identifications, it is determined whether the target part is accurately recognized by the other recognition. In addition, for example, even if an error occurs in the identification of the real part, the identification of the other stem part greatly reduces the probability of the error and enables accurate identification.
And by accurately grasping the fruit and stem part, for example, in the case of tomatoes or strawberries that are to be harvested by cutting the stem, the existence and position of the stem are grasped quickly and accurately. By picking and picking, the fruit can be harvested while maintaining a high-quality state without touching the fruit and without damaging the fruit with a machine.

  In addition to the input image, the processing process image and the processing result image can be stored on the shared memory, and the shared memory can share data with a plurality of programs. It is also possible to display an input image, an image of a processing process, an image of a processing result, and to perform collaborative work by transmitting and receiving data and commands.

In the present invention, if there is an actual xy chromaticity diagram displayed in color as shown in FIG. 5A and a diagram in which the color names divided on the color distribution are shown, the actual xy chromaticity diagram is displayed. With reference to the color specified in the xy chromaticity diagram, even if there is no camera color image, color pattern data can be obtained from that color, and the data can be registered.
In addition, for example, if the tomato fruit is the target part, it is possible to capture and store the color pattern data by photographing the section located in red with a digital camera, and if it is to be accurate, the actual use result It is also possible to make fine color adjustments while viewing the screen.
As described above, the color space data and color pattern data of the target part such as the tomato fruit have been explained, but the data processing can also be performed in the same process for the fruit, leaf, and flower parts of other plants. Is possible.

Next, another method is possible as to how to specify the existence position by collation between the color space data and the registered color pattern data, which will be described below.
When the color space data formed on the basis of the digital color image input on site matches the registered color pattern data, the same color pattern data as this color space data can be stored and registered in the computer. .
In this case, the color space data and the color pattern data completely match.
That is, the color pattern data as shown in FIG. 6C cannot be created from the color space data shown in FIG. 6B, and FIG. 6B is directly registered as the color pattern data.
Even in this case, the position can be specified by narrowing down the target portion in the same process as the above process.
Even in this case, it is possible to register a plurality of color pattern data, which makes it possible to identify a plurality of types of target portions and to label a plurality of data and process them in parallel.
As described above, on the monitor screen, the displayed image using the color pattern data in advance is displayed superimposed on the digital color image of all colors input at the work site, and the target portion captured by the color pattern data is displayed. An enclosed area of an image can be specified and labeled, but there is also a method of making a frame around the enclosed area and displaying the frame part on the screen to make it easier to grasp the specified target part.

The present Example 1 is a case where the fruit of a strawberry is made into object, and it demonstrates below based on FIG. 2 which shows the process about the case where the fruit of a mature strawberry is specified.
The strawberry has a fruit part, a leaf part, and a root part, and is actually attached to the tip of the stem (stem). The method of the present invention will be explained as to how to identify the location of the ripe fruit part. .
The process of this method includes a process until registering the color pattern in the previous stage, and a process of actually searching for the fruit of the strawberry, and will be described from the previous stage.
First, in the unique color selection step shown in FIG. 2, the digitized color image data of the mature strawberry fruit is input to the monitor screen as shown in FIG. In the reproduced image, a red portion which is a color characteristic of the portion is selected as a desired unique color.
This color data can be obtained from digital images of all color systems obtained from images of actual strawberries taken at the work site with a digital camera. Color images that have already been taken can be obtained with a scanner. It may be obtained from a digitized image.

Next, in the color element binary processing step, the color data of the entire red color system of the selected red portion is converted into color data of two color elements excluding the color elements related to light and dark by a computer program, and FIG. As shown in (b), color space data that can be displayed in color on a biaxial coordinate plane is obtained.
At this stage, the color of the peripheral part of the selected part of the strawberry is partially included.

In the next color pattern registration step, the color space data shown in (b) of FIG. 4 obtained in the color element binary processing step is displayed on the monitor screen, and a more characteristic red portion is displayed from the color image. Further, the dark red is selected, and this dark red is stored and registered in the memory as color data of the color pattern of the target portion as shown in FIG.
In this image processing, a color pattern is determined by selecting a block representing the most characteristic color in the color space data displayed in a block shape, and a desired matured strawberry is reliably identified. .
Once the color pattern obtained is registered and recorded in a memory such as a hard disk or flash memory, the color pattern data can be recalled from the stored memory even if the computer is turned off. This can be used any number of times.

For the process of searching for the fruit of the strawberry after that, in the next work area color element binary processing process, the area including the periphery of the strawberry seedling where the matured strawberry to be worked on is all-digital digital Take a color image with a digital camera, and use a computer system to convert the color data of all colors into color data of two color elements excluding color elements related to light and dark by a computer program, and color on the biaxial coordinate plane Obtain displayable color space data.
The color space data is stored in the memory as a variable on the biaxial coordinate plane.
Then, in the next target portion position display step, the color data obtained in the work area color element two-dimensionalization processing step is compared with the registered actual color pattern data which is dark red, and the two match. Only the portion is extracted, the data is stored in the memory, and the existing position is displayed in the image on the monitor screen as shown in FIG.

Then, in the next target part marking process, the size of the dark red part in the image to be recognized by the computer is adjusted with respect to the fruit of the dark red part whose existence position is displayed in the monitor screen image. The enclosure range in the screen is determined, and the position data of the dark red portion displayed in the monitor screen image in the target portion position display step is automatically present for the position data in the enclosure range. And marking processing is performed at the identification position.
In the marking priority order determination step, when a plurality of dark red portions are displayed in the monitor screen image in the target portion marking step, each dark red portion is subjected to a program in which priority conditions are set by a computer. Automatically mark in priority order.
This selects one to work on.

Recognition of strawberry fruit and stem is performed in the following combined sorting process.
This composite sorting step is performed after the color pattern data of the stem and the stem portion is also stored and registered in the previous step.
That is, the target part color pattern data is stored and registered separately for the real part of the strawberry and the red part of the stem and the green part of the stem are separately registered. The color pattern data is labeled, and the monitor screen image in actual work is set so that both can be processed simultaneously in parallel and displayed on the same screen, and the monitor screen image is displayed in the target partial position display step. It is automatically determined according to the program that there is a strawberry fruit with a fruit and a stem only when the enclosing range of both red and green partly overlaps among the fruit part of the strawberry with the existence position displayed inside, otherwise The position data of is discarded.
By doing so, an image that is erroneously identified and taken with respect to the real and the stem is deleted, and the position and range of the real and the stem are clarified.
Based on the simultaneous existence and position data of the fruit and stem obtained in this way, the work robot can avoid the fruit and harvest it by cutting the stem, or perform various operations such as picking only flowers and thinning out It becomes possible.

In the strawberry, the upper part of the strawberry is enlarged and displayed above the position of the enclosed range of the position of the matured strawberry specified in the image on the monitor screen, and the image of the area is separated and further converted into color space data. .
The processing result is stored on the memory as a variable representing the color space data.
Further, the color space data is collated based on the color pattern data, and only the pixels having the color of the target portion are extracted, and the processing result is stored in the memory as display image data.
By applying a labeling process to this display image, the positions of stems and stitches on the image are specified. This makes it possible to accurately identify the position of the stem for cutting and picking the stem without grasping the real part of the mature strawberry.

Stems are very difficult to identify by themselves because they are thin and small, and the fruit and flower parts were relatively easy to identify because of their size and color characteristics. There is a contradiction that it is easy to damage the fruit or the flower if it is directly picked up by a machine when picking.
Therefore, in the present invention, it is possible to efficiently identify a difficult-to-find stem by associating a real and a stem and simultaneously performing data processing in parallel.
At the same time, in the picking operation by the robot, if the stem portion is cut out without grasping the fruit or the flower, it is possible to harvest and collect without damaging the fruit or the flower.

Further, in the present invention, when only a desired target portion is identified from the same green similar colors such as cucumbers and bell peppers with leaves as a background, the resolution of the color space data is made finer, so that the subtle color It becomes possible to catch the difference.
Moreover, in such a case, the method of specifying the target part by performing a plurality of labeling similar to the case of identifying the strawberry can be used.
The present Example 2 shows an example in the case of cultivation in which green peppers and cucumbers of similar colors are mixed, and the case where it is difficult to discriminate by color because green peppers and cucumbers are the same green as follows. The second embodiment will be described with reference to FIG. 3 and FIG. 7. In the unique color selection step, the digitized color image data of peppers and cucumbers is reproduced on the monitor screen, From green peppers and cucumbers, the green color of green peppers and green parts of cucumbers, which are characteristic colors of those parts, are selected as desired intrinsic colors.

Next, in the color element binary processing step, the color data of the entire green color system of the selected green portion is converted into color data of two color elements excluding the color elements related to light and dark by the computer program, and the biaxial coordinate plane To obtain color space data that can be displayed in color.
In the color pattern registration step, the color space data obtained in the color element binarization processing step is displayed on a monitor screen. From the color image, a more characteristic green portion is further green for peppers and green for cucumbers. The green and yellowish green are selected and stored in the computer as color pattern data (shown in (b) of FIG. 7) of the target portions of pepper and cucumber.

  Next, in actual work, in the work area color element binarization process, an area including the periphery of the green pepper and the cucumber seedling to be worked is shown in a digital color image of all colors (shown in FIG. 7A). ) Using a digital camera, and using a computer system, the color data of all colors is converted into color data of two color elements excluding color elements related to light and dark by a computer program and displayed in color on a biaxial coordinate plane. Obtain each possible color space data.

In the target partial position display step, the green color space data of peppers and cucumbers obtained in the work area color element two-dimensionalization process step, the green color of the green peppers of the green color and the green color of the cucumbers of the registered color pattern data are displayed. Check color data.
Then, the existence position is displayed in the image of the monitor screen only in the portion where the color space data and the color pattern data that can be displayed in color on each of the biaxial coordinate planes match.

Then, in the target part marking step, the size of the green and yellow-green part in the image to be recognized by the computer with respect to the green pepper and yellow-green cucumber whose presence positions are displayed in the image on the monitor screen The enclosure range (shown in FIG. 7C) is determined in accordance with
Then, the position data of the green and yellowish green parts displayed in the image of the monitor screen in the target part position display step are automatically identified for those in the enclosed range, Marking process.
In this case, it may be difficult to identify the presence of only a color if it is a similar color.
Therefore, round bell peppers and long cucumbers have different enclosure sizes, so that the position of the bell pepper can be automatically identified by selecting an enclosure range that matches the peppers as a selection condition.

In FIG. 7, (a) is a color photographed image of the camera, and (b) is an image displayed in the image of the monitor screen only in a portion where both the color space data of cucumber and the color pattern data match. (2) is an image displayed in the image of the monitor screen only in a portion where both the color space data and the color pattern data of bell pepper match.
As shown in FIG. 7 (c), the cucumber has a vertically long frame with different enclosure ranges when the enclosure range is pepper and cucumber, that is, the color pattern data is misidentified due to the difference in the enclosure range. What is likely to be detected is identified, and the location of what is desired is specified.
Also, in the marking priority order determination process, if the enclosed area is set to the priority order by the yellow-green part and the longer frame in the monitor screen image in the target part marking process, the cucumber is automatically selected by the computer. Will be.
Further, by applying a labeling process to the display image, the size and position of the peppers and cucumbers on the image are distinguished, and each can be specified individually.

Next, a description will be given of an apparatus for specifying a position of a target portion of a plant by the method for specifying a position of the target portion of the plant.
As shown in FIGS. 8 (A) and 8 (B), this position specifying device has a computer main body equipped with a CPU 1 and a memory 2 on a base plate, and displays a color image on the computer main body. An input device 3 for inputting as digital data and a display device 4 for monitor display for viewing images are connected and placed.
Such a computer system C1 can use a widely used desktop type or notebook type personal computer as the apparatus itself.
The computer main body of the computer system C1 processes the color and position data of the digital color image according to the processing step by the method for locating the target part of the plant according to any one of claims 1 to 5. A program P 1 for executing the above is incorporated in the CPU 1 and the memory 2.

In implementing the above-described position specifying method, the memory 2 stores the color and position information of the digital image, and stores the color pattern of the target portion and the like and stores it in the computer main body as registered data.
In the CPU 1, data processing such as data processing, collation, comparison, calculation, and determination for obtaining color space data and color pattern data of the target portion is performed by a computer program.
In the display device 4 for monitoring, the digital color image data input by the digital color camera is output, or the color space data or color pattern data of the target portion of the processed image data is present as a color image. Etc. can be seen and confirmed.

In addition to the color image digital input device 3 such as a digital color camera, an input device includes a keyboard 10, a mouse, a touch panel, a light pen, and the like for registering a color pattern as necessary. Can be made available to them.
Then, by operating the input device and output device, registration of the color pattern of the target portion, confirmation of the identification result, execution of image processing, confirmation of the execution process, etc. can be performed on the monitor screen each time. It is possible to display the color space of the two color elements excluding the color elements related to light and dark and the color pattern of the target portion, and to appropriately check each identification process, identification result, and other parameter values on the screen.
Then, the position of a target portion such as a fruit of a plant or a flower in an image photographed by a digital color camera can be specified on the monitor screen of the display device 4.

Next, a robot for working on a plant using the above-described apparatus for specifying the position of a target part of the plant will be described with a strawberry picking robot. The working robot is arranged on a bench-type body 5 as shown in FIGS. Comprises driving means 6 for moving the entire apparatus.
In the present invention, a device that includes the drive means 6 that moves and performs various operations with respect to a device that does not actually perform work, such as the position specifying device, is referred to as a “working robot”.
In this working robot, as shown in FIG. 11, the track means 6 is provided with four-wheel wheels and a battery-powered motor drive unit, and the wheels are driven by a computer command via a drive control unit. Is called. At this time, the body 5 is provided with a body position confirmation sensor 11 for measuring the distance in front, so that it can detect surrounding obstacles and perform automatic traveling.
The digital color image input device 7 for photographing the plant provided on the body 5 is installed near the work hand and on the central support shaft so as to photograph the target portion, and the computer main body of the computer system C2. Connect to.
A work hand 8 for performing work is also installed on the upper portion of the central support shaft. The work hand 8 includes a front-rear drive manipulator 9c, a left-right drive manipulator 9a, and a vertical drive manipulator 9b. The work hand 8 can be freely moved in all directions so that the work can be performed.
A drive control unit that performs electronic control such as movement of the body 5 and driving of the work hand 8 is provided on the body 5.
Then, command signals and data processing signals are transmitted and received between the drive control unit and the computer C2, so that the computer system C2 can run and stop and drive control of each manipulator 9 can be performed by the drive program P2. .

The position of the target portion of the plant according to any one of claims 1 to 5, wherein the position of the target portion of the plant is specified from the digital image input by the input device 7 for the digital color image. The plant target part location specifying apparatus according to the third embodiment having a dedicated program P1 for performing the specifying method is mounted.
The part of the position specifying device may be mounted on the body 5 of the work robot, but it can be removed from the body 5 in order to reduce the weight of the work robot and improve operability.
In that case, the computer system C2 having the work robot drive program P2 and the computer system C2 having the dedicated program P1 for performing the position specifying method may be integrated.
If data communication, drive commands for drive control, and the like are connected via a wireless communication line, the movement of the work robot due to the connected wiring does not occur, and the work can be freely performed.
Then, the manipulator 9 is operated while specifying the position of the target portion of the plant, and the work hand 8 is automatically moved to the specified position of the target portion, so that the work can be executed.

  Then, for example, as shown in FIG. 13, the work robot for picking strawberries inputs an image to the surrounding strawberries M with the digital color image input device 7 while moving along the straw in the strawberry field. The position of the target part of the plant is quickly identified from the digital image by the above-described method, the manipulator is operated on the stem S of the ripe strawberry, the work hand is automatically moved, and the stem is shown in FIG. It becomes possible to execute the harvesting of ripe strawberry fruit M by cutting the stem S portion with a cutter provided in the hand.

When the work robot is used at the site, the color pattern data of the target part to be worked is stored in the computer and registered, so that the actual work at the site can be carried out from the work area color element binarization process. Become.
Among the method for specifying the position of the target part of the plant according to any one of claims 1 to 5, until the unique color selection step, the color element binary processing step, and the color pattern registration step, Since the work robot can be used without being performed at the actual work site, the work robot as a product can be configured to perform only actual work.

It is process drawing which shows a data processing process. It is process drawing which shows the data processing process of a strawberry. It is process drawing which shows the data processing process of a bell pepper and a cucumber. (A) is a digital all color system color image of strawberry, (b) is color space data of strawberry, (d) color pattern data of strawberry, and (c) is a monitor screen diagram showing a state where the position of the strawberry is specified. . (A) is a monitor screen diagram of color pattern data, (b) is a mathematical expression diagram, (c) is a monitor screen diagram of color pattern data, and (d) is a mathematical expression diagram of color pattern data. (A) is a chromaticity distribution diagram, (b) is a monitor screen diagram showing an xy chromaticity distribution obtained by inputting RGB. It is a monitor screen figure of an image of a bell pepper and a cucumber. (A) which shows the position identification apparatus of this invention is a front view, (b) is a side view. It is the perspective view seen from the front side of the working robot of the present invention. It is the perspective view seen from the rear surface side of the working robot of FIG. (A) which shows the work robot of this invention is a top view, (b) is a front view, (c) is a side view, (d) is a rear view. (A) which shows the work hand part of a work robot is a top view, (b) is a side view, (c) is a front view. It is a typical front view which shows the working state of a working robot.

Explanation of symbols

1 CPU
2 Memory 3 Color Image Input Device 4 Display Device C1 Positioning Computer System C2 Driving Computer System 5 Body 6 Driving Means 7 Digital Color Image Input Device 8 Working Hand 9 Manipulator 9a Manipulator 9b Manipulator 9c Manipulator 10 Keyboard 11 Body position confirmation sensor P2 drive program P1 Position identification program S Stem M Actual

Claims (9)

A computer system comprising a data and command input device including a digital camera, an output device including a display device, a computer main body having a CPU and a memory, and a computer program for data processing incorporated in the computer main body,
One or a plurality of parts of one or many kinds of plants are targeted, and digitized color image data of all the color parts of the target part are inputted and stored in a computer main body, which is then displayed on the monitor screen of the display device And a unique color selection step of selecting a characteristic color of the target portion in the reproduced image as a unique color;
Color element binary which obtains color space data which can be displayed in color on a biaxial coordinate plane by converting the color data of the selected all-color system of the specific color into color data of two color elements excluding color elements relating to light and dark Chemical treatment process,
The color space data obtained in the color element binary processing step is displayed on the monitor screen, and one or a few characteristic colors are further selected from the color image, and this is used as color pattern data for identification. As a color pattern registration process to be stored in the computer as
In actual work, using the computer system provided with the computer program, the area including the periphery where the target portion to be worked exists is photographed with the digital camera, and from the photographed all-color digital color image, A work area color element binarization process for obtaining color space data that can be displayed in color on a biaxial coordinate plane by converting to color data of two color elements excluding color elements relating to light and dark;
The color space data of the work area obtained in the two-dimensional processing step of the work area color element is collated with the stored color pattern data, and the two match in the image of the work area displayed on the monitor screen. Target part position display process for displaying only the part that has been differentiated,
A method for specifying the position of a target portion of a plant, characterized in that the position of the target portion can be accurately displayed from an area including the periphery in the monitor screen regardless of the brightness of the work environment.   The method for specifying a position of a target portion of a plant according to claim 1, wherein after the target portion position display step, the target portion whose presence position is displayed in the image of the monitor screen is included in an image to be recognized by a computer. An enclosure range in the screen is determined in accordance with the size of the target portion, and the target portion position data displayed in the monitor screen image in the target portion position display step is automatically detected for those in the enclosure range. A method for specifying a position of a target portion of a plant, comprising: a target portion marking step for identifying the position of the target and marking the data of the identification position.   3. In the target part marking step according to claim 2, for each of the plurality of target parts displayed in the image on the monitor screen, marking is automatically performed in priority order according to a program in which priority conditions are set by a computer. A method for specifying a position of a target part of a plant, comprising a marking priority order determination step.   The color pattern registration step according to claim 1, wherein a plurality of color pattern data of the target portion is stored from different locations for one target portion, and the color pattern data of the plurality of target portions is labeled by a computer. In addition, the labeled color pattern data is set so that it can be simultaneously displayed in the image of the monitor screen in actual work, and the target of the plant whose presence position is displayed in the image of the monitor screen in the target partial position display step 4. A compound selection step of extracting a portion where overlapping ranges of portions having different portions partially overlap and automatically determining that the target portion exists only in the extracted portion. The method for specifying the position of the target portion of the plant according to any one of the above. The biaxial coordinates, which are color space data that can be color-displayed on a biaxial coordinate plane by converting the color data of all color systems according to claim 1 into color data of two color elements excluding color elements relating to light and dark. How to ask
From the xy chromaticity diagram calculated from the CIE-XYZ color system,

5. The chromaticity coordinates (x) and (y) values are obtained by inputting R, G, and B luminance values into the mathematical formulas (a) and (b). The method for specifying the position of the target part of the plant according to Item.   The position specifying device of the target part of the plant by the method of specifying the position of the target part of the plant according to any one of claims 1 to 5.   A plant target part position specifying device according to the plant target part position specifying method according to any one of claims 1 to 5, wherein the computer main body includes a CPU (1) and a memory (2). A computer system (C1) comprising a digital input device (3) for color images and a display device (4) for monitoring to view the image, wherein the computer body is the computer system. A program (P1) for executing processing of color and position data of a digital color image according to a processing step in the method for specifying a position of a target portion of a plant according to any one of 1 to 5 is provided, An apparatus for locating a target part of a plant to be operated.   A working robot using the plant target part position specifying device according to claim 6 or 7.   A working robot using the device for specifying a position of a target portion of a plant according to claim 6 or 7, wherein the plant is mounted on a body (5) for mounting the device, which is provided with driving means (6) for traveling. A digital color image input device (7) for photographing, a work hand (8) for performing work, a manipulator (9) for driving the work hand, and a sensor (11) for checking the body position; The body (5), manipulator (9), work hand (8), digital color image input device (7) and body position confirmation sensor (11) are operated by computer instructions. A drive control unit, and a drive program (P2) for instructing an operation of each part constituting the device and a digital color image input device (7) on the body (5) or at another place. Input digital A computer main body (C2) comprising a data processing program (P1) for performing the method of locating a plant target part according to any one of claims 1 to 5, wherein the position of the plant target part is specified from a local image. The drive controller and the computer (C2) transmit and receive command signals and data processing signals, and the position specifying device specifies the position of the target portion of the plant while the manipulator (9 ) And the work hand (8) is automatically moved to the position of the specified target part, so that the work on the target part of the plant can be performed by the work hand (8). Robot for work.

Images