GB2151353A - Colour signal separating apparatus - Google Patents

Abstract

Luminance signals (Y) and two colour signals (R, B) from a TV colour camera viewing objects. particularly fruit to be harvested, are processed by multiplying (2) the luminance signal (Y) by adjustable (R1, R1') lower and upper limit coefficients (K1, K1') for red and similar coefficients (K2, K2') for blue. Where both red and blue signal intensities fall between their respective limits (K1 Y, K1' Y, K2 Y, K2' Y), comparators (3, 3') and AND gate (4) insert a logical 1 in a frame memory. Before multiplication and comparison, the signals are preferably converted to DC relative to an absolute dark level. A strobe light, synchronized to the camera, may illuminate the objects. The aperture of the camera may also be adjustable automatically to stabilize the light level. The pixels with logic 1 may be indicated on a monitor display (5). <IMAGE>

Description

SPECIFICATION Color signal separating apparatus Background of the Invention This invention relates to a color signal separating apparatus and especially to a color signal separating apparatus for separating and binarizing signal corresponding only to particular colors from image signals provided by a monitor camera thereby to obtain binary signals corresponding to the particular colors. More particularly, the invention relates to an apparatus suited for use with a fruit harvesting machine as means to exclude the sky, leaves and the like which lie in the background of fruit from image signals provided by a monitor camera acting as means to detect positions of fruit to be harvested.

In one known type of color signal separating apparatus, the monitor camera for picking up the image of an object includes an optical filter for passing particular colors only, in order to obtain binary signals corresponding to the particular colors from image signals from the monitor camera. Another type of known apparatus carries out logical processing to obtain a sum of or a difference between color signals corresponding to particular colors among a plurality of color signals output by a color video camera.

In order to binarize the color signals, the color signals are first stored in a frame memory or the like and are thereafter put to digital logical processing. Such a process requires a complicated construction and the logical processing is time-consuming. Where the optical filter is employed, it is necessary to have available many kinds of filter corresponding to particular colors to be binarized.

Furthermore, both of the known apparatus noted above have the serious disadvantage that the image signals obtained under conditions where brightness or luminance levels deviate from a certain value cannot be binarized without greatly complicating software therefor. Especially, it is impossible to maintain the brightness to a certain level where the apparatus isused with the above-noted fruit harvesting machine which operates outdoors.

On the other hand, the particular colors to be sampled have a certain range, further complicating the process with the known construction, and simple improvements in hardware are inadequate to permit the image signals to be processed on real time basis.

Summary of the Invention This invention has been made having regard to the above state of the art, and its object is to provide a color signal separating apparatus capable of obtaining binary signals corresponding to particular colors on real time basis and regardless of luminance levels.

In order to achieve this object, the color signal separating apparatus according to this invention comprises chief features based on the following principle: A certain color for may be expressed by a vector represented by the following equation (i): f = a for + b G + .... . . (i) wherein a, b and c are coefficients, and FR f and p3 correspond to the vectors of the three primary colors of light.

Therefore, particular colors are determinable by determining the coefficients a, b and c, but the color ffi cannot be determined without obtaining definite brightnes since each of the coefficients includes a brightness (luminance level) parameter.

Image information input means comprising a monitor camera such as a color video camera normally is adapted to output a luminance signal Y and signals R-, G and B of the three primary colors separately. The signals Y, R, G and B have the relationship represented by the following equations (ii)-(v): R/Y = KI .. . . (ii) B/Y = K2 . . . (iii) G/Y = K3 (iv) K1 + K2 + K3 = 1 . ... (v) Thus, the certain color ff is normalized and determined regardless of brightness by determining the luminance level Y and the parameters of any two of the gains K1 to K3.

More particularly, where, for example, a particular color put which is red or redish is to be picked out, a binary signal corresponding to the particular color if is obtained by a simple logical processing by predetermining K1 and K2 in the above equations (ii) and (iii) utilizing the luminance signal Y, the red color signal R and the blue color signal B.

The principle of this invention is also applicable where the read signal R, the green signal G and the blue signal B are available, for a sum in intensity of these signals R, G and B equals an intensity of the luminance signal Y. That is to say the principle of this invention is applicable if any three of the four signals are made available.

The divisions as equations (ii) and (iii), deteriorate processing precision because it is difficult to process actual signals in broad band. In order to avoid this advantage this invention modifies the above equations (ii) and (iii) into multiplications and the data are processed by multipliers which carry out the following equations (ii') and (iii'). Thus, the invention is capable of selecting particular colors on real time basis without requiring a memory for storing the signals prior to the processing.

R=K1Y (ii') B = K2Y (iii') in order to carry out the above signal processing, the apparatus according to this invention comprises multipliers for amplifying a luminance signal separated from the image signals by predetermined gains, comparison circuits for comparing color signals with upper limits and lower limits corresponding to separating levels output by the multipliers and binarizing the color signals, and a gate circuit for carrying out a logical operation on binary color signals provided by the comparison circuits, the binary signals corresponding to the particular colors being output by the gate circuit.

The features set out above produces the following excellent effect: Binary signals corresponding to particular colors are obtained with high precision on real time basis and regardless of brightness, only by setting ratios of the luminance signal and the color signals to values corresponding to the particular colors.

Furthermore, the sampling color may be selected as desired only by changing the ratios.

Since the range of the color to be selected may be determined as desired by the range of gains by which the luminance signal is multiplied, not only one particular color but signals including a certain range of color may be binarized. This is a very important advantage when binarizing a color such as of fruit which has a certain range.

In addition to the principal object as noted above, this invention has the following subsidiary objects: (i) To diminish adverse influences of various in the brightness of the object by incorporting into the described main features means to eliminate pedestal level variations from the luminance signal and the color signals which are output in DC coupling, and to convert these signals into DC component signals from an absolute ground level prior to logical processing thereof: (ii) To diminish adverse influences of brightness of the field where a fruit color indetifying operation is taking place, by incorporating into the described main features means to automatically adjust a stop of the monitor camera:: (iii) To provide the described main features with means to emit light in a predetermined amount to fruit or the like in synchronism with an image pickup in order to meet varied field conditions such as the light coming from behind, backlight, fine weather, dull weather, rainy weather and so forth: (iv) To provide the desired main features with means to emit light to the object in a predetermined amount a plural number of times from different directions, and means to carry out a logical operation on a plurality of binary image signals which are obtained in synchronism with the light emission, in order to separate and identify in a reliable manner a plurality of objects which lie close together one behind another.

Brief Description of the Drawings Several embodiments of this invention will hereinafter be described with reference: the drawings, in which: Figure 1 is a block diagram showing a color signal separating apparatus according to a principal embodiment of the invention, Figure 2 is a schematic view of a control panel used in the above embodiment, Figure 3 is a block diagram of the above apparatus where a light emitting device and an automatic shutter speed adjusting device are additionally provided, Figure 4 is a circuit diagram of the above embodiment where signal modifying means are additionally provided, Figure 5 is a time chart showing operation of the signal modifying means, Figure 6 is a circuit diagram showing an automtic camera stop adjusting device, Figure 7 is a side view of another embodiment, Figure 8 is a time chart showing operation of a light emitting device used in the embodiment of Fig. 6, Figure 9 is a block diagram showing a further embodiment, Figure 10 is a perspective view showing use of the embodiment of Fig. 8, Figures 11A to 11Care views showing images responsive to binary signals obtained by the embodiment of Fig. 8, respectively.

Detailed Description of the Preferred Embodiments Referring to Fig. 1, a monitor camera 1 comprising a color video camera picks up a fruit tree A and its picture information in the form of luminance signal Y is input to multipliers 2 and 2'. These multipliers 2 and 2' multiply the luminance signal Y by gains K1, K1', K2 and K2' set by potentiometers R1, R1', R2 and R2', respectively. Resulting values are input to comparison circuits 3 and 3' as reference levels K1Y, K1'Y, K2Y and K2'Y. The comparison circuit 3 selects and binarizes red color signals R between upper limit reference level K1 Y and lower limit reference level K2Y, and the comparison circuit 3' selects and binarizes blue color signals B between upper limit reference level K1 'Y an lower limit reference level K2'Y. A gate 4 obtains 3 logical AND of binary color signals R' and B' output from the comparison circuits 3 and 3'. Thus the gate 4 outputs only those particular colors ss:, in the form of particular binary color signals F, corresponding to fruit a lying within the range determined by using the gains K1. K1', K2 and K2', and an image responsive to the binary signals corresponding only to the fruit a is displayed on a screen 5a of a monitor television 5.

In order to facilitate determination of the particular colors to be binarized, the color signal separating appratus according to this embodiment comprises a control panel 6a having control knobs Cl, C1', C2 and C2' arranged two dimensionally thereon for controlling the potentiometers R1, R1', R2 and R2', respectively. More particularly, as shown in Fig. 2, the control knobs C1 and Cl' for controlling the potentiometers R 1 and R 1' to determine binarizing levels of the red color signals R are arranged one above the other to be movable in horizontal directions, whereas the control knobs C2 and C2' for controlling the potentiometers R2 and R2' to determine binarizing levels of the blue color signals B are arranged side by side to be movable in vertical directions.The panel 6a provides, above the control knobs C1 and C1' and at a righthand side of the control knobs C2 and C2' a color display which shades off in accordance with positions of the control knobs C1, C1', C2 and C2', that is to say the aforesaid gains K1, K1', K2 and K2'. Thus it enables the operator to recognize by intuition that the colors to be binarized are in range f defined by extension lines from th four control knobs.

The portion having a triangular shape in Fig. 2 is colored such that green shades off into red in the righthand direction and into blue in the upward direction, whereby the range of binarized color signals is readily perceptible.

It is to be noted that reference R, B and G in Fig. 2 indicate ranges of the three primary colors, red, blue and green, and that reference W indicates a range of white.

The apparatus of Figs. 1 and 2 may be modified to include a light emitting device and an automtic shutter speed regulating device as described hereinafter.

Referring to Fig. 3, a monitor camera 1 comprising a color video camera contains a shutter 50 acting as means to control incident light into the camera and carries a strobe 52 acting as light emitting means. The camera 1 further comprises a shutter controller 54 to control shutter speed of the shutter 50 and cause the strobe 52 and the shutter 50 to operate in synchronism by means of a trigger signal TG for the strobe 52. The shutter 50 is thus actuated simultaneously with light emission by the strobe 52 so that the camera 1 picks up fruit with a constant light. (See the embodiment of Figs. 7 and 8 for details of strobe control.) Color picture signals for one frame provided by the monitor camera 1 having the above construction are converted by a color identifying device 20 into binary signals corresponding only to the fruit, and the binary signals are stored in a frame memory 22.At the same time the results are displayed on a monitor television 5 and a control device 24 carries out fruit position detection and other processing, whereby the apparatus is capable of automatically discriminating fruit from other objects.

Referring to Fig. 4 showing details of the color identifying device 20, the pictorial information or luminance signal Y provided by the monitor camera 1 is fed through one of DC restoration circuits 28 to be described later to a multiplication circuit 30 having two multipliers 2 and 2'. At the multiplication circuit 30 the luminance signal Y is multiplied by the gains K1, K1', K2 and K2' set by potentiometers R1, R1', R2 and R2', respectively, and is thereafter input to comparison circuits 3 and 3'. The values thus obtained are utilized as the reference levels Kl Y, Kl 'Y, K2Y and K2'Y for binarizing the red color signals R and the blue color signals B.

The red and blue color signals R and B output together with the luminance signal Y by the monitor camera 1 are also fed to the comparison circuits 3 and 3'. Each of these comparison circuits 3 and 3' constitutes a window comparator having two comparators Al and A2 and a reverse buffer G1. The color signals R and B lying between the upper limits K1Y and K2Y and the lower limits K1'Y and K2'Y of the reference levels are selected and binarized. The AND gate 4 obtains a logical AND of the resulting binary signals R' and B', and the frame memory 22 receives the particular binary color signals F corresponding only to the fruit lying in the range determined by the potentiometers R1, R1', R2 and R2'.

The monitor camera 1 outputs the luminance signal Y, the red color signals R and the blue color signals B in AC coupling, and therefore these signals have an inconstant pedestal level Pd. The DC restoration circuits 28 are provided to control the pedestal level Pdto be an absolute ground level (GND level, OV). Each of the DC restoration circuits 28 comprises a sample-hold circuit A3 to samplehold an input signal V1 in synchronism with rises HD of a horizontal synchronization signal input from a horizontal amplifier Ao, as shown in Fig. 5, and a differential amplifier A4 to obtain a difference between the input signal Vi and an output signal Vref of the samplehold circuit A3. In other words, the DC restoration circuit 28 is adapted to convert the input signal Vi into a DC component signal Vo from the absolute ground level.

Where the construction of Figs. 3 to 5 is incorporated into the apparatus of Figs. 1 and 2, the luminance signal Y and the color signals R and B are processed at the DC restoration circuits 28 prior to input to the multiplication circuit 30 and the comparison circuits 3 and 3'. Therefore, the color signals are separated with high precision even if there are variations in the brightness of fruit under measurement.

Fig. 6 shows an example of stop control for the camera. While the relationship between the controller 54 and the control device 24 is slightly modified, this example is capable of eliminating the influence of the brightness of fruit still further.

In Fig. 6 reference number 40 denotes a stop encoder operatively connected to a stop of the camera and reference number 42 denotes a motor for adjusting an opening degree of the stop. Signals from the encoder 40 are input to two comparators 44 and 44', while the luminance signals Y from the monitor camera 1 are also input to these comparators 44 and 44'. Output signals from the comparators 44, 44' are input to a motor drive circuit including switch circuits 48 and 48'. When the switch circuit 48 is energized, for example, the motor 42 is put to forward rotation, and when the switch circuit 48' is energized, the motor 42 is put to reverse rotation. Thus the opening degree of the stop is automatically adjusted in accordance with variations of the luminance signals Y.

It is of course possible to incorporate only the construction of Fig. 6 into the apparatus of Figs. 1 and 2.

An embodiment shown in Figs. 7 and 8 will be described next, which is suited for use against the light, in cloudy weather and late in the afternoon. As seen in Fig. 7, this embodiment has a monitor camera 1 of the color signal separating apparatus 6 mounted at an extreme end of an articulate arm 62 of a fruit harvesting robot hand. The extreme end of the arm 62 carries adjacent the camera 1 at least one strobe 52 acting as light emitting means. Reference number 53 dentes a fruit case. The arm 62 which is vertically oscillatable about joints is attached to be swivellable at its base end to a carriage 51. The arm 62 is adjusted to extend in a right direction and then the camera 1 picks up fruit a. Means is provided to select and binarize only those particular colors in the picture signals corresponding to the fruit a (as already described with reference to Figs. 1 and 2).This means provides information concerning fruit position, direction and the like. Brightness at times of picking up fruit a is kept constant to a certain degree by causing the strobe 52 to emit light in synchronism with the fruit pickup action.

This embodiment is characterized by a control system for the strobe 52.

As shown in Fig. 8, a vertical synchronization signal VD is output by the camera, and a memory intake signal FMCD is also generated for taking the picture signals into a frame memory (such as denoted by reference number 22 in Fig. 3) which temporarily stores the picture signals from the camera 1. The signal FMCD generates a first synchronization signal F1 which, together with the signal VD, outputs a second synchronization signal F2. Rises of the signal F2 are differentiated to produce a third synchronization signal SH1 which in turn produces a starter pulse signal SH2 for causing the strobe 52 to emit light. Thus, the strobe 52, actuated by the starter pulse signal SH2, is adapted to automatically emit light in synchronism with fruit image pickup or with image intake by the frame memory.

The strobe 52 emits light in an amount enabling proper exposure within an operational range L of the arm 62 or a range of arm movement wide enough to harvest fruit a.

Formation of unsatisfactory fruit image due to underexposure is thus avoided.

On the other hand, the camera 1 may be stopped down to a fixed value to maintain proper exposure to the amount of light emitted by the strobe 52 within the operational range L. Thus the camera 1 is fit for use as sufficiently stopped down, permitting the influence of natural light to be ignored and satisfactory picture information to be obtained in a reliable manner whether in favorable light or backlight and regardless of the weather or time of the day.

The embodiment of Figs. 9 to 11 will now be described which is adapted to pick up a particular object with a single camera where, for example, a plurality of fruit are close to one another such that one fruit is seen behind another.

Referring to Figs. 9 and 10, a single pickup monitor camera 1 such as a video camera is provided with strobes 52 opposed to each other across the camera 1 and acting as light emitting means. The three elements are arranged such that optical paths thereof meet substantially at one point. (However, the light emitting means may comprise one strobe 52 switchable between the right side and left side of the camera 1.) The pickup video camera 1 picks up an object or fruit a in this embodiment as in the preceding embodiments. Resulting image signals D1 are binarized by a binarizing means or color identifying device 20 on the basis of particular colors corresponding to the fruit.

Binary signals D2 are output to a frame memory 22' and are then used to show an image of the fruit a on a display means such as a monitor television 5.

The frame memory 22' comprises memories 22a and 22b corresponding in number to the strobes 52a and 52b for emitting light in a predetermined amount a plural number of times from different directions. Output of the frame memory 22' is fed to the display means 5 by way of an AND circuit 60 acting as means to obtain a logical AND. The memory 22a receives output signals from the color identifying device 20 and trigger signals TG for the strobe 52a, both through an AND gate 62, and the memory 22b receives output signals from the color identifying device 20 and the trigger signals for the strobe 52b, both through an AND gate 64.

A switch SW is operable to alternately provide the plurality of strobes 52a and 52b with the trigger signals TG, whereby the fruit a receives flashes from different directions. At the sqme time the video camera 1 picks up the fruit a, and binary image signals D2 are stored in the memories 22a and 22b corresponding to the strobes 52a and 52b, respectively. The logical AND of the image signals stored in these memories 22a and 22b is obtained by the AND circuit 60.

As shown in Fig. 11A, the binary image D2' stored in one of the memories 22a is such that three fruits Al, A2 and A3 overlap one another and two of the fruits Al and A3 are separated by a shade appearing along a border therebetween owing to the light emission by one of the strobes 52a.

The binary image D2" stored in the other memory 22b, which is as shown in Fig. 11 B, is such that two fruits Al and A2 are separated by a shade appearing along a border therebetween since light is emitted from the different direction.

Accordingly, an image signal D3 as shown in Fig. 11 C wherein the three fruits Al, A2 and A3 are separated from one another is automatically obtained only by subjecting the two binary images D2' and D2" to the logical processing to obtain the logical AND of the two. Thus the invention provides the apparatus capable of separating and selecting fruit where a plurality of fruits are close together, by means of the signal monitor camera and by the very simple process.

The logical processing described above may not only be matrix processing to treat the signals D2' and D2" stored in the two memories 22a and 22b of the frame memory 22' but may employ a simple AND gate to carry out high speed processing. It is thereby possible to simplify the processing required for separating and selecting fruit.

Furthermore, the described logical processing may be carried out likewise whether it takes the analog mode or the digital mode.

The apparatus in use does not require the shooting direction of the camera, and it is sufficient to emit light to the object a plural number of times from different direction.

Therefore, the image signals stored in the plurality of memories of the frame memory have their coordinates in perfect agreement.

This has the advantage of necessitating no special coordinate changing processes for separating and selecting a plurality of fruits.

Claims (5)

1. A color signal separating apparatus for separating and binarizing signals corresponding only to particular colors (off) from image signals provided by a stop adjustable monitor camera (1) thereby to obtain binary signals (F) corresponding to the particular colors (fF), said apparatus comprising;; multipliers (2,2') for amplifying a luminance signal (Y) separated from said image signals by predetermined gains (K1, K1', K2, K2'), comparison circuit (3,3') for comparing color signals (R,B) with upper limits (k1Y,K2Y) and lower limits (k1 'Y,K2') corresponding to separating levels output by said multipliers (2,2') and binarizing said color signals (R,B), and a gate circuit (4) for carrying out a logical operating on binary color signals (R',B') provided by said comparison circuits (3,3'), and binary signals (F) corresponding to said parti cular colors ( J7:) being output by said gate circuit (4).

2. A color signal separating apparatus as claimed in claim 1, further comprising; DC restoration circuit (28) for converting said liminance signal and color signals into DC component signals from an absolute groung level prior to input to said multipliers and said comparison circuits.

3. A color signal separating apparatus as claimed in claim 1 or 2 further comprising; at least one strobe for emitting light simultaneously with a fruit image pickup on the basis of a vertical synchronization signal (VD) output by said camera.

4. A color signal separating apparatus as claimed in any one of claims 1 to 3 wherein said camera is provided with strobes opposed to one another across the camera, said apparatus further comprising means to binarize both an image signal resulting from a light emission by one of said strobes and an image signal resulting from a light emission by the other strobe following the light emission by said one of the strobes, and means to carry out a logical operation on the two signals and produce a single composite binary signal.

5. A color signal separating apparatus as claimed in any one of claims 1 to 4 further comprising means to automatically adjust a stop of said camera.