JP7258345B2 - OBJECT IDENTIFICATION DEVICE AND OBJECT IDENTIFICATION METHOD - Google Patents

Description

The present invention relates to technology for identifying objects in an object recycling process or the like.

In recent years, resource saving has become one of the most important social themes, and techniques for sorting waste products have been devised in order to realize recycling (see Patent Document 1).

JP 2017-109161 A

Patent document 1 discloses a waste sorting system and the like that utilizes machine learning using a convolutional neural network. In order to perform highly accurate sorting, a large number of image data are collected and the machine learning unit 43 is made to learn. There is a need.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an object identification apparatus and an object identification method capable of highly accurate object identification with less learning data.

In order to solve the above problems, the present invention creates a first background image according to the density of the sample, and creates a three-dimensional composite image by pasting a three-dimensional image of the sample on the first background image. Then, for each pixel in the two-dimensional composite image obtained by attaching the two-dimensional image of the sample to the second background image, according to the pixel value of each pixel in the three-dimensional composite image at the same position as each pixel An object identification device is provided, which includes identification means for creating image data obtained by adding the parameters obtained by adding the above parameters, and identifying an object by a convolutional neural network model trained on the image data.

Further, in order to solve the above problems, the present invention provides a first step of measuring the weight of the sample body, a second step of calculating the volume of the sample body from the three-dimensional image of the sample body, and a first step A third step of calculating the density of the sample body using the weight measured in and the volume calculated in the second step, and creating a first background image according to the density calculated in the third step a fourth step of attaching a three-dimensional image of the sample to the first background image to create a three-dimensional composite image; and attaching a two-dimensional image of the sample to the second background image creating image data obtained by adding, to each pixel in the two-dimensional composite image obtained by adding a parameter corresponding to the pixel value of each pixel in the three-dimensional composite image located at the same position as each pixel; a seventh step of training a convolutional neural network model with the image data as learning data; and the image data obtained by executing the first to sixth steps for an object to be identified. and an eighth step of inputting the learned convolutional neural network model obtained in the seventh step to identify the object.

According to the present invention, it is possible to obtain an object identification device and an object identification method capable of highly accurate object identification with less learning data.

It is a figure showing the whole object identification device 1 composition concerning an embodiment of the invention. 4 is a flow chart showing an object identification method according to an embodiment of the present invention; 3 is a flow chart showing a specific example of a method for creating the three-dimensional composite image shown in FIG. 2; 4 is a diagram for explaining the creation method shown in FIG. 3; FIG. 3 is a flow chart showing a specific example of a method for creating the two-dimensional composite image shown in FIG. 2; 6 is a diagram for explaining the creation method shown in FIG. 5; FIG. 3 is a flow chart showing a specific example of a method for creating image data shown in FIG. 2; 8 is a diagram for explaining the creation method shown in FIG. 7; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. The same reference numerals in the drawings indicate the same or corresponding parts.

FIG. 1 is a diagram showing the overall configuration of an object identification device 1 according to an embodiment of the invention. As shown in FIG. 1, an object identifying apparatus 1 according to an embodiment of the present invention includes a belt conveyor type weight scale 4 for measuring the weight of a waste product 2 while conveying the waste product 2 supplied from a supply device 3. , a belt conveyor 5 that conveys the waste product 2, a three-dimensional (3D) camera photosensor 6, a 3D camera linear laser 7, a 3D camera 8, a 2D camera photosensor 9, and a 2D camera 10 , sorting actuators 11 and 12 for sorting and recovering the waste products 2 into the recovery containers 14 to 16, and a control device 13 that integrally controls these operations and identifies the waste products 2 by a method to be described later.

If it is necessary to increase the amount of identification per unit time, it is preferable to arrange a plurality of the object identification devices 1 shown in FIG. 1 in parallel and to operate them simultaneously.

An outline of the operation of the object identification device 1 will be described below. The weight of the waste product 2 individually supplied as a sample from the supply device 3 is measured by the belt conveyor type weighing scale 4 , and data indicating the result is transmitted to the control device 13 . Subsequently, when the waste product 2 is detected by the photosensor 6 , the 3D camera 8 suspended above receives the detection signal and operates to capture a 3D image of the waste product 2 , which is transmitted to the control device 13 . be done.

Subsequently, when the waste product 2 is detected by the photo sensor 9, the 2D camera 10 suspended above operates according to the detection signal, and a 2D color image of the waste product 2 is captured and transmitted to the control device 13. be.

The control device 13 performs arithmetic processing based on the weight, 3D image, and 2D image of the waste product 2 acquired as described above to identify the waste product 2. This arithmetic processing will be described later in detail. .

The identified waste products 2 are controlled in their falling position from the belt conveyor 5 by the sorting actuators 11 and 12 according to the identification result, and are stored in the corresponding collection containers 14 to 16, respectively.

In FIG. 1, a cylinder capable of controlling the on/off of the pushing operation is used as the separating mechanism, but it is not limited to this, and for example, an air gun, a paddle operated by an electromagnet, or a robot arm can be used. It can be.

In the above, the waste products 2 are transferred from the supply device 3 to the belt conveyor 5 in a random state in that the position in the width direction of the belt conveyor 5 and the angle formed by the traveling direction of the belt and the long side of the waste products 2 are not constant. Fall.

In addition, the 3D camera 8 detects the position change in the height direction of the reflected light line of the linear laser light on the surface of the waste product 2 moving in a certain direction using a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). A 3D image is recorded as digital data in a memory (not shown) by a so-called light section method.

FIG. 2 is a flowchart illustrating an object identification method according to an embodiment of the invention. A case of realizing the object identification method shown in FIG. 2 by using the object identification device 1 shown in FIG. 1 as an example will be described below.

Needless to say, the present object identification method is not limited to being implemented using the object identification device 1 shown in FIG. 1, and can be widely applied.

First, in step S1, the weight of the sample body is measured. Specifically, for example, the weight of the sample body is measured by the belt conveyor type weight scale 4 as described above, and the obtained data is transmitted to the control device 13 .

Next, in step S2, the volume of the sample is calculated from the three-dimensional image of the sample. Specifically, for example, the control device 13 calculates the volume of the waste product 2 based on the received 3D image by the method described above.

Next, in step S3, using the weight of the sample measured in step S1 and the volume of the sample calculated in step S2, for example, the control device 13 calculates the density of the waste product 2 as the sample. .

Next, in step S4, for example, the control device 13 creates a first background image according to the density calculated in step S3.

Next, in step S5, for example, the control device 13 creates a three-dimensional composite image by attaching a three-dimensional image of the waste product 2 as a sample to the first background image.

FIG. 3 is a flow chart showing a specific example of a method for creating the three-dimensional composite image shown in FIG. 4A and 4B are diagrams for explaining the creation method shown in FIG. In the following, the method of creating the three-dimensional composite image will be described in detail with reference to FIGS. 3 and 4. FIG.

Here, a case will be described in which the 3D image G0 shown in FIG. 4 is obtained by imaging the individual waste products 2 conveyed on the belt conveyor 5 shown in FIG. 1 with the 3D camera 8 . It should be noted that the position and orientation of the waste product 2 in the image are not constant, and can take various states.

Since each pixel constituting the 3D image G0 represents the height level of the object waste product 2, the control device 13 determines that the pixel value is higher than the pixel value of the surface of the belt conveyor 5 forming the background of the waste product 2. The volume value of the waste product 2 is calculated by extracting large pixels, calculating the sum, and converting to the actual size.

Subsequently, the control device 13 calculates the density value by dividing the weight value measured by the belt conveyor type weighing scale 4 by the volume value, and converts the density value into a relative pixel value at the pixel level. Here, the relative pixel value is, for example, a pixel level directly proportional to the magnitude of the density value such that the maximum value of the density value corresponds to the maximum value of the pixel level.

Then, the control device 13 creates a relative pixel value-embedded background image G3 shown in FIG.

Also, in step S50 shown in FIG. 3, the control device 13 creates a copy image G2 of the 3D image G0 as shown in FIG. 4 and stores it in the memory.

Further, the control device 13 converts the 3D image G0 into the binarized image G1 shown in FIG. Detect the contours of (here, the waste product).

Subsequently, the control device 13 extracts the rectangle having the smallest area from among the rectangles surrounding the outline of the object detected in step S54, and detects the vertex coordinates of the four corners and the rotation angle. At this time, as shown in the binary image G1 of FIG. 4, the rotation angle is the angle when the rectangle rotates so that the long side of the rectangle is parallel to the vertical direction of the image, which is the movement direction of the belt conveyor 5. FIG. Specifically, for example, if the side 34 connecting the vertex 3 and the vertex 4 is a short side, the rotation angle is the angle θ in the figure, and if the side 34 is a long side, the rotation angle is the angle (θ+90°).

Next, the control device 13 uses the vertex coordinates previously detected in step S55 to cut out the range of the minimum area rectangle from the copy image G2. In FIG. 3, the dashed arrow drawn between steps S54 and S55 indicates that the vertex coordinates detected in step S54 are used in step S55.

Subsequently, in step S56, the control device 13 reads the relative pixel value-embedded background image G3 from the memory. A 3D image in which the background is converted and the image of the object is arranged in a fixed direction and fixed position (hereinafter referred to as "background conversion, fixed direction, fixed position, 3D image") is pasted at a predetermined position of the embedded background image G3 .) Create G4. In FIG. 3, the dashed arrow drawn between steps S54 and S57 indicates that the rotation angle detected in step S54 is used in step S57.

Here, the above-mentioned "predetermined position" means that the center line of the background image G3 with embedded relative pixel values and the center line of the clipped image after rotation match, and the The upper side of the clipped image is located at a position lower than the upper side of the relative pixel value-embedded background image G3 by a constant distance d.

Next, in step S6 shown in FIG. 2, each pixel in the two-dimensional composite image obtained by attaching the two-dimensional image of the sample to the second background image is added to each pixel at the same position as each pixel. Image data obtained by adding a parameter corresponding to the pixel value of each pixel of the three-dimensional composite image is created.

Here, FIG. 5 is a flow chart showing a specific example of the method of creating the two-dimensional composite image shown in FIG. FIG. 6 is a diagram for explaining the creation method shown in FIG. In the following, the method of creating the two-dimensional composite image will be described in detail with reference to FIGS. 5 and 6. FIG.

First, a 2D color image G10 photographing the waste products imaged in the 3D image G0 with the same image size and scale, and a background image G13 photographing the surface of the belt conveyor 5 where no waste products are present are prepared.

The control device 13 creates a copy image G12 shown in FIG. 6 in step S60 in the same manner as in the method shown in FIG. G11 is created, and steps S62 to S68 are executed to create a 2D color image in which the image of the object is arranged in a fixed direction and a fixed position in the background image G13 (hereinafter referred to as a “fixed direction/fixed position/2D color image”). ) Create G14. In FIG. 5, the dashed arrow between steps S65 and S66 indicates that the vertex coordinates detected in step S65 are used in step S66, and the arrow between steps S65 and S68 is used. A dashed arrow indicates that the rotation angle detected in step S65 is used in step S68.

FIG. 7 is a flow chart showing a specific example of a method for creating the image data shown in FIG. Also, FIG. 8 is a diagram for explaining the creation method shown in FIG. The method for creating the image data will be described in detail below with reference to FIGS. 7 and 8. FIG.

As shown in FIG. 7, the control device 13 uses the weight value measured by the belt conveyor type weighing scale 4 as described above and the 3D image captured by the 3D camera 8 to obtain the data shown in FIG. The method creates a background transformation, oriented, fixed position, and 3D image. On the other hand, the control device 13 uses the 2D color image captured by the 2D camera 10 to create a fixed-direction, fixed-position, 2D color image by the method shown in FIG.

Here, the background conversion/unidirectional/fixed-position/3D image and the fixed-direction/fixed-position/2D color image have the same image size with the same number of vertical and horizontal pixels. are affixed in the same position and in the same direction.

In addition, the background conversion, fixed orientation, fixed position, 3D image is a grayscale image, and one pixel is represented by one component. On the other hand, unidirectional, fixed position, 2D color images are RGB color images using three color components of red, green, and blue (or CMYK color images using four color components of cyan, magenta, yellow, and black). , and one pixel is represented by three (or four) components.

Here, in step S69, the control device 13 causes the fixed-orientation/fixed-position/2D color image G14 shown in FIG. ) In order to obtain a channel array image, each pixel value of the background conversion, fixed direction, fixed position, 3D image G4 is changed to the fixed direction, fixed position, and the 4th (or 5th) channel of the pixel at the same position in the 2D color image G14. A 4 (or 5) channel image (hereinafter referred to as "background conversion/fixed direction/fixed position/4ch image") G15 is created by adding parameters to G15.

The fourth (or fifth) component parameter other than the above color information is a value indicating the transparency (or opacity) of each pixel. is a value corresponding to the three-dimensional shape (height) of the waste product 2, and to the density of the waste product 2 in pixels where the background is shown.

Therefore, according to the background conversion/fixed-direction/fixed-position/4ch image G15, it is possible to provide more information about the features of individual waste products 2 as one image data than a normal 2D color image.

Next, in step S7 shown in FIG. 2, the convolutional neural network model is made to learn the image data as learning data. Specifically, for example, a certain number of samples are extracted from a set of waste products 2 to be identified, product information indicating the characteristics of the waste product 2 such as item, model, manufacturer, etc. is collected, and the waste product 2 A database is created in association with the background conversion/fixed-direction/fixed-position/4ch image (hereinafter abbreviated as "4ch image") created for each. At this time, after classifying the waste product 2 and its 4ch image into a plurality of groups according to the above characteristics according to the purpose of identification, a deep convolutional neural network model consisting of the group name and group number for each 4ch image Create an image dataset that defines teacher labels for learning by , and train a deep convolutional neural network model.

This deep convolutional neural network model may be created independently, but an existing model that is open to the public can be diverted.

Next, in step S8 shown in FIG. 2, the image data obtained by executing steps S1 to S6 for the object to be identified is applied to the trained convolutional neural network model obtained in step S7. Enter to identify the object. In the above example, inputting the 4ch image created for the waste product 2 with unknown properties, i.e., the group to which it belongs, into the trained deep convolutional neural network model with optimized internal parameters. , the group to which the waste product 2 belongs is identified, and the waste product 2 is sorted and collected in the collection containers 14 to 16 according to the identification result.

Note that the above identification can also be realized by the following method. An intermediate layer output vector (first output vector) obtained by inputting the image data created through steps S1 to S6 shown in FIG. 2 for a sample body whose characteristics are known to the convolutional neural network model are recorded in correspondence with the above characteristics.

Then, in step S8 shown in FIG. 2, the intermediate layer output vector (second output The property of the object is identified by identifying, from among the first output vectors recorded in the list, the first output vector that provides the shortest distance to the object vector.

As described above, according to the object identification device 1 and the object identification method according to the embodiment of the present invention, by using 4ch images, an object identification device capable of highly accurate object identification with less learning data than before. and object identification methods can be obtained.

1 object identification device 4 belt conveyor type weight scale 8 3D camera 10 2D camera 13 control device

Claims (10)

creating a first background image corresponding to the density of the sample body, and pasting the three-dimensional image of the sample body onto the first background image to create a three-dimensional composite image;
For each pixel in the two-dimensional composite image obtained by attaching the two-dimensional image of the sample to the second background image, according to the pixel value of each pixel of the three-dimensional composite image at the same position as each pixel Create image data obtained by adding the parameters
An object identification device comprising identification means for identifying an object by means of a convolutional neural network model trained on the image data. weight measuring means for measuring the weight of the sample body;
Further comprising 3D imaging means for imaging the three-dimensional image,
The identification means further calculates the volume of the sample body from the three-dimensional image captured by the 3D imaging means, and calculates the volume of the sample body from the calculated volume and the weight measured by the weight measurement means. 2. The object identification device according to claim 1, wherein said density is calculated. 3. The object identification device according to claim 2, further comprising 2D imaging means for imaging said two-dimensional image. The identifying means pastes the three-dimensional image in a predetermined direction and position with respect to the first background image, and attaches the two-dimensional image in a predetermined direction and position with respect to the second background image. 2. The object identification device according to claim 1, which is attached to the . 2. The object identification device according to claim 1, wherein said two-dimensional image is an RGB color image or a CMYK color image. a first step of weighing the sample body;
a second step of calculating the volume of the sample body from the three-dimensional image of the sample body;
a third step of calculating the density of the sample body using the weight measured in the first step and the volume calculated in the second step;
a fourth step of creating a first background image according to the density calculated in the third step;
a fifth step of pasting the three-dimensional image of the sample body onto the first background image to create a three-dimensional composite image;
For each pixel in the two-dimensional composite image obtained by attaching the two-dimensional image of the sample to the second background image, according to the pixel value of each pixel of the three-dimensional composite image at the same position as each pixel a sixth step of creating the resulting image data by adding the parameters
a seventh step of training a convolutional neural network model with the image data as training data;
The image data obtained by executing the first to sixth steps for an object to be identified is input to the trained convolutional neural network model obtained in the seventh step to identify the object and an eighth step of identifying the object identification method. pasting the three-dimensional image in a predetermined direction and position with respect to the first background image;
7. The object identification method according to claim 6, wherein said two-dimensional image is pasted in a predetermined direction and position with respect to said second background image. 7. The object identification method according to claim 6, wherein said two-dimensional image is an RGB color image or a CMYK color image. a step of preliminarily classifying the sample bodies into a plurality of groups according to their characteristics;
further comprising the step of creating a database whose elements are the image data created for each of the groups through the first step to the sixth step;
In the seventh step, training the convolutional neural network model on the database;
7. The object identification method according to claim 6, wherein in said eighth step, said group to which said object belongs is identified. The first output vector of the intermediate layer obtained by inputting the image data created through the first step to the sixth step for the sample body whose characteristics are known to the convolutional neural network model to the further comprising pre-creating a list of records associated with the characteristics;
In the eighth step, a second output vector of the intermediate layer obtained by inputting the image data created through the first step to the sixth step for the object into the convolutional neural network model 7. The object identification according to claim 6, wherein the characteristic of the object is identified by identifying the first output vector that is the shortest distance from the first output vector recorded in the list. Method.

Images