JP2021002259A - Object identification device and object identification method - Google Patents

Abstract

To provide an object identification device and an object identification method capable of accurately identifying an object with less learning data.SOLUTION: The object identification device 1 includes a controller 13, which creates a first background image according to a density of a sample body and pastes a three-dimensional image of the sample body on the first background image to create a three-dimensional composite image. Further, for each pixel in a two-dimensional composite image obtained by pasting a two-dimensional image of the sample body on a second background image, the controller creates image data obtained by parameter addition according to a pixel value of each pixel of the three-dimensional composite image at a same position as the above-mentioned each pixel, and identifies an object by a convolutional neural network model trained with the above image data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for identifying an object in an object recycling process or the like.

In recent years, resource saving has become one of the important social themes, and a technology for sorting waste products has been devised in order to realize recycling (see Patent Document 1).

JP-A-2017-109161

Patent Document 1 discloses a waste sorting system using machine learning by a convolutional neural network, but in order to perform highly accurate sorting, a large amount of image data is collected and trained by the machine learning unit 43. There is a need.

The present invention has been made to solve such a problem, and an object of the present invention is to provide an object identification device and an object identification method capable of accurately identifying an object with less training data.

In order to solve the above problems, the present invention creates a first background image according to the density of the sample body, and attaches a three-dimensional image of the sample body to the first background image to create a three-dimensional composite image. Then, each pixel in the two-dimensional composite image obtained by pasting the two-dimensional image of the sample body on the second background image corresponds to the pixel value of each pixel in the three-dimensional composite image at the same position as the above pixels. An object identification device provided with an identification means for identifying an object by a convolutional neural network model trained with the image data is provided by creating image data obtained by adding the above parameters.

Further, in order to solve the above problems, the present invention has a first step of measuring the weight of the sample body, a second step of calculating the volume of the sample body from the three-dimensional image of the sample body, and a first step. Create a third step of calculating the density of the sample body using the weight measured in step 3 and the volume calculated in the second step, and the first background image according to the density calculated in the third step. The fourth step to create a three-dimensional composite image by pasting the three-dimensional image of the sample body on the first background image, and the second step of pasting the two-dimensional image of the sample body on the second background image. The sixth is to create image data obtained by adding parameters corresponding to the pixel values of each pixel of the three-dimensional composite image at the same position as each pixel to each pixel in the two-dimensional composite image obtained. , The seventh step of training the convolution neural network model with the image data as training data, and the image data obtained by executing the first to sixth steps for the object to be identified. Provided is an object identification method having an eighth step of inputting into the trained convolutional neural network model obtained in the seventh step to identify an object.

According to the present invention, it is possible to obtain an object identification device and an object identification method capable of accurately identifying an object with less learning data.

It is a figure which shows the whole structure of the object identification apparatus 1 which concerns on embodiment of this invention. It is a flowchart which shows the object identification method which concerns on embodiment of this invention. It is a flowchart which shows the specific example of the method of creating a three-dimensional composite image shown in FIG. It is a figure for demonstrating the production method shown in FIG. It is a flowchart which shows the specific example of the method of creating a two-dimensional composite image shown in FIG. It is a figure for demonstrating the production method shown in FIG. It is a flowchart which shows the specific example of the image data creation method shown in FIG. It is a figure for demonstrating the production method shown in FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the figure, the same reference numerals indicate the same or corresponding parts.

FIG. 1 is a diagram showing an overall configuration of an object identification device 1 according to an embodiment of the present invention. As shown in FIG. 1, the object identification device 1 according to the embodiment of the present invention is a belt conveyor type scale 4 that measures the weight of the waste product 2 while transporting the waste product 2 supplied from the supply device 3. , A belt conveyor 5 that conveys the waste product 2, a photosensor 6 for a three-dimensional (3D) camera, a linear laser 7 for a 3D camera, a 3D camera 8, a photosensor 9 for a 2D camera, and a 2D camera 10. It is provided with the sorting actuators 11 and 12 for separating and collecting the waste products 2 into the collection containers 14 to 16, and the control device 13 for controlling the operations of these and identifying the waste products 2 by a method described later.

When it is necessary to increase the amount of identification per unit time, it is preferable to arrange, for example, a plurality of object identification devices 1 shown in FIG. 1 in parallel and operate them at the same time.

The outline of the operation of the object identification device 1 will be described below. The weight of the waste product 2 individually supplied from the supply device 3 as a sample body is measured by the belt conveyor type weight scale 4, and the data indicating the result is transmitted to the control device 13. Subsequently, when the scrap product 2 is detected by the photo sensor 6, the 3D camera 8 suspended upward operates in response to the detection signal to capture a 3D image of the scrap product 2 and transmit it to the control device 13. Will be done.

Subsequently, when the scrap product 2 is detected by the photo sensor 9, the 2D camera 10 suspended upward operates in response to the detection signal, and the 2D color image of the scrap product 2 is captured and transmitted to the control device 13. To.

The control device 13 identifies the waste product 2 by performing arithmetic processing based on the weight of the waste product 2 acquired as described above, the 3D image, and the 2D image, but this arithmetic processing will be described in detail later. ..

The identified waste products 2 are stored in the corresponding collection containers 14 to 16 in which the drop position from the belt conveyor 5 is controlled by the sorting actuators 11 and 12 according to the identification result.

In FIG. 1, a cylinder that can control the on / off of the pushing operation is used as the sorting mechanism, but the present invention is not limited to this, and for example, a paddle operated by an air gun or an electromagnet, a robot arm, or the like is used. There may be.

In the above, the waste product 2 is transferred from the supply device 3 to the belt conveyor 5 in a random state in that the position in the width direction of the belt conveyor 5 and the angle formed by the traveling direction of the belt and the long side of the waste product 2 are not constant. Fall.

Further, the 3D camera 8 changes the position of the reflected light line of the linear laser light on the surface of the waste product 2 moving in a certain direction in the height direction by a charge coupling element (CCD) or a complementary metal oxide film semiconductor (CMOS). A 3D image is recorded as digital data in a memory (not shown) by a so-called optical cutting method detected by.

FIG. 2 is a flowchart showing an object identification method according to an embodiment of the present invention. In the following, a case where the object identification method shown in FIG. 2 is realized by using the object identification device 1 shown in FIG. 1 as an example will be described.

Needless to say, this object identification method is not limited to the case where it is realized by using the object identification device 1 shown in FIG. 1, and can be widely applied.

First, in step S1, the weight of the sample body is measured. Specifically, for example, the weight of the sample body is measured by the belt conveyor type weight scale 4 as described above, and the obtained data is transmitted to the control device 13.

Next, in step S2, the volume of the sample body is calculated from the three-dimensional image of the sample body. Specifically, for example, the control device 13 calculates the volume of the waste product 2 based on the 3D image received by the method as described above.

Next, in step S3, for example, the control device 13 calculates the density of the waste product 2 as the sample body by using the weight of the sample body measured in step S1 and the volume of the sample body calculated in step S2. ..

Next, in step S4, for example, the control device 13 creates a first background image according to the density calculated in step S3.

Next, in step S5, for example, the control device 13 attaches a three-dimensional image of the scrap product 2 as a sample body to the first background image to create a three-dimensional composite image.

FIG. 3 is a flowchart showing a specific example of the method of creating the three-dimensional composite image shown in FIG. Further, FIG. 4 is a diagram for explaining the production method shown in FIG. In the following, the method of creating the three-dimensional composite image will be described in detail with reference to FIGS. 3 and 4.

Here, the case where the 3D image G0 shown in FIG. 4 is obtained by photographing the individual waste products 2 conveyed on the belt conveyor 5 shown in FIG. 1 with the 3D camera 8 will be described. It should be noted that the position and orientation of the waste product 2 in the image is not constant, and various states can be taken.

Since each pixel constituting the 3D image G0 represents the height level of the waste product 2 which is the subject, the control device 13 has a numerical value larger than the pixel value on the surface of the belt conveyor 5 forming the background of the waste product 2. The volume value of the waste product 2 is calculated by extracting large pixels, calculating the total sum, and converting it to the actual size.

Subsequently, the control device 13 calculates the density value by dividing the weight value measured by the belt conveyor type weight scale 4 by the volume value, and converts this density value into a relative pixel value at the pixel level. Here, the relative pixel value is, for example, a pixel level that is directly proportional to the magnitude of the density value so that the maximum value of the density value corresponds to the maximum value of the pixel level.

Then, the control device 13 creates the relative pixel value embedded background image G3 shown in FIG. 4 with the relative pixel value as the background, and saves it in the memory.

Further, in step S50 shown in FIG. 3, the control device 13 creates a copy image G2 of the 3D image G0 as shown in FIG. 4 and stores it in the memory.

Further, the control device 13 converts the 3D image G0 into the binarized image G1 shown in FIG. 4 in step S51, removes noise by image processing such as expansion and contraction in step S52, and then an object in step S53. Detects the contour of (here, scrap product).

Subsequently, the control device 13 extracts the rectangle surrounding the outline of the object detected in step S54 and having the smallest area, and detects the coordinates of the vertices of the four corners and the angle of rotation. At this time, the rotation angle is an angle when the rectangle is rotated so that the long side of the rectangle is parallel to the vertical direction of the image, which is the moving direction of the belt conveyor 5, as shown in the binarized image G1 of FIG. Specifically, for example, when the side 34 connecting the apex 3 and the apex 4 is a short side, the rotation angle is the angle θ in the figure, and when the side 34 is the long side, the rotation angle is the angle (θ + 90 °).

Next, the control device 13 cuts out the range of the minimum area rectangle from the copy image G2 by using the vertex coordinates detected earlier in step S55. The broken line arrow drawn between step S54 and step S55 in FIG. 3 indicates that the vertex coordinates detected in step S54 in step S55 are used.

Subsequently, the control device 13 reads the relative pixel value embedded background image G3 from the memory in step S56, and in step S57, rotates the cropped image by the previous rotation angle to obtain the relative pixel value. Embedded background image A 3D image (hereinafter referred to as "background conversion / fixed direction / fixed position / 3D image"" in which an image of the object is arranged in a fixed direction and a fixed position by pasting it at a predetermined position on the G3. .) Create G4. The broken line arrow drawn between step S54 and step S57 in FIG. 3 indicates that the rotation angle detected in step S54 in step S57 is used.

Here, the "predetermined position" means, for example, as shown in FIG. 4, that the center line of the relative pixel value embedded background image G3 and the center line of the cut-out image after rotation coincide with each other and after rotation. The upper side of the cropped image is set to be a position lowered by a certain distance d from the upper side of the relative pixel value embedded background image G3.

Next, in step S6 shown in FIG. 2, each pixel in the two-dimensional composite image obtained by pasting the two-dimensional image of the sample body on the second background image is at the same position as each pixel. Image data obtained by adding parameters corresponding to the pixel values of each pixel of the three-dimensional composite image is created.

Here, FIG. 5 is a flowchart showing a specific example of the method for creating the two-dimensional composite image shown in FIG. Further, FIG. 6 is a diagram for explaining the production method shown in FIG. In the following, the method of creating the two-dimensional composite image will be described in detail with reference to FIGS. 5 and 6.

First, a 2D color image G10 obtained by photographing the waste product captured in the 3D image G0 at the same image size and scale, and a background image G13 obtained by photographing the surface of the belt conveyor 5 in which the waste product does not exist are prepared.

The control device 13 creates the copy image G12 shown in FIG. 6 in step S60, converts the 2D color image G10 into a gray image in step S61, and then binarizes the image, in the same manner as the method shown in FIG. A 2D color image in which an image of the object is arranged in a fixed direction and a fixed position in the background image G13 by creating G11 and executing steps S62 to S68 (hereinafter, "fixed direction / fixed position / 2D color image"". ) Create G14. Note that the broken line arrow marked between steps S65 and S66 in FIG. 5 indicates that the vertex coordinates detected in step S65 in step S66 are used, and are marked between steps S65 and S68. The dashed arrow indicates that the rotation angle detected in step S65 is used in step S68.

FIG. 7 is a flowchart showing a specific example of the method of creating the image data shown in FIG. Further, FIG. 8 is a diagram for explaining the production method shown in FIG. 7. In the following, the method of creating the above image data will be described in detail with reference to FIGS. 7 and 8.

As shown in FIG. 7, the control device 13 is shown in FIG. 3 using the weight value measured by the belt conveyor type weight scale 4 as described above and the 3D image captured by the 3D camera 8. Background conversion, fixed direction, fixed position, and 3D image are created by the method. On the other hand, the control device 13 creates a fixed direction / fixed position / 2D color image by the method shown in FIG. 5 using the 2D color image captured by the 2D camera 10.

Here, the background conversion / fixed direction / fixed position / 3D image and the fixed direction / fixed position / 2D color image have the same image size with the same number of vertical and horizontal pixels, and the same surface of the same scrap product 2 is the background image. It is said that it was attached to the same position and in the same direction.

Further, the background conversion, the fixed direction, the fixed position, and the 3D image are grayscale images, and one pixel is represented by one component. On the other hand, the fixed-direction, fixed-position, and 2D color images are RGB color images that use the three color components of red, green, and blue (or CMYK color images that use the four color components of cyan, magenta, yellow, and black). Yes, one pixel is represented by three (or four) components.

Here, in step S69, the control device 13 has 4 (or 5) components in which the fixed direction / fixed position / 2D color image G14 shown in FIG. 8 has four (or five) components per pixel. ) The 4th (or 5th) channel of the pixels at the same position in the fixed direction, fixed position, and 2D color image G14 by setting each pixel value of the background conversion, fixed direction, fixed position, and 3D image G4 so that the image has a channel arrangement. A 4 (or 5) channel image (hereinafter referred to as "background conversion / fixed direction / fixed position / 4ch image") G15 is created by performing a process of adding a parameter to.

The parameter of the 4th (or 5th) component other than the color information as described above is a value indicating the transparency (or opacity) of each pixel. Specifically, in the pixel in which the disused product 2 is shown. Is a value according to the three-dimensional shape (height) of the waste product 2 and according to the density of the waste product 2 in the pixels showing the background.

Therefore, according to the background conversion / fixed direction / fixed position / 4ch image G15, it is possible to provide more information as one image data than a normal 2D color image about the features of each scrap product 2.

Next, in step S7 shown in FIG. 2, the convolutional neural network model is trained by using the image data as training data. Specifically, for example, a certain number of samples are extracted from a set of waste products 2 to be identified, product information indicating the characteristics of the waste products 2 such as items, models, and manufacturers is collected, and the waste products 2 are collected. Create a database associated with the above-mentioned background conversion, fixed direction, fixed position, and 4ch image (hereinafter, abbreviated as "4ch image") created for the above. At this time, after classifying the scrap product 2 and its 4ch image into a plurality of groups according to the purpose of identification, a deep convolutional neural network model consisting of the group name, group number, etc. for each 4ch image. Create an image dataset that defines a teacher label for training by, and train it in a deep convolutional neural network model.

The deep convolutional neural network model may be created independently, but an existing model that is open to the public can be used.

Next, in step S8 shown in FIG. 2, the image data obtained by executing steps S1 to S6 for the object to be identified is transferred to the trained convolutional neural network model obtained in step S7. Input to identify the object. In the above example, the above-mentioned characteristic, that is, the above-mentioned group to which the above-mentioned group belongs, inputs the 4ch image created for the unknown waste product 2 into the above-mentioned trained deep convolutional neural network model having the optimized internal parameters. As a result, which group the waste product 2 belongs to is identified, and the waste products 2 are separately collected in the collection containers 14 to 16 according to the identification result.

The above identification can also be realized by the following method. The output vector of the intermediate layer (first output vector) obtained by inputting the image data created through steps S1 to S6 shown in FIG. 2 into the convolutional neural network model for the sample body having the known characteristics. Is recorded in advance in association with the above characteristics.

Then, in step S8 shown in FIG. 2, the output vector (second output) of the intermediate layer obtained by inputting the image data created for the object to be identified into the trained convolutional neural network model. The characteristics of the object are identified by specifying the first output vector, which is the shortest distance to the vector), from the first output vectors recorded in the list.

From the above, according to the object identification device 1 and the object identification method according to the embodiment of the present invention, the object identification device capable of highly accurate object identification with less learning data than the conventional one by using the 4ch image. And an object identification method can be obtained.

1 Object identification device 4 Belt conveyor type weighing scale 8 3D camera 10 2D camera 13 Control device

Claims (10)

A first background image corresponding to the density of the sample body is created, and a three-dimensional image of the sample body is attached to the first background image to create a three-dimensional composite image.
Each pixel in the two-dimensional composite image obtained by pasting the two-dimensional image of the sample body on the second background image corresponds to the pixel value of each pixel of the three-dimensional composite image at the same position as each pixel. Create the image data obtained by adding the above parameters
An object identification device provided with an identification means for identifying an object by a convolutional neural network model trained with the image data. A weight measuring means for measuring the weight of the sample body and
Further provided with a 3D imaging means for capturing the three-dimensional image,
The identification means further calculates the volume of the sample body from the three-dimensional image captured by the 3D imaging means, and from the calculated volume and the weight measured by the weight measuring means, the sample body The object identification device according to claim 1, which calculates the density. The object identification device according to claim 2, further comprising a 2D imaging means for capturing the two-dimensional image. The identification means attaches the three-dimensional image to the first background image in a predetermined direction and position, and attaches the two-dimensional image to the second background image in a predetermined direction and position. The object identification device according to claim 1, which is attached to the above. The object identification device according to claim 1, wherein the two-dimensional image is an RGB color image or a CMYK color image. The first step in measuring the weight of the sample body,
The second step of calculating the volume of the sample body from the three-dimensional image of the sample body, and
A third step of calculating the density of the sample body using the weight measured in the first step and the volume calculated in the second step,
A fourth step of creating a first background image according to the density calculated in the third step, and
A fifth step of attaching a three-dimensional image of the sample body to the first background image to create a three-dimensional composite image, and
Each pixel in the two-dimensional composite image obtained by pasting the two-dimensional image of the sample body on the second background image corresponds to the pixel value of each pixel of the three-dimensional composite image at the same position as each pixel. The sixth step of creating the image data obtained by adding the above parameters,
The seventh step of training the convolutional neural network model using the image data as training data,
The image data obtained by executing the first to sixth steps for the object to be identified is input to the trained convolutional neural network model obtained in the seventh step to input the object. An object identification method having an eighth step of identifying the object. The three-dimensional image is attached to the first background image in a predetermined direction and position.
The object identification method according to claim 6, wherein the two-dimensional image is attached to the second background image in a predetermined direction and position. The object identification method according to claim 6, wherein the two-dimensional image is an RGB color image or a CMYK color image. A step of preliminarily classifying the sample body into a plurality of groups according to the characteristics, and
Further having a step of creating a database having the image data as an element created for each group through the first step to the sixth step.
In the seventh step, the database is trained by the convolutional neural network model.
The object identification method according to claim 6, wherein in the eighth step, the group to which the object belongs is identified. The first output vector of the intermediate layer obtained by inputting the image data created from the first step to the sixth step of the sample body having known characteristics into the convolutional neural network model is obtained. It also has the step of creating a pre-recorded list that corresponds to the characteristics.
In the eighth step, the image data created from the first step to the sixth step for the object is input to the convolutional neural network model and used as the second output vector of the intermediate layer. The object identification according to claim 6, wherein the characteristic of the object is identified by specifying the first output vector having the shortest distance from the first output vector recorded in the list. Method.

Images