JP2021068293A - Object identification device - Google Patents

Abstract

To provide a technique capable of easily and individually identifying a plurality of objects (including a plurality of objects having the same shape) arranged in a target area.SOLUTION: Imaging means 10 outputs imaging information indicating an imaging screen that captures an imaging area including a target area and a plurality of objects (including a plurality of objects having the same shape) arranged in the target area. Target area determination means 23 determines the target area based on the imaging information. Object detection means 24 detects categories and positions of the plurality of objects arranged in the determined target area. Object arrangement information generation means 25 generates identification information of the plurality of objects and object arrangement information indicating their positions based on the categories and positions of the plurality of objects arranged in the target area detected by the object detection means 24 and stores the generated information in storage means 30. Object identification means 27 individually identifies the plurality of objects arranged in the target area based on the categories and positions of the plurality of objects arranged in the target area detected by the object detection means 24 and the object arrangement information stored in the storage means 30.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object identification device that identifies an object based on imaging information, particularly an object identification device that individually identifies a plurality of objects (including a plurality of objects having the same shape) arranged in a target area.

Technology for detecting objects using deep learning (machine learning method using multi-layer neural networks) is being researched. For example, there are many end-to-end methods such as SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) that detect the position and category of the target object at the same time based on the imaging information. Proposed. These methods are based on multi-task learning in which learning by a multi-layer neural network for detecting the position of an object and learning by a multi-layer neural network for discriminating an object category are performed at the same time.
The object detection technology by SSD is disclosed in Non-Patent Document 1, for example, and the object detection technology by YOLO is disclosed in Non-Patent Document 2, for example.

“SSD: Single Shot MulitiBox Detector”, Wei Liu, Dragomir Anguelov. Domitru Erhan, Christian Szegedy, Scott reed, Cheng-Yang Fu and Alexsander C. berg (2015), https://arxiv.org/pdf/1512.02325.pdf “You Only Look Once Unified, Real-Time Object Detection”, Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi (2016), https://pjreddie,com/media/files/papers/yolo.pdf

The object detection technology disclosed in Non-Patent Document 1 and Non-Patent Document 2 can detect the positions and categories of a plurality of objects arranged in the target area, but a plurality of objects arranged in the target area. It is not possible to identify an object, particularly a plurality of objects having the same shape (including "substantially the same shape").
Therefore, conventionally, arrangement information (positions and IDs of a plurality of objects) indicating the arrangement state of a plurality of objects arranged in the target area is created (registered) as a two-dimensional map and output from the imaging means. The object detection information (positions and categories of multiple objects arranged in the target area in the imaging area) detected based on the imaging information and the arrangement information (positions and IDs of multiple objects) shown in the two-dimensional map are displayed. By collating, a plurality of objects were individually identified.
However, it is necessary to manually create a two-dimensional map every time an object is registered, and it takes labor and time to create the two-dimensional map.
The present invention has been devised in view of these points, and is a technique capable of easily identifying a plurality of objects (including a plurality of objects having the same shape) arranged in a target area. The purpose is to provide.

The object identification device of the present invention is used to identify a plurality of objects arranged in a predetermined target area, particularly a plurality of objects including a plurality of objects having the same shape (including "substantially the same shape"). Be done.
The present invention includes an image pickup means, a target area determination means, an object detection means, an object arrangement information generation means, an object identification means, and a storage means.
The imaging means outputs imaging information indicating an imaging screen that captures an imaging region including a target region and a plurality of objects (including a plurality of objects having the same shape) arranged in the target region. As the imaging means, a digital camera using a CCD or CMOS is preferably used.
The target area determining means discriminates the target area on the imaging screen based on the imaging information from the imaging means.
The object detection means detects the categories and positions of a plurality of objects arranged in the target area on the image pickup screen based on the image pickup information from the image pickup means and the target area on the image pickup screen determined by the target area determination means. To do. Preferably, the object detection means outputs object detection information indicating a category of a plurality of objects arranged in the target area and a position in the target area. Various known methods can be used as a method for discriminating the target area by the target area discriminating means and a method for detecting the category and position of the object by the object detecting means. For example, a detection method by deep learning such as SSD or YOLO can be used.
The object arrangement information generation means determines the arrangement state of a plurality of objects in the target area based on the categories and positions (object detection information) of the plurality of objects arranged in the target area on the imaging screen detected by the object detection means. The object arrangement information to be shown is generated and stored in the storage means. The object placement information includes identification information (ID) for identifying the object and position information indicating the position of the object in the target area. The identification information (ID) is given to each object by an appropriate method. Preferably, identification information (ID) including categorical information indicating the category of the object is used.
The object identification means includes object detection information output from the object detection means (categories and positions of a plurality of objects arranged in the target area on the imaging screen) and object arrangement information stored in the storage means (in the target area). Based on the identification information and position of the plurality of arranged objects), the plurality of objects arranged in the target area on the imaging screen are individually identified. For example, the correspondence between a plurality of objects arranged in the target area on the imaging screen and a plurality of objects indicated by the object arrangement information is determined.
The present invention is configured to be configurable in the object arrangement information generation mode and the object identification mode. When the object arrangement information generation mode is set, the object arrangement information is generated by the target area determination means, the object detection means, and the object arrangement information generation means and stored in the storage means. When the object identification mode is set, a plurality of objects arranged in the target area on the imaging screen are individually identified by the target area discrimination means, the object detection means, and the object identification means. Various methods can be used as a method for configuring the object arrangement information generation mode or the object identification mode so as to be settable. For example, the object arrangement information generation mode is set by inputting the object arrangement information generation mode setting information from the input means or the like, and the object identification mode is set by inputting the object identification mode setting information from the input means or the like. It can be configured as follows. Alternatively, by inputting the object identification start information from the input means or the like, the object arrangement information generation mode can be set first, and then the object identification mode can be set.
In the present invention, in the object arrangement information generation mode, object arrangement information indicating the arrangement state of a plurality of objects in the target area is generated, and in the object identification mode, a plurality of objects arranged in the target area with reference to the object arrangement information. Since the objects are individually identified, it is possible to easily identify a plurality of objects arranged in the target area.
Different embodiments of the present invention include means for determining the amount of change in the angle of view. The angle of view change amount discriminating means includes an imaging screen (M) used when generating object arrangement information stored in the storage means and an imaging screen (P) indicated by imaging information output from the imaging means. It is determined whether or not the amount of change in the angle of view between them exceeds a predetermined range. As a method for determining whether or not the amount of change in the angle of view between the imaging screen (M) and the imaging screen (P) exceeds a predetermined range, the imaging screen (M) and the imaging screen (P) are preferably used. It is possible to use a method of determining whether or not the similarity of is low. For example, the feature points of the imaging screen (M) and the imaging screen (P) are detected, and among the feature points of both screens, the similarity of the feature vectors including the feature points having the same correspondence (for example, the cosine distance of both vectors). ) Is greater than or equal to the predetermined value, it is determined that the amount of change in the angle of view does not exceed the predetermined range, and if it is less than the predetermined value, it is determined that the amount of change in the angle of view exceeds the predetermined range. Of course, other methods can be used as a method for determining whether or not the amount of change in the angle of view exceeds a predetermined range. The amount of change in the angle of view between the target area (M) on the image pickup screen (M) and the target area (P) on the image pickup screen (P) is between the image pickup screen (M) and the image pickup screen (P). It is equivalent to the amount of change in the angle of view.
Then, when it is determined by the angle of view change amount discriminating means that the angle of view change amount exceeds a predetermined range when the object identification mode is set, the target area discriminating means, the object detecting means, and the object arrangement information generation The object arrangement information is generated by the means and stored in the storage means.
When the imaging position or imaging angle (imaging direction) of the imaging means is changed, the arrangement state of a plurality of objects on the imaging screen and the imaging screen used when generating the object arrangement information stored in the storage means are displayed. The deviation from the arrangement state of a plurality of objects (particularly, the position on the imaging screen) becomes large, and the individual identification accuracy of a plurality of objects based on the collation of the object detection information and the object arrangement information by the object identification means may decrease. is there. In this embodiment, the amount of change in the angle of view between the image pickup screen used when generating the object arrangement information stored in the storage means and the image pickup screen indicated by the image pickup information output from the image pickup means is within a predetermined range. When the amount exceeds the limit, the object arrangement information is regenerated and stored in the storage means. Therefore, even if the imaging position or the imaging angle (imaging direction) of the imaging means is changed, a plurality of objects arranged in the target area. Can be reliably identified as an individual.
In a different embodiment of the present invention, the object identification means is indicated by the categories and positions of a plurality of objects arranged in the target area on the imaging screen detected by the object detecting means, and the object arrangement information stored in the storage means. A plurality of objects are individually identified by collating the identification information and positions of the plurality of objects while allowing a predetermined error range. As a method of collating the position of the object, preferably, the coordinates of the points at the four corners of the bounding box (candidate for the existence area of the object estimated by the object detecting means), or the width and height of the bounding box and the position of the center of gravity. A method of collating coordinates is used.
In this embodiment, individual identification of a plurality of objects can be easily performed.
In different embodiments of the present invention, the object detecting means detects the categories of a plurality of objects and their relative positions in the target area, and the object placement information generating means indicates the identification information of the plurality of objects and the object placement indicating the relative positions in the target area. Generate information. As the relative position, for example, the coordinates in the coordinate system set on the imaging screen are used.
The object identification means collates the object detection information with the object arrangement information by using the relative position of the object.
In this embodiment, the collation process of the object detection information and the object arrangement information can be easily performed.
In a different embodiment of the present invention, the target area discriminating means and the object detecting means are configured by a multi-layer neural network.
In this embodiment, it is possible to easily perform the determination process of the target area and the detection process of the categories and positions of a plurality of objects arranged in the target area.

In the present invention, a plurality of objects arranged in the target area can be easily identified as individuals.

It is a block diagram of one Embodiment of the object identification apparatus of this invention. It is a figure which shows the neural network configuration of SSD. It is a figure which shows an example of the captured image. It is a figure which shows the target area in the captured image shown in FIG. It is a figure which shows an example of the object detection result based on the captured image shown in FIG. It is a figure which shows an example of the object detection information based on the captured image shown in FIG. It is a figure which shows an example of the object arrangement information based on the object detection information shown in FIG. It is a figure explaining the operation which identifies a plurality of objects individually based on the object detection information and the object arrangement information. It is a flowchart explaining one Embodiment of the operation in the object arrangement information generation mode. It is a flowchart explaining one Embodiment of operation in an object identification mode. It is a figure which shows the method of discriminating the amount of change of the angle of view by the feature vector of the captured image. It is a flowchart explaining another embodiment of operation in an object identification mode.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram of an embodiment of the object identification device of the present invention.
The object identification device of the present embodiment includes an imaging means 10, a processing means 20, a storage means 30, an input means 40, a display means 50, and the like.

The imaging means 10 is preferably configured by a digital camera using a CCD or CMOS. The imaging means 10 is arranged so that the target region and the imaging region including a plurality of objects arranged in the target region can be imaged. In the following description of the embodiment, it is assumed that the plurality of objects include at least one type of a plurality of objects having the same shape.
In the object detection process described later, the category (type) of the object is determined based on the "shape of the object". The description "same shape" is used to include "substantially the same shape" which is determined to be an object of the same category by the object detection process.

The processing means 20 is composed of a CPU or the like. The processing means 20 includes a management means 21, an imaging information input means 22, a target area discriminating means 23, an object detecting means 24, an object arrangement information generating means 25, an angle of view change amount discriminating means 26, and an object discriminating means 27. .. In the present embodiment, the means 21 to 27 are provided in the CPU constituting the processing means 20. Of course, at least one of the means 21 to 27 may be configured by a CPU different from the CPU constituting the processing means 20.

The processing of each means 21 to 27 will be described with reference to FIGS. 2 to 8.
In the following, as shown in FIG. 3, on the operation panel 110 of the operation panel 100, three volumes 120a to 120c having the same shape, two dials 130a and 130b having the same shape, and 11 eleven having the same shape. A case where the buttons 140a to 140k are arranged will be described.
The operation panel 110 is surrounded by an edge portion 111 formed in a quadrangular shape by an upper edge portion 111a, a left side edge portion 111b, a lower side edge portion 111c, and a right side edge portion 111d.
Reference numeral 11 denotes an imaging screen in which the imaging region including the operation panel 110, volumes 120a to 120c, dials 130a and 130b, and buttons 140a to 140k is imaged by the imaging means 10.
The operation panel 110 corresponds to the "target area" of the present invention, and the edge portion 111 (upper edge portion 111a to right side edge portion 111d) of the operation panel 110 corresponds to the "target area edge portion (target area upper edge portion)" of the present invention. ~ Right edge of target area) ”, and the volumes 120a to 120c, dials 130a, 130b and buttons 140a to 140k correspond to“ a plurality of objects arranged in the target area ”of the present invention, and the volumes 120a to The 120c or dials 130a, 130b or buttons 140a-140k correspond to the "plurality of objects of the same shape" of the present invention. Also, volumes, dials and buttons correspond to the "object categories" of the present invention.

The management means 21 manages the processes of the means 22 to 27.
The image pickup information input means 22 inputs the image pickup information from the image pickup means 10.

The target area determination means 23 determines the target area based on the image pickup information from the image pickup means 10 input by the image pickup information input means 22.

As a method for discriminating the target area by the target area discriminating means 23, various methods can be used. For example, the SSD disclosed in Non-Patent Document 1 can be used.
As shown in FIG. 2, the SSD is based on a multi-layered CNN (Convolutional neural network), and has a layer for estimating the existence area candidate of an object and a category of the object in the existence area candidate. It is composed of a layer to be discriminated. In the layer for estimating the existence area candidate of the object, the image information is divided into a plurality of rectangular areas (default box) of a predetermined size, and the existence area candidate (bounding box) of the object is estimated while considering the deviation of the rectangular area. .. In the layer for determining the category of the object in the existing area candidate, the category of the object in the existing area candidate is determined by using the separately learned CNN.
The SSD used in the target area determining means 23 is composed of a layer for estimating an existing area candidate of a target area in which a plurality of objects are arranged, and a layer for discriminating a target area in the existing area candidate. The target area determination means 23 configured by the SSD determines the target area and its position on the imaging screen in which the imaging area is imaged.
In the present embodiment, as shown in FIG. 4, the operation panel 110 surrounded by the edge portion 111 is determined as the target area 110.

The object detection means 24 is arranged (displayed) in the target area on the image pickup screen based on the image pickup information from the image pickup means 10 input by the image pickup information input means 22 and the target area determined by the target area determination means 23. Detect the categories and positions of multiple objects.
As a method for detecting the category and position of a plurality of objects arranged in the target area by the object detection means 24, various methods can be used. For example, the SSD shown in FIG. 2 can be used as in the target area determination means 23.
The SSD used in the object detection means 24 is composed of a layer for estimating the existence area candidate (bounding box) of each of a plurality of objects in the target area and a layer for detecting the category of the object in each existence area candidate. ..
In the present embodiment, the bounding box shown by the broken line in FIG. 5, the volumes 120a to 120c, the dials 130a and 130b, and the buttons 140a to 140k in each bounding box are detected.

Then, the object detection means 24 outputs object detection information indicating the categories and positions of a plurality of objects arranged in the target area. As the position of the object, preferably, the coordinates of the points at the four corners of the bounding box or the coordinates of the width and height of the bounding box and the position of the center of gravity are used. As the coordinates, preferably, the xy coordinates in the xy coordinate system set in the target area are used. The "xy coordinates in the xy coordinate system set in the target area" correspond to the "relative position in the target area" of the present invention.
In this embodiment, as shown in FIG. 6, three volume Vs, two dials D, and eleven switches S are arranged at the positions of the bounding boxes shown by the broken lines. Object detection information indicating is output.

The object arrangement information generation means 25 creates object arrangement information based on the object detection information (categories and positions of a plurality of objects arranged in the target area) output from the object detection means 24, and stores the object arrangement information in the storage means 30. To do. The object arrangement information includes identification information (ID) for identifying each object and position information indicating a position. The method of assigning the identification information (ID) to each object can be appropriately set. The identification information (ID) includes category information indicating the category of the object. Preferably, identification information including category information (common information) and number (individual information) indicating the type of the object is given to the objects having the same shape. The numbers are assigned so that, for example, the variance of the position of the center of gravity of the object increases in order from the larger direction.
As the position of the object indicated by the object arrangement information, a position in the target area is preferably used, and more preferably, a relative position in the target area is used.
In FIG. 7, an xy coordinate system is set in which the upper edge portion 111a and the left side edge portion 111b (the upper edge portion of the target area and the left edge portion of the target area) of the operation panel 110 are the x-axis and the y-axis, respectively, in the xy coordinate system. The xy coordinates are used as the relative position.

For example, as shown in FIG. 7, the object arrangement information generating means 25 imparts identification information [V1] to [V3] to the three volumes [V] in order from the left side along the x-axis. Then, identification information [D1] and [D2] are given to the two dials [D] in order from the left side along the x-axis, and the upper seven buttons [S] out of the eleven buttons [S] [ Identification information [S1] to [S7] are given to S] in order from the left side along the x-axis, and identification information [S8] to [S7] are given to the four buttons [S] on the right side in order from the top. S11] is given.
In FIG. 7, the positions of the volumes [V1] to [V3], the dials [D1], [D2], and the buttons [S1] to [S11] are the width, height, and center of gravity of the respective bounding boxes. It is indicated by G. The position of the point G is represented by the xy coordinates in the xy coordinate system.
The configuration of "representing the position of the center of gravity by the xy coordinates in the imaging screen or the target area of the imaging screen in the xy coordinate system" is included in the configuration of "representing the position of the object by the relative position" of the present invention.

The angle of view change amount determining means 26 will be described later.

The object identification means 27 includes object detection information (categories and positions of a plurality of objects arranged in the target area) output from the object detection means 24 and object arrangement information (identification of each object) stored in the storage means 30. Based on information and position), multiple objects arranged in the target area are individually identified.
As the object identification method by the object identification means 27, various methods can be used. For example, the position of an object having the same shape is extracted from the object detection information and the object arrangement information. Then, the position extracted from the object detection information and the position extracted from the object arrangement information are collated, and it is determined whether or not both positions match. At this time, it is determined whether or not both positions match while allowing a predetermined error range. That is, when the difference between the two positions is within a predetermined error range, it is determined that the two positions match, and when the difference exceeds the predetermined error range, it is determined that the two positions do not match. When it is determined that both positions match, the object placed at the position extracted from the object detection information is the object indicated by the identification information given to the object placed at the position extracted from the object placement information. Identify as.

Specifically, as shown in FIG. 8, the object identification means 27 collates the object detection information output from the object detection means 24 with the object arrangement information stored in the storage means 30.
For example, the position of the volume [V] (1) included in the object detection information (xy coordinates of the width and height of the bounding box and the position of the center of gravity) and the volume [V] included in the object arrangement information. The positions of the volumes [V1] to [V3] of the same shape (same category) as in (1) (xy coordinates of the width and height of the bounding box and the position of the center of gravity) are collated, and whether the difference between the two is within a predetermined range. Judge whether or not. At this time, the difference between the position of the volume [V] (1) and the positions of the volumes [V2] and [V3] exceeds a predetermined range, but the position of the volume [V] (1) and the position of the volume [V1] When the difference in position is within a predetermined range, the volume [V] (1) included in the object detection information is identified as the volume [V1] included in the object placement information (individual identification). To do.

The storage means 30 is composed of a ROM, a RAM, or the like, and stores a program for executing the processes of the means 21 to 27 and various data. The object arrangement information created by the object arrangement information generation means 25 is stored in the object arrangement information storage unit 31 of the storage means 30.
The input means 40 is composed of a keyboard or the like, and inputs various information.
The display means 50 is composed of a liquid crystal display device, an organic EL display device, and the like, and displays various information. If the display means 50 is a display means that allows information to be input by touching a display unit displayed on the display screen, the input means 40 may be omitted.

Next, the operation of the object identification device of the present embodiment will be described. The object identification device of the present embodiment is configured to be set to the object arrangement information generation mode or the object identification mode.
For example, by inputting the object arrangement information generation mode setting information or the object identification mode setting information from the input means 40, the object arrangement information generation mode or the object identification mode is set. Alternatively, when the object identification start information is input from the input means 40, the object arrangement information generation mode is first set, the object arrangement information is generated and stored in the storage means 30, and then the object identification mode is set. It is configured as follows.

The operation when the object information generation mode is set will be described with reference to the flowchart shown in FIG.
In step A1, the imaging information captured by the imaging means 10 is input. The process of step A1 is executed by the imaging information input means 22.
In step A2, the target area is determined based on the input imaging information. The process of step A2 is executed by the target area determination means 23.
In step A3, based on the input imaging information and the target area determined in step A2, the categories and positions of a plurality of objects arranged in the target area are detected and the object detection information is output. The process of step A3 is executed by the object detection means 24.
In step A4, based on the object detection information indicating the category and position of the object arranged in the target area detected in step A3, the identification information for identifying each object and the object arrangement information including the position of each object are generated. To do. The process of step A4 is executed by the object arrangement information generation means 25.
In step A5, the object arrangement information generated in step A4 is stored in the storage means 30. The process of step A5 is executed by the object arrangement information generation means 25.

The operation when the object identification mode is set will be described with reference to the flowchart shown in FIG.
In step B1, the image pickup information captured by the image pickup means 10 is input. The process of step B1 is executed by the imaging information input means 22.
In step B2, the target area is determined based on the input imaging information. The process of step B2 is executed by the target area determination means 23.
In step B3, based on the input imaging information and the target area determined in step B2, the categories and positions of a plurality of objects arranged in the target area are detected and the object detection information is output. The process of step B3 is executed by the object detection means 24.
In step B4, the object detection information indicating the category and position of the object arranged in the target area detected in step B3 and the object detection information stored in the storage means 30 (generated in step A4 of the object arrangement information generation mode). The object arrangement information (identification information and position of each object) is collated, and a plurality of objects arranged in the target area detected in step B3 are individually identified. The process of step B4 is executed by the object identification means 27.
After completing the process of step B4, the process returns to step B1. The processes of steps B1 to B4 may be executed continuously and repeatedly, or may be executed at predetermined time intervals.

Next, different embodiments of the present invention will be described.
When the imaging position and the imaging angle (imaging direction) of the imaging means 10 are changed, the positions (xy coordinates) of a plurality of objects arranged in the target area on the imaging screen change.
If the positions of a plurality of objects arranged in the target area on the imaging screen change significantly, the matching accuracy between the object detection information and the object arrangement information, that is, the individual identification accuracy of each object may decrease.
Therefore, when the imaging position or the imaging angle (imaging direction) of the imaging means 10 is changed, it is preferable to prevent the individual identification accuracy of each object from being lowered by regenerating the object arrangement information.
The change in the imaging position and the imaging angle (imaging direction) of the imaging means 10 can be determined by the amount of change in the angle of view of the imaging screen, as will be described later.

In this embodiment, the angle of view change amount determining means 26 shown in FIG. 1 is provided.
The angle of view change amount determining means 26 includes an imaging screen (M) used when generating object arrangement information stored in the storage means 30, and an imaging screen (P) indicated by imaging information output from the imaging means 10. ), It is determined whether or not the amount of change in the angle of view exceeds a predetermined range. The angle of view changes depending on the imaging position and the imaging angle (imaging direction) of the imaging means 10.
Various methods can be used as a method for determining whether or not the amount of change in the angle of view exceeds a predetermined range (method for determining the amount of change in the angle of view) by the angle of view change amount determining means 26.
For example, a method of determining whether or not the similarity between the image pickup screen (M) and the image pickup screen (P) is lower than a predetermined value can be used. The degree of similarity between the imaging screen (M) and the imaging screen (P) can be determined by the similarity between the feature points of the imaging screen (M) and the feature points of the imaging screen (P). Here, each feature point of the image imaging screen (M) and the imaging screen (P) can be detected by using a method based on a local feature amount such as SIFT. As will be described later, the similarity between the imaging screen (M) and the imaging screen (P) can be determined by using the feature points of the imaging screen (M) and the imaging screen (P).

Here, the amount of change in the angle of view between the target area (M) on the image pickup screen (M) and the target area (P) on the image pickup screen (P) is between the image pickup screen (M) and the image pickup screen (P). Is equivalent to the amount of change in the angle of view.
In the present embodiment, as shown in FIG. 11, the angle of view change amount determining means 26 is a target area on the imaging screen (M) used when generating the object arrangement information stored in the storage means 30. By determining whether or not the similarity between the feature points of (M) and the feature points of the target region (P) on the image pickup screen (P) indicated by the image pickup information output from the image pickup means 10 is lower than a predetermined value. It is determined whether or not the amount of change in the angle of view between the imaging screen (M) and the imaging screen (P) exceeds a predetermined range.
As the feature points of the target area, for example, the points illustrated by the broken lines in FIG. 11 are detected. It should be noted that preferably, a plurality of feature points are detected.
Then, among the feature points of both target regions, the feature vector of both target regions is generated from the set of the feature points that are determined to have the same correspondence from the similarity of the local feature quantities of the feature points. Then, it is determined whether or not the similarity of the feature vectors of both target regions is equal to or higher than a predetermined value.
The similarity of the feature vectors is represented by the cosine distances of both vectors shown below, for example, when the feature vector of the target region (M) is <M> and the feature vector of the target region (P) is <P>. ..
Sim (<M>, <P>) = (<M>, <P>) / (| <M> || <P> |)
When the cosine distance is equal to or more than a predetermined value, it is determined that the similarity of the feature vectors is high and the amount of change in the angle of view does not exceed the predetermined range, and when it is less than the predetermined value, the similarity of the feature vectors is high. It is low, and it is determined that the amount of change in the angle of view exceeds a predetermined range.
As the feature point detection, feature vector calculation method, and similarity determination method, various methods other than the above methods can be used.
In addition to the above method, various methods such as a method of calculating the amount of change due to pixel movement by optical flow as a velocity vector can be used as a method of detecting the amount of change in the angle of view.
The feature points of the image pickup screen (M) and the image pickup screen (P) are not limited to the feature points of the target area (M) on the image pickup screen (M) and the target area (P) on the image pickup screen (P).

The operation of the object identification device of the present embodiment will be described. The object identification device of the present embodiment is configured to be set to the object arrangement information generation mode or the object identification mode.
Since the operation when the object arrangement information generation mode is set is the same as the operation shown in FIG. 9, the description thereof will be omitted.

The operation when the object identification mode is set will be described with reference to the flowchart shown in FIG.
The operations of steps C1 and C2 are the same as the operations of steps B1 and B2 shown in FIG.
In step C3, it is determined whether or not the amount of change in the angle of view exceeds a predetermined range. If the amount of change in the angle of view exceeds the predetermined range, the process proceeds to step C4. On the other hand, if the amount of change in the angle of view does not exceed the predetermined range, the process proceeds to step C7. The process of step C3 is executed by the angle of view change amount determining means 26.
In steps C4 to C6, object arrangement information is generated (regenerated) and stored in the storage means 30 in the same manner as in steps A3 to A5 shown in FIG.
In steps C7 and C8, as in steps B3 and B4 shown in FIG. 10, a plurality of objects arranged in the target area are individually identified.

As a preferred mode of use of the present invention, a monitoring device or the like for monitoring the state of a plurality of operation units (including a plurality of operation units having the same shape) arranged on an operation panel or the like of an operation panel is assumed.
When the SSD disclosed in Non-Patent Document 1 or the object detection means by YOLO disclosed in Non-Patent Document 2 is used for such a monitoring device, as described above, the same shape existing in the target region is used. It is not possible to identify multiple objects as individuals.
Therefore, conventionally, arrangement information (position and ID of each object) indicating the arrangement state of a plurality of objects arranged in the target area is created (registered) as a two-dimensional map and output from the imaging means. Collate the object detection information (positions and categories of multiple objects placed in the target area) detected based on the imaging information with the object placement information (position and ID of each object) shown in the two-dimensional map. Each object was individually identified by.
However, it is necessary to manually create a two-dimensional map every time an object is registered, and it takes labor and time to create the two-dimensional map.
On the other hand, when the object identification device of the present invention is used, the object arrangement information is automatically generated, so that it is not necessary to manually create a two-dimensional map showing the arrangement state of the objects arranged in the target area. ..
In addition, each time the imaging information is input, the object detection information and the object arrangement information are collated to individually identify a plurality of operation units arranged on the operation panel. The state of the operation unit can be easily identified.

The present invention is not limited to the configuration described in the embodiment, and various changes, additions, and deletions can be made.
The processing means 20 is composed of the management means 21 to the object identification means 27, but the configuration of the processing means 20 is not limited to this. For example, unnecessary means can be deleted, or multiple means can be integrated.
The configurations of the target area discriminating means 23, the object detecting means 24, the object arrangement information generating means 25, the angle of view change amount discriminating means 26, and the object discriminating means 27 are not limited to the configurations described in the embodiments, and the gist of the present invention is described. Various changes can be made within the range that does not change.
The processes shown in FIGS. 9, 10 and 12 can be changed in various ways.

10 Imaging means 11 Imaging screen 20 Processing means 21 Management means 22 Imaging information input means 23 Target area discrimination means 24 Object detection means 25 Object placement information creation means 26 Object identification means 27 Angle change discrimination means 30 Storage means 31 Object placement information storage Unit 40 Input means 50 Display means 100 Operation panel 110 Operation panel (target area)
111 Edge (Target area edge)
111a Upper edge (upper edge of target area)
111b Left edge (left edge of target area)
111c Lower edge (lower edge of target area)
111d Right edge (right edge of target area)
120a-120c Volume 130a, 130b Dial 140a-140k Button

Claims (5)

An object identification device that identifies a plurality of objects including a plurality of objects having the same shape, which are arranged in a target area.
An image pickup means, a target area determination means, an object detection means, an object arrangement information generation means, an object identification means, and a storage means are provided.
The imaging means outputs imaging information indicating an imaging screen that images an imaging region including the target region and the plurality of objects arranged in the target region.
The target area determining means discriminates the target area on the imaging screen based on the imaging information output from the imaging means.
The object detection means is arranged in the target area on the image pickup screen based on the image pickup information output from the image pickup means and the target area on the image pickup screen determined by the target area determination means. Detects the categories and positions of the plurality of objects that are present
The object placement information generating means obtains identification information and positions of the plurality of objects based on the categories and positions of the plurality of objects arranged in the target area on the imaging screen detected by the object detecting means. The object arrangement information to be shown is generated, stored in the storage means, and stored.
The object identification means is indicated by the categories and positions of the plurality of objects arranged in the target area on the imaging screen detected by the object detection means, and the object arrangement information stored in the storage means. Based on the identification information and the position of the plurality of objects, the plurality of objects arranged in the target area on the imaging screen are individually identified.
It is possible to set the object placement information generation mode and the object identification mode.
When the object arrangement information generation mode is set, the object arrangement information is generated by the target area determination means, the object detection means, and the object arrangement information generation means and stored in the storage means.
When the object identification mode is set, the target area determination means, the object detection means, and the object identification means are used to individually identify the plurality of objects arranged in the target area on the imaging screen. An object identification device characterized by being configured in. The object identification device according to claim 1.
Equipped with a means for determining the amount of change in the angle of view
The angle-of-view change amount determining means is between an image pickup screen used when generating the object arrangement information stored in the storage means and an image pickup screen indicated by the image pickup information output from the image pickup means. Determine whether the amount of change in the angle of view exceeds the predetermined range,
When it is determined by the angle of view change amount discriminating means that the angle of view change amount exceeds the predetermined range when the object identification mode is set, the target area discriminating means, the object detecting means, and the object detecting means An object identification device characterized in that the object arrangement information generation means is configured to generate the object arrangement information and store it in the storage means. The object identification device according to claim 1 or 2.
The object identification means is an object represented by the category and position of the object arranged in the target area on the imaging screen detected by the object detecting means and the object arrangement information stored in the storage means. An object identification device characterized in that a plurality of objects are individually identified by collating the identification information and the position while allowing a predetermined error range. The object identification device according to any one of claims 1 to 3.
The object detection means detects the categories of the plurality of objects and their relative positions in the target area, and detects the categories.
The object identification information generation means is an object identification device that generates identification information of the plurality of objects and object arrangement information indicating a relative position in the target area. The object identification device according to any one of claims 1 to 4.
An object identification device characterized in that the target area discriminating means and the object detecting means are composed of a multi-layer neural network.

Images