JP2017204085A - Image recognition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for improving a success rate of labeling.SOLUTION: An image recognition system 1 is configured to perform semantic segmentation for an external environment of a robot which is necessary for a robot capable of executing a task (work) consisting of a plurality of phases (work processes) to execute the task. The image recognition system 1 comprises: a phase detection part 4 (work process switching detection means) for detecting switching of the phase being executed by the robot; a label selection part 5 (recognition candidate acquisition means) for, when the switching of the phase is detected by the phase detection part 4, acquiring a plurality of labels (recognition candidates) corresponding to the switched phase; and an image recognition part 7 (image recognition means) for executing the semantic segmentation to an image obtained by imaging the external environment of the robot by using an identification model (identifier) corresponding to the plurality of recognition candidates.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image recognition system.

  Non-Patent Document 1 discloses a semantic labeling technique that creates a hierarchical label tree with an abstract label as an upper layer and a specific label as a lower layer, and assigns the lowest layer label with a certain likelihood or more. ing.

Chenxia Wu, Ian Lenz and Ashutosh Saxena, “Hierarchical Semantic Labeling for Task-RelevantRGB-D Perception”, page 1, [online], June 3, 2014, [Search April 26, 2014], Internet (URL : http: //www.cs.cornell.edu/~asaxena/papers/wulenzsaxena_hierarchicallabel-rgbd_rss2014.pdf)

  However, there is a high possibility that labeling will fail if the number of label trees or branches and leaves is large.

  An object of the present invention is to provide a technique for improving the success rate of labeling.

  According to an aspect of the present invention, there is provided an image recognition system that performs semantic segmentation on an external environment of the robot, which is necessary for a robot capable of performing a work including a plurality of work steps to perform the work, When the switching of the work process is detected by the work process switching detection means for detecting the switching of the work process being executed by the robot and the work process switching detection means, the work process after the switching or the switching after the switching Obtained by imaging the external environment of the robot using recognition candidate acquisition means for acquiring at least one recognition candidate corresponding to any one of the profiles of the robot and a classifier corresponding to the at least one recognition candidate. Perform semantic segmentation on the resulting image An image recognition means, an image recognition system with a provided.

  According to the present invention, since different recognition candidates are used for each work process, the success rate of labeling is improved.

It is a functional block diagram of an image recognition system. It is a flowchart of operation | movement of an image recognition system. It is a top view of the service environment of a robot.

(First embodiment)
Hereinafter, the first embodiment will be described with reference to FIGS. 1 and 2.

  FIG. 1 shows a functional block diagram of the image recognition system 1. The image recognition system 1 performs semantic segmentation (image recognition and labeling) on the external environment of a robot, which is necessary for a robot capable of executing a task (work) composed of a plurality of phases (work processes) to execute the task. System.

  The image recognition system 1 includes a label set DB 2, a label determination profile DB 3, a phase detection unit 4 (work process switching detection unit), a label selection unit 5 (recognition candidate acquisition unit), an identification model switching unit 6, an image A recognition unit 7 (image recognition means).

  Hereinafter, for convenience of explanation, it is assumed that the tasks of the robot include a movement phase, a door opening phase, and an object gripping phase. The movement phase is a phase in which the robot moves to the front of a refrigerator installed in the service environment of the robot. The door opening phase is a phase in which the robot grasps the handle of the refrigerator door and opens the door. The object gripping phase is a phase in which the robot grips an object stored in the refrigerator.

  The label set DB 2 stores a label set (recognition candidate set) composed of a plurality of labels (recognition candidates) in association with each phase. For example, a label set {floor, refrigerator, shelf, obstacle} is associated with the movement phase. A label set {door, handle, floor, seal} is associated with the door opening phase. A label set {table, wall, can, plastic bottle, other object} is associated with the object gripping phase.

  The phase detection unit 4 detects the change of the phase being executed by the robot. Specifically, the phase detection unit 4 is configured such that the phase being executed by the robot is switched from the movement phase to the door opening phase, or the phase being executed by the robot is switched from the door opening phase to the object gripping phase. Is detected. The phase detection unit 4 detects a phase change by receiving a report of the completion of the phase currently being executed, for example, from the control unit of the robot.

  When the phase detection unit 4 detects a phase change, the label selection unit 5 acquires a label set corresponding to the phase after the change with reference to the label set DB2.

  The identification model switching unit 6 acquires an identification model (identifier) corresponding to the label set acquired by the label selection unit 5. Here, the discrimination model is a discriminator that estimates and outputs a label based on input information such as RGB and depth. The identification model switching unit 6 may select and acquire from a plurality of identification models generated in advance, or may generate an identification model each time, or the identification used immediately before the phase switching. A new identification model may be generated by performing transfer learning on the model. Here, transfer learning means that the number of outputs in the output layer is corrected, or learning that matches the GroundTruth of the training data with the label set is performed.

  Specifically, when the phase executed by the robot is switched from the movement phase to the door opening phase, the identification model switching unit 6 acquires an identification model corresponding to the label set {door, handle, floor, seal}. . Similarly, when the phase executed by the robot is switched from the door opening phase to the object gripping phase, the identification model switching unit 6 performs identification corresponding to the label set {table, wall, can, plastic bottle, other object}. Get the model.

  Then, the image recognition unit 7 uses the identification model acquired by the identification model switching unit 6 to execute semantic segmentation on an image obtained by imaging the external environment of the robot.

  Next, the operation flow of the image recognition system 1 will be described with reference to FIG.

  First, when the robot starts a task, the phase detection unit 4 determines the type of phase currently being executed by the robot (S100). Next, the phase detection unit 4 determines whether the previously determined phase is different from the currently determined phase (S110). When the phase detection unit 4 determines that the phase determined last time and the phase determined this time are the same (S110: NO), the robot uses the identification model currently used by the image recognition unit 7 as it is. Semantic segmentation is performed on an image obtained by imaging the external environment (S140), post-processing is performed (S150), and the process returns to S100.

  On the other hand, in S110, when the phase detection unit 4 determines that the previously determined phase is different from the currently determined phase (S110: YES), the label selection unit 5 sets the label set corresponding to the currently determined phase. Is obtained with reference to the label set DB2 (S120).

  Next, the identification model switching unit 6 acquires an identification model corresponding to the label set acquired by the label selection unit 5 (S130).

  Then, the image recognition unit 7 performs semantic segmentation on an image obtained by imaging the external environment of the robot using the identification model acquired by the identification model switching unit 6 (S140).

  Although the first embodiment has been described above, the first embodiment has the following features.

  The image recognition system 1 is a system that performs semantic segmentation on the external environment of a robot, which is necessary for a robot capable of executing a task (work) composed of a plurality of phases (work processes) to execute the task. The image recognition system 1 responds to the phase after switching when the phase detection unit 4 (work process switching detection means) that detects the phase switching performed by the robot and the phase detection unit 4 detects the phase switching. Using a label selection unit 5 (recognition candidate acquisition means) that acquires a plurality of labels (recognition candidates) and an identification model (discriminator) corresponding to the plurality of recognition candidates, the external environment of the robot is imaged. An image recognizing unit 7 (image recognizing means) that performs semantic segmentation on the obtained image. According to the above configuration, since different recognition candidates are used for each work process, the success rate of labeling is improved.

  The label set is composed of a plurality of labels. The label set only needs to be composed of at least one label.

  The phase detector 4 may determine the current phase based on the position of the robot in the service environment shown in FIG. For example, when the robot is not in front of the refrigerator, the phase detection unit 4 can determine that the phase being executed by the robot is the movement phase.

(Second Embodiment)
Next, a second embodiment will be described. Hereinafter, the present embodiment will be described with a focus on differences from the first embodiment, and overlapping description will be omitted.

  In the first embodiment, the label selection unit 5 acquires the label set corresponding to the phase after the switching when the phase switching is detected by the phase detection unit 4.

  However, instead of this, in this embodiment, the label selection unit 5 may acquire a label set corresponding to the profile of the robot after switching. In other words, the label selection unit 5 may flexibly acquire a label set that is a recognition candidate from information that can be acquired by a robot sensor or the like. Here, the profile means the state / attribute of the robot excluding the phase. The label determination profile DB 3 stores the robot profile in association with the label set.

  For example, when the robot controls the movement of the hand, the label selection unit 5 obtains a label set corresponding to the movement of the hand corresponding to the movement of the hand. The label set corresponding to the movement of the hand is, for example, {table, wall, can, plastic bottle, other object}. Similarly, when the robot controls the operation of the carriage, the label selection unit 5 obtains a label set corresponding to the movement of the carriage corresponding to the movement of the carriage. The label set corresponding to the operation of the carriage is, for example, {floor, refrigerator, shelf, obstacle}.

  Further, the label selection unit 5 obtains a coarser particle size, that is, a more abstract label set when the field of view of the camera provided in the robot is wide, and if the camera is narrow, the particle size is fine, that is, more specific. A simple label set. If {Furniture} is included in the abstract label set, {Refrigerator} is included in the specific label set. In addition, in the service environment shown in FIG. 3, considering the case where the robot is looking around the refrigerator from the left end of the room and the case where the robot is looking around the refrigerator from the center, in the former case, the robot is located far from the refrigerator. The label selection unit 5 obtains an abstract label set such as {floor, structure, furniture, ceiling} because it is located and has a viewpoint overlooking the entire room. This is because if the label selection unit 5 obtains a label set such as {refrigerator, washing machine, microwave oven} when the viewpoint is to look around the entire room, the labeling success rate is significantly reduced. On the other hand, in the latter case, the robot is located near the refrigerator, and the label selection unit 5 acquires a specific label set such as {refrigerator, washing machine, microwave oven}. Note that the label selection unit 5 can determine the width of the field of view of the camera provided in the robot, for example, by monitoring the average value of the output values of the Depth sensor. In other words, when the average value of the output values of the Depth sensor is a relatively large value, the robot looks over the entire room, and when the average value is relatively small, the robot is in the room. I will be watching the department.

  Further, the other profile may simply be a variable or flag information used for controlling the robot. That is, the label selection unit 5 may acquire a label set corresponding to a variable used for robot control or a label set corresponding to flag information used for robot control.

1 Image recognition system 2 Label set DB
3 Label determination profile DB
4 Phase detection unit 5 Label selection unit 6 Identification model switching unit 7 Image recognition unit

Claims (1)

An image recognition system for performing semantic segmentation on the external environment of the robot, which is necessary for a robot capable of performing a work consisting of a plurality of work steps to perform the work,
A work process switching detecting means for detecting switching of the work process being executed by the robot;
When the switching of the work process is detected by the work process switching detecting means, the recognition candidate acquisition for acquiring at least one recognition candidate corresponding to either the work process after switching or the profile of the robot after switching. Means,
Image recognition means for performing semantic segmentation on an image obtained by imaging the external environment of the robot using a discriminator corresponding to the at least one recognition candidate;
An image recognition system.

Images