JP7249928B2 - Tactile information estimation device, tactile information estimation method and program - Google Patents

Description

The present invention relates to a haptic information estimation device, a haptic information estimation method, and a program.

BACKGROUND ART Today, research on machine learning that handles a plurality of information such as visual information, audio information, motion information, etc. is widely conducted. Research on controlling an object using information such as force detection input from a tactile sensor is also widely conducted.

However, in these research results, it is difficult to acquire tactile information from the obtained visual information, such as the action performed empirically by humans, for example, determining the location where it is difficult to slip on an object from visual information. For this reason, it is a difficult task to make a robot look at an object, determine where the object is difficult to slip, and grip the object based on the determination result, as humans usually do.

Kuniaki Noda, 3 others, "Multimodal Integration Learning of Robot Behavior using Deep Neural Networks", (Netherlands), Robotics and Autonomous Systems, June 2014, p.721-p.736

Accordingly, embodiments of the present invention propose a tactile information estimation device, a tactile information estimation method, and a program for estimating tactile information from visual information.

A tactile information estimation device according to one embodiment automatically calculates a visual-tactile feature amount, which is a feature amount related to the visual information and the tactile information, based on the visual information of an object and the tactile information linked to the visual information. A model generation unit that generates a model including an organizing intermediate layer, and a visual-tactile feature amount extraction unit that extracts the visual-tactile feature amount self-organized via the model.

According to one embodiment, the tactile information of an object can be inferred from the visual information of the object.

1 is a block diagram showing functions of a haptic information estimation device according to an embodiment; FIG. FIG. 3 illustrates an example of a self-organized space according to one embodiment; 4 is a flow chart showing the flow of a learning phase according to one embodiment. 4 is a flow chart showing the flow of an estimation phase according to one embodiment; 1 is a block diagram showing functions of a haptic information estimation device according to an embodiment; FIG. The figure which shows typically a mode that an object is hold|gripped. The figure which shows typically the holding|grip information acquisition part which concerns on one Embodiment. 1 is a block diagram showing functions of a haptic information estimation device according to an embodiment; FIG. 4 is a flow chart showing the flow of an estimation phase according to one embodiment; 4 is a flow chart showing the flow of an estimation phase according to one embodiment; 1 is a block diagram showing functions of a haptic information estimation device according to an embodiment; FIG.

(First embodiment)
This embodiment describes a tactile information estimation device that estimates and outputs tactile information when visual information is input. This tactile information estimation device may output not only the result of estimating tactile information from visual information, but also a generative model for generating tactile information from visual information. A detailed description will be given below with reference to the drawings.

FIG. 1 is a block diagram showing functions of a haptic information estimation device 1 according to this embodiment. The tactile information estimation device 1 includes a visual sensor 100 , a tactile sensor 102 , an input unit 104 , a model generation unit 106 , an output unit 108 , and a visual-tactile feature quantity extraction unit 110 .

A visual sensor 100 acquires visual information of an object. For example, the visual sensor 100 has a camera, acquires visual information of an object as image information, and inputs it to the input unit 104 . The camera may acquire the image as a color image or may acquire the image as a grayscale image. Also, a device such as an RGB-D camera that acquires image information including depth information from the visual sensor 100 may be mounted.

The tactile sensor 102 acquires tactile information of an object. For example, the tactile sensor 102 may comprise multiple pressure point sensors arranged in an array. These multiple pressure point sensors may be covered with soft silicone or the like so as to be able to sense pressure as a surface. As another example, a commonly distributed sheet-like or film-like tactile sensor may be used.

As still another example of the tactile sensor 102, a flexible and permeable material such as silicone is provided as a contact portion that contacts an object, and displacement of the contact portion is photographed by a camera or the like, and the photographed state is detected. It may be a sensor that acquires the pressure as a surface based on . The camera that captures the displacement may be a stereo camera, or a device that can determine the unevenness of the surface using ultrasonic waves or the like.

As in the example described above, the tactile sensor 102 may be any sensor that can acquire pressure information acquired as points as surface pressure information. Also, the sensor does not necessarily have to be able to acquire the pressure information of the surface at the same timing, and the sensor capable of acquiring the pressure information of one or more points scans the points on the surface of the object and obtains the pressure information at each point. It may be possible to acquire pressure information as a surface by sensing.

The tactile sensor 102 may measure the pressure applied to the tactile sensor 102 from the surface of the object, as described above. Another example is a sensor that can measure the repulsive force from inside the object.

The input unit 104 receives visual information sensed by the visual sensor 100 and tactile information sensed by the tactile sensor 102 as sensor signals in a learning phase for generating a model (hereinafter simply referred to as a learning phase). The received sensor signal is output to the model generator 106 .

At least one of the visual sensor 100 and the tactile sensor 102 may be another device provided outside the tactile information estimation device 1 . In this case, the input unit 104 receives signals from the external visual sensor and/or the external tactile sensor and outputs them to the model generation unit 106 . In this way, the input unit 104 may not only receive sensor signals within the tactile information estimation device 1, but may also receive input of signals from the outside.

As another example, in the process of model generation, sensor signals sensed by the visual sensor 100 and the tactile sensor 102 may be input to the model generation unit 106 without going through the input unit 104 . Furthermore, at least one of the visual sensor 100 and the tactile sensor 102 may be provided as part of the input unit 104 .

On the other hand, in an estimation phase for estimating tactile information (hereinafter simply referred to as an estimation phase), the input unit 104 receives visual information from the visual sensor 100 . As described above, the visual sensor 100 may be provided inside the tactile information estimation device 1, or may receive visual information from an external visual sensor. Furthermore, visual information may be input from an external file server or the like via a network or the like. The input visual information is transmitted to the visual-tactile feature extraction unit 110 .

Based on the input visual information and the tactile information linked to the visual information, the model generation unit 106 generates a model that outputs the visual information and the tactile information when the visual information and the tactile information are input. This model is, for example, a model generated based on a learning model such as a CNN (Convolutional Neural Network) or FNN (Feedforward Neural Network). As another example, a model may be generated that outputs tactile information when visual information is input.

As another example of input/output information, the generated model is a model that outputs tactile information when visual information and tactile information are input, or a model that outputs visual information and tactile information when visual information is input. It may be a model that

In this way, the generated model is generated based on the visual information and the tactile information linked to the visual information, and the information on the visual and tactile feature amount, which is the feature amount that links the visual information and the tactile information, is self Anything that is automatically generated by organization may be used.

The model generation unit 106 transmits the generated model to the output unit 108 and the visual-tactile feature amount extraction unit 110 . Note that when the model is not used outside the haptic information estimation device 1, it is not necessarily output to the output unit . In the learning phase, parameters such as loss and slope of the generated model may be output from the output unit 108 so that the user can check the learning status of the model.

The output unit 108 outputs the model output from the model generation unit 106 to the outside in the learning phase. By outputting the model to the outside in this way, it becomes possible to use the same model in other haptic information estimation devices. On the other hand, in the estimation phase, tactile information converted from visual information, visual-tactile feature amounts, and the like are output.

In the learning phase, the visual-tactile feature amount extraction unit 110 acquires self-organized visual-tactile feature amounts based on the model generation unit 106 and generates a self-organized space of the visual-tactile feature amounts. This self-organized space is a space formed such that the state changes continuously or discretely along an axis, unlike ordinary class classification and category classification.

FIG. 2 is a diagram showing an example of visualization of the visual-tactile feature quantity extracted by the visual-tactile feature quantity extraction unit 110. As shown in FIG. In the example of FIG. 2, the relationship between slipperiness and hardness is shown as a feature amount indicating tactile sensation. The vertical axis indicates slipperiness, and the closer to the origin O, the easier it is to slip, and the farther from the origin O, the harder it is to slip. On the other hand, the horizontal axis indicates hardness, and the closer to the origin O, the harder, and the farther from the origin O, the softer.

The description of the region in the figure is the addition of the description of the above axis. For example, in the figure, the area close to the origin O is an area where objects are slippery and hard. A region that is far from the origin O in the vertical direction and close to the origin O in the horizontal direction is a non-slip and hard region. Similarly, there are slippery and soft regions and non-slippery and soft regions as shown in the figure.

Although elliptical regions are shown in the drawing, the feature values change continuously or discretely along the axis as described above, and these regions cannot be clearly distinguished. do not have. In other words, the self-organization performed by the visual-tactile feature amount extraction unit 110 is not general class classification and categorization, but indices such as hardness and slipperiness change along the axis.

These classifications are not limited to slipperiness and hardness, and may include other indicators as tactile features. If another index is provided, the space is not a two-dimensional space but a three-dimensional or higher-dimensional space. In the case of a two-dimensional surface, it is not limited to slipperiness and hardness, and an appropriate one may be used according to the purpose of the information to be output.

Moreover, even indices of slipperiness and hardness may not be clearly expressed in two-dimensional space due to the dimensions in the intermediate layer of the model generated by the model generation unit 106 . In such a case, the indexes of slipperiness and hardness may be self-organized so that they can be expressed in a three-dimensional space or more.

In this way, the visual-haptic feature amount extraction unit 110 extracts self-organized visual-haptic feature amounts based on the generated model.

In the estimation phase, for example, when the model generation unit 106 generates a model using an autoencoder, the visual tactile feature amount extraction unit 110 encodes the input visual information using the encoder part of the model. Then, a visual-tactile feature quantity is extracted according to the position in which the encoded information exists in the self-organized space. The extracted visual-tactile feature quantity is output via the output unit 108 .

Next, the operation of the learning phase will be described using a flowchart. FIG. 3 is a flowchart showing the operation of the haptic information estimation device 1 in the learning phase.

First, visual information and tactile information are obtained from the visual sensor 100 and the tactile sensor 102 via the input unit 104 (S100). For example, at the timing when tactile information of an object is acquired by the tactile sensor 102, an image of the object is acquired as visual information, and the visual information and the tactile information are input to the tactile information estimation device 1 via the input unit 104. be done. As described above, the visual sensor 100 and the tactile sensor 102 may transmit the acquired information to the model generation unit 106 without using the input unit 104 .

As another example, the user may acquire the visual information and the tactile information by specifying the timing at which the visual sensor 100 and the tactile sensor 102 sense the respective sensor information via the input unit 104 . In this case, the input unit 104 may be provided with a user interface for the user to input instructions. As still another example, visual information acquired in advance and tactile information linked to the visual information may be input via the input unit 104 . The information obtained in advance may be stored in a file server or the like.

As described above, the visual sensor 100 is, for example, a camera, and receives an image captured by the camera from the input unit 104 as visual information. If the image information was acquired in color, it may be converted to grayscale for input. Also, in order to eliminate the way light such as illumination hits, the brightness, brightness, saturation, contrast, dynamic range, etc. may be adjusted for each region before input. As an example, the visual sensor 100 acquires texture information on the surface of an object as visual information and inputs it to the input unit 104 . These preprocessings for learning are not limited to being performed in the visual sensor 100, but may be performed in the input unit 104 or other modules.

The tactile sensor 102 is, for example, a sensor having a plurality of pressure point sensors arranged in an array, as described above. In the case of such a sensor equipped with a plurality of pressure point sensors, when the tactile sensor 102 is pressed against an object with a predetermined force, the data sensed by each sensor is summarized as tactile information in the input section. Input from 104. More specifically, when the tactile sensor 102 is pressed against an object with a predetermined force, the pressure information sensed by each of the pressure point sensors is two-dimensionally arranged to sense pressure information as a surface. make sure to In other words, what corresponds to the pixels in the image is assumed to be the pressure point sensor, and pressure information as a surface is input as tactile information.

Here, the predetermined force may be, for example, an external force applied from an object to the tactile sensor 102 after sensing that the object has been touched. As another example, when the tactile sensor 102 is provided on fingers of a gripper or the like, when the distance between the fingers is controlled to be a predetermined distance, the force applied to the tactile sensor 102 from an object There may be. In this way, when touching various objects, it is sufficient to be able to sense them in the same state.

By doing so, it is possible to appropriately acquire tactile information not only for hard objects, but also for soft or deformable objects.

Further, the tactile sensor 102 is moved from the above state by a predetermined distance in a direction parallel to the direction in which the pressure points of the tactile sensor 102 are arranged, and the sensing information of the pressure applied during the movement or the sensing information of the pressure at the position after the movement is collected. may be added as haptic information. In this way, by sensing the information of moving a predetermined distance, in addition to the hardness information that can be obtained by applying a predetermined force, the object can be detected based on the sensing of each pressure point sensor when the object is moved a predetermined distance. It is also possible to obtain information on the slipperiness of the surface. In the case of acquiring sensory information during movement, it is possible not only to move a predetermined distance, but also to apply a predetermined force to move and obtain pressure information as tactile information. When acquiring sensory information during movement, tactile information may be acquired as information along a time series.

The information input from the tactile sensor 102 is not limited to this, and the information may be tactile information using one pressure sensor. Alternatively, the tactile information sensed by the user may be input from the input unit 104 .

As described above, the visual information and tactile information input to the input unit 104 are high-dimensional data represented by tensors (eg, vectors or matrices) having elements of the number of pixels or the number of pressure points. When acquiring information when moving a predetermined distance as tactile information, two or more matrices having the number of elements of the number of pressure points, that is, a matrix of a non-moving state and a matrix of a state of movement or after movement. may be learned using as tactile information. In the case of acquiring information in chronological order during movement, the input haptic information may be a three-dimensional matrix or a multi-channel matrix.

Next, the model generator 106 generates a model that outputs visual information and tactile information when visual information and tactile information are input (S102). The model generation unit 106 generates the model by using an autoencoder, for example. When using an autoencoder, visual information and tactile information are input to a model, and a network is generated through learning so that the visual information and the tactile information can be obtained. The model generated may be a model based on CNN, as described above, or may be another network model such as FNN.

When visual information and tactile information are used for both input and output, an image is used as visual information, pressure information obtained by a pressure point sensor is used as tactile information, and these data are used as data of another channel to implement CNN. may When the pressure information is acquired in time series, the generative model may be implemented as a CNN in which the pressure information arranged in the time series is set as a separate channel as described above, or it may be input as a three-dimensional matrix. can be implemented as If other networks are used, visual and tactile information data may be implemented, for example, an input layer into which respective pixel values are input and an output layer into which respective pressure point values are output.

In learning, the activation function may be an identity map or a non-identity map. Also, as the loss function, a squared error may be used, or another linear or nonlinear error function may be used. A mini-batch may be appropriately configured for training data for learning. As an optimization algorithm, Momentum, Adam, or the like may be used, but not limited to these, and other optimization algorithms may be used.

As another example, when visual information is input, a network that outputs tactile information may be generated as a model through learning. As a model, a model based on a network such as CNN or FNN is generated in the same manner as described above. Furthermore, a network may be constructed in which the intermediate layer has a low-dimensional layer such as a two-dimensional layer so that the feature amount can be extracted. In this case, for example, when visual information is input to the input layer, the model generation unit 106 generates a model that outputs tactile information linked to the visual information by performing supervised learning.

As described above, it is a model that inputs visual information and tactile information and outputs tactile information, or a model that inputs visual information and outputs visual information and tactile information. You may generate a model that has These models can also be generated by performing supervised learning.

Next, the visual-haptic feature amount extraction unit 110 extracts visual-haptic feature amounts that link the visual information and the tactile information based on the model generated by the model generation unit 106 (S104). This visual and tactile feature amount is extracted, for example, by acquiring an intermediate layer of a network that generates a feature amount indicating how the visual information input to the model affects the tactile information.

The intermediate layer of the model generated in S102 is a low-dimensional representation of the feature amount between the visual information and the tactile information. Therefore, the visual-tactile feature quantity extraction unit 110 extracts and outputs a self-organized visual-tactile feature quantity space based on this intermediate layer. Further, in S104, the visual-haptic feature amount extraction unit 110 may generate a model that outputs tactile information when visual information is input.

As an example, when the model generation unit 106 generates a model using an autoencoder based on CNN, that is, when the model is generated using a so-called convolutional autoencoder, the visual-haptic feature amount extraction unit 110 extracts a layer of a code part in the model. Get the middle layer that is The obtained intermediate layer is a layer that indicates a compressed low-dimensional feature amount for high-dimensional (number of pixels, number of pressure points) data of visual information and tactile information input to the model. The model generation unit 106 may generate the model so that the feature amount can be visualized by compressing the dimension in the intermediate layer to a low dimension such as two dimensions or three dimensions. In this case, the visual-haptic feature amount extraction unit 110 extracts the self-organized feature amount by extracting this intermediate layer.

The model generated by the model generation unit 106 does not need to be generated to have a lower dimension such as 2D, and may be generated to have a higher dimensional intermediate layer such as 16×16 dimensions. good. In this way, when the intermediate layer is not as low-dimensional as two-dimensional or the like, the visual-haptic feature quantity extraction unit 110 extracts the intermediate layer of the generated model, and adjusts the input/output to match this intermediate layer. , an autoencoder may be used to generate an encoder that reduces the dimension to a lower dimension, and the visual and tactile feature amount may be extracted.

Low-dimensional feature quantities are shown, for example, as shown in FIG. In FIG. 2, slipperiness and stiffness form a space along two-dimensional axes. The visual-tactile feature quantity extraction unit 110 forms a space around these indices by self-organizing the feature quantities. As described above, this space does not have to be two-dimensional, and may be formed as a three- or more-dimensional space.

For example, when the tactile sensor 102 and the object are in contact with each other, at least part of the tactile sensor 102 moves with a speed relative to the object. , the data acquired by the tactile sensor 102 is information in chronological order. In this case, tactile information can be obtained with hardness as a component in the normal direction and slipperiness as a component in the planar direction. By acquiring the tactile information in this way, the visual-tactile feature amount considering hardness and slipperiness is mapped by self-organization.

As another example, in an autoencoder using the encoder portion in a convolutional autoencoder, various features are extracted in the encoding layer. Based on this layer of encoding, the hardness and slipperiness indices may be mapped. The visual and haptic feature quantity extraction unit 110 may use the layers of this encoder to perform FNN learning, generate a new network, and extract the feature quantity. In this case, supervised learning may be performed in generating a new network, and by doing so, it is possible to explicitly extract the feature values of hardness and slipperiness.

In any of the cases described above, fine tuning may be performed using the model generated by the model generation unit 106 . The visual and tactile feature quantity extraction unit 110 fine-tunes the generated model to generate a new model for extracting visual and tactile feature quantities such as hardness and slipperiness when visual information is input. good too.

In this way, without explicit labeling by the user, the visual and tactile feature values using hardness and slipperiness as indices are self-organized from the visual information and tactile information acquired by forming the CNN autoencoder. Also, by executing supervised learning using a self-organized hidden layer, visual tactile features are extracted explicitly using hardness and slipperiness as indices is also possible.

Note that the space of self-organized feature quantities does not always contain information that can be clearly understood by humans, such as hardness and slipperiness. However, as described above, if the tactile sensor 102 acquires information that includes hardness and slipperiness with respect to visual information, the hardness and slipperiness can be included as some sort of index by self-organization. , it becomes possible to form a self-organized space.

That is, it is not essential that the visual and tactile features are formed as a space as shown in FIG. It may also be formed as an index of impossibility (for example, a linear or non-linear combined index of hardness and slipperiness for visual information). In other words, when the self-organized visual-tactile feature quantity is expressed as a numerical value, it is not necessarily an index that humans can clearly understand as a tactile sensation, and it is an index that humans cannot intuitively understand. good too.

The visual-tactile feature quantity extraction unit 110 stores a space of self-organized visual-tactile feature quantity generated in this way or a model for extracting the visual-tactile feature quantity. It may be stored not only in the visual-tactile feature amount extraction unit 110 but also in a storage unit (not shown) in the haptic information estimation device 1 . As another example, the generated visual-haptic feature quantity extraction model may be output via the output unit 108 so that it can be used by other haptic information estimation devices.

In the learning phase, by performing learning as described above, when visual information and tactile information are input, a model is generated in which the visual and tactile information is output, and based on the generated model, visual information is generated. A space of self-organized visual and tactile features is extracted that outputs tactile information when input.

Next, the estimation phase will be explained using a flowchart. FIG. 4 is a flowchart showing the flow of processing in the estimation phase for extracting visual-tactile feature amounts from visual information. In the estimation phase, data is transmitted and received as indicated by the dashed lines in FIG.

First, visual information is obtained through the input unit 104 (S200). For example, visual information sensed by the visual sensor 100 is input to the tactile information estimation device 1 via the input unit 104 . In addition, data including visual information acquired outside may be input to the input unit 104 . The input visual information is, for example, visual information of an object different from the object used for learning.

Next, the visual-tactile feature amount extraction unit 110 extracts the visual-tactile feature amount from the input visual information (S202). Based on the self-organized space prepared in the learning phase or a model for extracting visual and tactile feature amounts, the visual and tactile feature amount extraction unit 110 acquires tactile information from the input visual information.

The visual-tactile feature amount extraction unit 110 acquires tactile information from visual information using the model generated by the model generation unit 106 and the self-organized visual-tactile feature amount space extracted by the visual-tactile feature amount extraction unit 110. do. For example, visual information is input to the model generated by the model generating unit 106, and the output in the intermediate layer is obtained. After that, in the space of visual and tactile feature amounts, the coordinates to which the obtained output of the intermediate layer is mapped are obtained. Then, haptic information is obtained based on the mapped output of the intermediate layer.

Next, the output unit 108 outputs the tactile information acquired by the visual-tactile feature quantity extraction unit 110 (S204). For example, by operating a gripper such as a gripper connected to the robot based on the output tactile information, it is possible to grip an object based on the tactile information.

As another example, the tactile information estimation device 1 may convert the signal into a signal for controlling the grip portion in advance and output the signal to the robot, or the tactile information estimation device 1 may be provided with a grip portion. Alternatively, a control signal may be output to the gripping portion to control the gripping portion. The gripping unit provided in the tactile information estimation device 1 may be a gripper or the like provided with the tactile sensor 102 .

As described above, according to this embodiment, by self-organizing input visual information and tactile information, it is possible to acquire tactile information from input visual information without labeling or the like by the user. Become. In this way, it is possible to acquire tactile information of an unknown object from which visual information was obtained by acquiring tactile information using feature values self-organized through learning without labeling by the user. becomes.

Furthermore, according to the present embodiment, instead of categorized or labeled values as tactile information, indices such as hardness and slipperiness are output as numerical values, so that the grasping portion of the robot can perform more precise movements. It becomes possible to output the index. In other words, it becomes possible to control the force for operating the gripping part of the robot, etc., with finer precision, rather than using rough indicators such as hardness and softness. can be grasped.

For example, it is not limited to gripping objects with fixed shapes and materials in factories, etc., but also objects such as cloth and pouches that deform due to gripping force, whose shape and hardness are difficult to model. The tactile information estimating apparatus 1 according to the present embodiment can accurately estimate tactile information of objects such as foodstuffs for lunch boxes, which are costly to model, and objects that are likely to have individual differences. It becomes possible.

(Second embodiment)
FIG. 5 is a block diagram showing the functions of the haptic information estimation device 1 according to this embodiment. In addition to the functions of the tactile information estimation device 1 according to the first embodiment described above, a grip information acquisition unit 112 is further provided.

The gripping information acquisition unit 112 includes, for example, a gripper capable of gripping an object. The gripping information acquisition unit 112 grips an object, and the visual sensor 100 senses the gripped state as visual information. Information on the object and the gripping position is acquired as visual information while the object is gripped in this way, and the acquired visual information is transmitted to the model generation unit 106 via the input unit 104 . Information as to whether or not the object can be stably grasped may be transmitted in association with this visual information. Grasping the shape of the object at various positions may be attempted, that is, the gripping state of the object may be changed, and visual information, tactile information, and information as to whether or not the object is grippable may be obtained and used as training data.

For example, the user may input through the input unit 104 whether or not the object is being gripped. As another example, the grasping information acquisition unit 112 may be provided with a weighing scale to determine whether or not the object can be kept in a lifted state. Whether or not the object can be held in a lifted state is determined by moving the gripping information acquisition unit 112 in the direction opposite to the direction of gravity, and the weight information reaches a predetermined value (the weight of the object). The determination may be made by grasping the situation, such as the state continues for a predetermined number of seconds, or the state in which the weight information does not change from a predetermined value continues even after moving a predetermined distance.

Based on this visual information, the model generation unit 106 learns about gripping positions and generates a gripping position model for positions where an object can be stably gripped. This gripping position model is generated as a model different from the model for extracting the visual-tactile feature quantity described above. The gripping position model can be learned based on various learning methods and various models. For example, supervised learning may be performed in which the shape and grasping position of an object are input as visual information from the input layer, and whether or not the object can be grasped is output from the output layer. As another example, when the shape of an object is input, a model may be generated that outputs the position at which it is easy to grasp. In this case, not only two-dimensional information but also three-dimensional information may be acquired.

In the learning phase, the model generation unit 106 transmits the learned gripping position model to the gripping position estimation unit 114 . The gripping position estimation unit 114 stores the received gripping position model. In the estimation phase, the gripping position estimation unit 114 estimates the gripping position from the visual information according to the stored gripping position model, and outputs it via the output unit 108 .

Also, a tactile sensor 102 may be provided at a portion of the gripper that grips an object. When the tactile sensor 102 is provided, the grip information acquisition unit 112 may input tactile information sensed by the tactile sensor 102 to the model generation unit 106 via the input unit 104 . By providing the gripping information acquisition unit 112 with the tactile sensor 102, it is possible to acquire the gripped position and tactile information and the data associated with the visual information at the same timing.

FIG. 6 schematically illustrates the gripping information acquisition unit 112 including the tactile sensor 102. As shown in FIG. For example, suppose that the grasping information acquisition unit 112 could grasp the object when it was at the position 112A, and could not grasp the object when it was at the positions 112B and 112C.

At the position 112A, the gripping information acquisition unit 112 transmits information indicating that the object can be gripped and the tactile information thereof to the input unit 104. At the same timing, the visual sensor 100 transmits visual information to the input unit 104. . In this way, the grippable position and tactile information, and the visual information linked thereto are input to the model generation unit 106 .

On the other hand, at the positions 112B and 112C, the gripping information acquisition unit 112 transmits information indicating that the object cannot be gripped. In this case, the tactile information is transmitted based on the data when the object and the tactile sensor 102 are in contact with each other. The visual sensor 100 transmits sensing information of contact with an object.

For example, the visual sensor 100 captures an image and the tactile sensor 102 acquires tactile information when the object and the gripping information acquisition unit 112 come into contact with each other. Thereafter, as described above, in order to determine whether or not it is possible to hold the object at that position, the holding information acquisition unit 112 is arbitrarily moved to grasp the holding state. After grasping the grasping state, the graspable information detected by the grasping information acquisition unit 112, the tactile information detected by the tactile sensor 102, and the visual information detected by the visual sensor 100 are linked and transmitted. By doing so, it is possible to transmit information from each sensor under the same control in both of the two cases described above, that is, when the object can be grasped and when the object cannot be grasped.

In the estimation phase, when the visual information is input, the visual information is input to the visual tactile feature amount extraction unit 110 and the gripping position model generated by the model generation unit, and the visual tactile feature amount and at which position Whether it can be grasped is output. In this way, the texture data of the object is input to the visual and tactile feature amount extraction unit 110, and the shape data of the object is input to the gripping position model. It is possible to acquire information as to whether or not the object should be grasped.

As described above, according to the present embodiment, in addition to linking tactile information and visual information, graspable information based on the shape of an object is modeled by linking it with visual information. From the texture information and shape information obtained, it is possible to output both force and position to control when grasping.

Note that the position of the visual sensor 100 is not particularly limited in the above description. For example, but not limited to, the visual sensor 100 in a fixed position, as in the situation described above. FIG. 7 is a diagram showing another example regarding the position of the visual sensor 100. As shown in FIG.

As shown in FIG. 7 , the visual sensor 100 may be provided in the gripping information acquisition section 112 . By providing the visual sensor 100 in the gripping information acquisition unit 112 in this way, it is possible to acquire more highly accurate gripping position information and tactile information.

For example, if a visual sensor 100 is provided as shown in FIG. 7, it is possible to acquire the shape information of the object and the gripping position information in association with each other, and to acquire the visual information based on the position of the tactile sensor 102. It is possible to more accurately acquire the texture information of the location where the tactile sensor 102 is in contact with the sensing information of 102 .

In this case, the visual information may be obtained by shifting the timing of obtaining the shape information and the timing of obtaining the texture information. That is, the shape information may be acquired at a position where the entire object can be acquired, and the texture information may be acquired while the tactile sensor 102 is in contact with the object.

In the learning phase, learning of the grip position based on the shape and the visual and tactile feature amount based on the texture are performed based on the information such as the visual information thus acquired.

In the estimation phase, first, the visual information of the shape of the entire object is acquired, the gripping position is estimated, and then the texture information at the gripped position is acquired to estimate the visual-tactile feature amount. In this way, a two-step configuration is also possible.

However, as in the case where the visual sensor 100 is fixed, it is not essential to acquire visual information in two stages in both the learning phase and the estimation phase, and visual information of the entire object is acquired as visual information. Alternatively, texture information may also be obtained from the obtained visual information.

(Third embodiment)
FIG. 8 is a block diagram showing the functions of the haptic information estimation device 1 according to this embodiment. The tactile information estimation device 1 further includes a gripping position determining unit 116 , an object property estimating unit 118 , a gripping force determining unit 120 , a gripping control unit 122 and a gripping unit 124 .

The gripping position determination unit 116 determines the gripping position based on the gripping position estimated by the gripping position estimation unit 114 . The gripping position estimation unit 114 may estimate the gripping position based on the gripping position model generated as in the above-described second embodiment, or may estimate the gripping position from visual information by another method. can be Further, when there is feedback of information from the gripping unit 124, the gripped position of the object is updated based on the feedback information.

The object property estimation unit 118 estimates properties of the object from the visual and tactile feature quantity extracted by the visual and tactile feature quantity extraction unit 110 . Furthermore, the object property estimating unit 118 may estimate the property of the gripped part of the object based on the gripping position determined by the gripping position determination unit 116 as well as the visual-tactile feature amount. Also, when information is fed back from the gripping unit 124, the properties of the object are updated based on the information fed back. When updating, the conversion information from the visual-tactile feature quantity output by the visual-tactile feature quantity extraction unit 110 to the property of the object may also be updated.

The gripping force determining unit 120 determines the gripping force based on the properties of the object estimated by the object properties estimating unit 118 .

The gripping position determining unit 116 and the gripping force determining unit 120 output the determined gripping position and gripping force from the output unit 108, and an object to be gripped by an external gripping device, that is, an object different from the object used for learning. An object to be grasped, which is another object, may be grasped. As shown in FIG. 8, when the tactile sense information estimation device 1 is provided with a grasping unit 124, the information may be output to the grasping control unit 122 without outputting to the outside.

The grip control unit 122 outputs a signal for controlling the grip unit 124 to the grip unit 124 based on the grip position determined by the grip position determination unit 116 and the grip force determined by the grip force determination unit 120. . The grip control unit 122 is not an essential component, and the grip position determination unit 116 and the grip force determination unit 120 directly output information to the grip unit 124, generate a control signal in the grip unit 124, It may work. As another example, the gripping position determining unit 116 and the gripping force determining unit 120 may each generate control signals to operate the gripping unit 124 .

The gripping unit 124 grips an object to be gripped, which is another object that is actually scheduled to be gripped, and feeds back information about the gripping state to the gripping position determining unit 116 and/or the object property estimating unit 118 .

The operation in the learning phase is the same as in each of the above-described embodiments. Also, in FIG. 8, the gripping information acquisition unit 112 in the second embodiment is omitted, but the gripping information acquisition unit 112 may be provided. The operation in the estimation phase will be described below.

FIG. 9 is a flowchart showing operations in the estimation phase according to this embodiment.

First, the visual sensor 100 acquires the visual information of the object to be gripped, and inputs the visual information to the visual-tactile feature amount extraction unit 110 and the gripping position estimation unit 114 via the input unit 104 (S300). Next, the visual-tactile feature amount extraction unit 110 extracts the visual-tactile feature amount from the visual information using the generated model (S302).

The gripping position estimation unit 114 estimates the grippable position of the object to be gripped from the input visual information, and the gripping position determination unit 116 determines the grippable position of the object to be gripped from the grippable position estimated by the gripping position estimation unit 114. is determined (S304). For example, the gripping position estimating unit 114 quantifies and estimates an index indicating how grippable a plurality of grippable positions is, and based on the quantified index, the most appropriate grippable position is determined. The gripping position determination unit 116 determines. When the grip information acquisition unit 112 is provided and a grip position model is generated by the model generation unit 106, the grip position is estimated and determined using the grip position model.

The object property estimation unit 118 estimates properties of the object from the visual and tactile feature quantity extracted by the visual and tactile feature quantity extraction unit 110 (S306). The properties of an object are properties necessary for gripping the object, taking into consideration both hardness and slipperiness based on the extracted visual-tactile feature amount. As an example, the properties of an object refer to values obtained by performing a predetermined conversion based on the numerical values of feature amounts of hardness and slipperiness. The predetermined conversion may be a conversion defined in advance, or may be a conversion obtained by learning the defined conversion through reinforcement learning or the like.

As described above, the visual and tactile feature amounts self-organized in the intermediate layer of the model generated by the model generation unit 106 are not necessarily information that humans can intuitively understand. In such a case, the object property estimating unit 118 uses visual and tactile feature amounts that are difficult for humans to directly perceive what kind of sensation or touch it is, and calculates the necessary information for grasping the object. Calculate the properties of the object required to calculate the force. In this way, the self-organized visual-tactile feature amount may be any feature amount that can extract the characteristics of an object that can be converted into a force applied to grip the object.

Also, in S306, the object property estimation unit 118 may estimate the property of the object based on the gripping position estimated by the gripping position estimation unit 114, not only from the visual-tactile feature amount. For example, at the grippable position estimated by the gripping position estimation unit 114, the characteristics of the object may be estimated from the texture information of the object to be grasped at the point where the gripping unit 124 and the object to be grasped come into contact.

Next, the gripping force determination unit 120 determines the gripping force, which is the force applied when gripping, from the estimated properties of the object (S308). The estimated properties of the object are, for example, quantified properties based on both hardness and slipperiness, as described above. to decide. Conversely, in order to determine the gripping force, the properties of the object are the numerical values of hardness and slipperiness. Estimate the characteristic value for

The output unit 108 outputs the grip position determined by the grip position determination unit 116 and the grip force determined by the grip force determination unit 120 (S310). As described above, the haptic information estimation device 1 according to the present embodiment determines and outputs the gripping position and the gripping force when the visual information of the object to be gripped is input. By performing such an output, it becomes possible to output information necessary for more specifically controlling a gripping device such as a gripper, rather than an abstract value such as a visual-tactile feature amount.

FIG. 10 is a flowchart showing an example of the operation of the tactile information estimation device 1 when the grip portion 124 is provided. When the gripping portion 124 is provided, the gripping portion 124 actually grips the object to be gripped and feeds back the gripped state, thereby further improving the determination accuracy of the gripping position and gripping force.

The operations up to the determination of the gripping force in S308 are the same as the operations shown in FIG. 9 described above. That is, when the visual information of the object to be gripped is acquired, the tactile information estimation device 1 determines the gripping position and the gripping force.

Next, the gripping control unit 122 executes a gripping operation by controlling the gripping unit 124 (S312). The gripping control unit 122 generates a control signal that causes the gripping unit 124 to operate at the determined gripping position and the determined gripping force, and transmits the control signal to the gripping unit 124 . The gripping unit 124 grips the object to be gripped by operating based on the control signal received from the gripping control unit 122 .

Next, the gripping control unit 122 determines whether or not the gripping operation has ended (S314). If the grasping motion has ended (S314: Yes), the operation of the tactile information estimation device 1 ends.

On the other hand, if the gripping motion has not ended (S314: No), the gripping unit 124 feeds back information on the gripping state to the gripping position determining unit 116 and/or the object property estimating unit 118 (S316). The information to be fed back is, for example, information as to whether or not the object is stably gripped, or information such as sensing information of the tactile sensor when the grip section 124 is provided with the tactile sensor.

Next, the gripping position determining unit 116 and/or the object property estimating unit 118 updates various information necessary for controlling the gripping unit 124 based on the feedback information (S318).

For example, the gripping position determination unit 116 determines the gripping position from the shape of the object to be gripped, but updates the gripping position if this determination is not appropriate. The gripping position may be updated using, for example, reinforcement learning.

When using reinforcement learning, as an example, a reward is set for the stability of grasping, and learning is performed by MDP (Markov Decision Process). The reward may be set according to the load condition of the object to be grasped when lifted a predetermined distance, as in the above-described embodiment, or if the grasping unit 124 is provided with a tactile sensor, a tactile sensation may be set. It may be set according to the state of the sensor. A Semi-Markov decision process may be used instead of MDP. If it is desired to use information on a portion of the object to be grasped that cannot be sensed by the visual sensor 100, a partially observable Markov decision process may be used. Reinforcement learning methods are not limited to those described above, and any method that enables appropriate learning may be used.

This reinforcement learning may be performed not only for determining the gripping position but also for determining the gripping force. For example, the object property estimation unit 118 updates object property information by reinforcement learning based on feedback information from the grip unit 124 . The gripping force determining unit 120 updates the gripping force based on the object property information updated by the object property estimating unit 118 .

This information update may be reflected in various models generated by the model generation unit 106 . That is, the network may be updated by using information obtained by reinforcement learning as teacher data for the generated model. In this case, when a further unknown object is to be grasped, the visual-tactile feature quantity extraction unit 110, the object property estimation unit 118, or the gripping position estimation unit 114 extracts the visual-tactile feature quantity, estimates the object property, Alternatively, it is possible to improve the accuracy of estimating the gripping position.

As described above, according to the present embodiment, by updating the gripping position and gripping force, it is possible to appropriately update the gripping position and gripping force when the object to be gripped starts to be gripped. It is also possible to appropriately update the gripping force during gripping. By appropriately updating the gripping position and gripping force, it becomes possible to grip the object more precisely.

The gripping unit 124 does not necessarily have to be provided in the tactile information estimation device 1 , and may be an external device such as a gripper connected to the tactile information estimation device 1 . Further, the feedback information is not transmitted from the gripping unit 124, but the visual information of the gripping unit 124 or an external device such as a gripper is sensed by the visual sensor 100, and the feedback information is transmitted based on the visual information. You may make it

(Fourth embodiment)
In the above-described embodiment, the grasping unit 124 is provided, but the grasping information acquisition unit 112 in the above-described second embodiment may also serve as the grasping unit 124 . FIG. 11 is a block diagram showing a configuration that includes a gripping section 124 and that the gripping section 124 also functions as the gripping information acquisition section 112 .

As shown in FIG. 11 , the grip portion 124 includes a tactile sensor 102 and a visual sensor 100 . The visual sensor 100 need not be provided on the grip portion 124 and may be provided separately from the grip portion 124 . As another example, the tactile information estimation device 1 may further include a visual sensor 100 different from the visual sensor 100 provided in the grip portion 124 .

By using the same device as the gripping unit 124 that finally grips the object and the gripping information acquisition unit 112 that includes the tactile sensor 102 that acquires tactile information, the estimation and determination of the gripping force and the gripping position can be facilitated. It is possible to do this precisely.

In the tactile information estimation device 1 in each embodiment as shown in FIGS. 1, 5, 8, and 11, control of data input from the visual sensor 100 and the tactile sensor 102, control of model generation in the model generation unit 106, Visual tactile feature amount extraction control in the visual tactile feature amount extraction unit 110, estimation control in the grip position estimation unit 114 and object property estimation unit 118, determination control in the grip position determination unit 116 and grip force determination unit 120, gripping All or part of the control by the control unit 122 may be performed collectively or individually by a control unit (not shown). The controller may be analog, digital, or a control circuit implemented by FPGA or the like.

In all of the above descriptions, at least part of the tactile information estimation device 1 may be configured by hardware, or may be configured by software, and may be implemented by a CPU or the like through software information processing. In the case of software, the tactile information estimation device 1 and a program that implements at least some of its functions are stored in a storage medium such as a flexible disk or CD-ROM, and are read and executed by a computer. may The storage medium is not limited to a detachable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk device or a memory. That is, information processing by software may be specifically implemented using hardware resources. Further, the software processing may be implemented in a circuit such as an FPGA and executed by hardware. Generation of the learning model and processing after inputting the learning model may be performed using an accelerator such as a GPU, for example.

Also, the data estimation model according to this embodiment can be used as a program module that is part of artificial intelligence software. That is, the CPU of the computer performs operations based on the model stored in the storage unit and outputs the result.

The format of the image, which is the visual information input in the above-described embodiment, may be any format such as raw data, PNG format, etc., as long as the texture information and shape information can be expressed appropriately.

Additions, effects, or various modifications of the present invention may be conceived by those skilled in the art based on the entire description above, but aspects of the present invention are not limited to the individual embodiments described above. do not have. Various additions, changes, and partial deletions are possible without departing from the conceptual idea and spirit of the present invention derived from the content defined in the claims and equivalents thereof.

1: tactile information estimation device, 100: visual sensor, 102: tactile sensor, 104: input unit, 106: model generation unit, 108: output unit, 110: visual and tactile feature amount extraction unit, 112: grip information acquisition unit, 114 116: grip position determination unit 118: object property estimation unit 120: grip force determination unit 122: grip control unit 124: grip unit

Claims (26)

an end effector that manipulates the first object;
a control unit;
The control unit
obtaining information used for manipulating the first object by inputting at least visual information of the first object into a neural network;
controlling the end effector to manipulate the first object based on the obtained information;
The neural network is trained using visual information of a second object and tactile information of the second object,
the visual information of the first object includes texture information of the surface of the first object;
the visual information of the second object includes texture information of the surface of the second object;
the neural network is trained such that tactile information of the second object is generated by inputting visual information of the second object into the neural network ;
The information is obtained based on a feature amount that links the visual information of the first object and the tactile information of the first object, which is generated using the neural network.
Object manipulation device. The information is obtained based on an output from an intermediate layer of the neural network,
The object manipulation device according to claim 1. The information is obtained based on visual and tactile features generated using the neural network,
The object manipulation device according to claim 1 or 2. The control unit outputs a result of mapping the visual-tactile feature amount to a two-dimensional or three-dimensional space.
The object manipulation device according to claim 3. The neural network is a model generated using an autoencoder trained to output tactile information of the second object when visual information of the second object is input,
The object manipulation device according to any one of claims 1 to 4. The neural network is an encoder generated using an autoencoder trained to output tactile information of the second object upon input of visual information of the second object,
The object manipulation device according to any one of claims 1 to 4. The information includes at least either tactile information of the first object or characteristic information of the first object,
The object manipulation device according to any one of claims 1 to 6. The control unit determines a gripping force for gripping the first object with the end effector based on the acquired information.
The object manipulation device according to any one of claims 1 to 7. the information is a control signal for the end effector;
The object manipulation device according to any one of claims 1 to 6. The control unit acquires the information by inputting at least visual information of the first object into the neural network while the end effector is not operating the first object.
The object manipulation device according to any one of claims 1 to 9. The neural network is learned using tactile information of the second object obtained by a tactile sensor,
The object manipulation device according to any one of claims 1 to 10. the tactile information of the second object includes a plurality of pressure information;
The object manipulation device according to any one of claims 1 to 11. The neural network is learned using a time series of tactile information of the second object,
The object manipulation device according to any one of claims 1 to 12. The control unit estimates a gripping position of the first object based on visual information of the first object.
The object manipulation device according to any one of claims 1 to 13. The control unit acquires the information based on the estimated gripping position of the first object.
15. An object manipulation device according to claim 14. further comprising a tactile sensor that acquires tactile information of the first object;
The object manipulation device according to any one of claims 1 to 15. The information is information acquired by inputting visual information of the first object and tactile information of the first object acquired by the tactile sensor into the neural network.
17. An object manipulation device according to claim 16. The control unit updates the operation method of the first object by the end effector based on the tactile information of the first object acquired by the tactile sensor.
The object manipulation device according to claim 16 or 17. The control unit updates at least one of the gripping position of the first object and the gripping force of the end effector.
19. An object manipulation device according to claim 18. The control unit performs the update using reinforcement learning.
The object manipulation device according to claim 18 or 19. wherein the end effector is provided with a visual sensor that acquires visual information of the first object;
The object manipulation device according to any one of claims 1 to 20. the visual information of the first object is image information of the first object;
wherein the visual information of the second object is image information of the second object;
An object manipulation device according to any one of claims 1 to 21. the visual information of the first object includes depth information of the first object;
wherein the visual information of the second object includes depth information of the second object;
An object manipulation device according to any one of claims 1 to 22. Manipulating the first object is gripping the first object,
The end effector is a gripper,
An object manipulation device according to any one of claims 1 to 23. The control unit
obtaining information used for manipulating the first object by inputting at least visual information of the first object into a neural network;
controlling an end effector to manipulate the first object based on the obtained information;
An object manipulation method comprising:
The neural network is trained using visual information of a second object and tactile information of the second object,
the visual information of the first object includes texture information of the surface of the first object;
the visual information of the second object includes texture information of the surface of the second object;
the neural network is trained such that tactile information of the second object is generated by inputting visual information of the second object into the neural network;
The information is obtained based on a feature amount that links the visual information of the first object and the tactile information of the first object, which is generated using the neural network.
object manipulation method. When executed by one or more processors,
obtaining information used for manipulating the first object by inputting at least visual information of the first object into a neural network;
controlling an end effector to manipulate the first object based on the obtained information;
A program for executing a method,
The neural network is trained using visual information of a second object and tactile information of the second object,
the visual information of the first object includes texture information of the surface of the first object;
the visual information of the second object includes texture information of the surface of the second object;
the neural network is trained such that tactile information of the second object is generated by inputting visual information of the second object into the neural network;
The information is obtained based on a feature amount that links the visual information of the first object and the tactile information of the first object, which is generated using the neural network.
program.

Images