JP2020119024A - Information processing device, information processing method, and recording medium - Google Patents

Abstract

To improve recognition accuracy for a part of an object.SOLUTION: An information processing device comprises and a recognition unit which recognizes an object region corresponding to a part of an object in visual information, on the basis of the condition of each region in the visual information, and the condition of the part of the object based on detection data detected by a detection unit included in the object contained in the visual information. This technology is applicable to, for example, a smartphone, a tablet computer, etc.SELECTED DRAWING: Figure 2

Description

The present technology relates to an information processing device, an information processing method, and a recording medium, and particularly to an information processing device, an information processing method, and a recording medium that improve the recognition accuracy of a part of an object.

In recent years, face recognition technology for recognizing a human face in an image has become widespread (for example, see Patent Document 1).

JP, 2017-91059, A

However, the face recognition technology cannot recognize a part other than the human face. In general, an object recognition technique using an image such as a face recognition technique is good at recognizing a portion having many visual features such as a face, but is not good at recognizing a portion having few visual features. I am not good at it.

The present technology has been made in view of such a situation, and is to improve the recognition accuracy of a part of an object.

The information processing device according to one aspect of the present technology is based on the state of each area in the visual information and the state of a part of the object based on the detection data detected by the detection unit included in the visual information included in the object. , A recognition unit that recognizes a target area corresponding to a part of the object in the visual information.

An information processing method according to an aspect of the present technology is that an information processing device includes a state of each region in visual information, and a part of the object based on detection data detected by a detection unit included in the visual information. A target area corresponding to a part of the object in the visual information is recognized based on the state of.

The program recorded on the recording medium according to one aspect of the present technology is a state of each area in visual information, and a part of the object based on detection data detected by a detection unit included in the visual information included in the object. The computer is caused to execute a process of recognizing a target area corresponding to a part of the object in the visual information, based on the state.

In one aspect of the present technology, based on a state of each area in visual information and a state of a part of the object based on detection data detected by a detection unit included in the visual information included in the object, the visual A target area corresponding to a portion of the object in the information is recognized.

It is a block diagram showing an embodiment of an information processing system to which this art is applied. It is a block diagram which shows the structural example of an information processing part. It is a figure which shows the structural example and installation example of an imaging|photography part. It is a flow chart for explaining recognition processing. It is a figure which shows the example of the calculation result of a cross correlation coefficient. It is a figure for demonstrating the example of recognition of a target area. It is a figure for demonstrating the example of recognition of a target area. It is a figure for demonstrating the example of recognition of a target area. It is a figure for demonstrating the example of recognition of a target area. It is a figure for demonstrating the 1st modification of a recognition process. It is a figure for demonstrating the 2nd modification of a recognition process. It is a figure for demonstrating the 3rd modification of recognition processing. FIG. 13 is a diagram illustrating a configuration example of a computer.

Hereinafter, modes for carrying out the present technology will be described. The description will be given in the following order.
1. Embodiment 2. Modification 3. Other

<<1. Embodiment>>
First, an embodiment of the present technology will be described with reference to FIGS. 1 to 9.

<Example of configuration of information processing system 11>
The information processing system 11 is a system that performs object recognition processing. In particular, the information processing system 11 can recognize not only the entire object but also a part of the object.

The information processing system 1 includes a detection unit 21-1 to a detection unit 21-m, a photographing unit 22, and an information processing unit 23.

The detection unit 21-1 is provided on the target object 12-1 and detects data regarding the state of a part of the target object 12-1 (hereinafter referred to as the target part). The target portion is a portion to be detected by the detection unit 21-1 and includes the detection unit 21-1.

The detection unit 21-1 transmits the detection data indicating the detection result to the information processing unit 23. Note that wired communication or wireless communication of any method can be used to transmit the detection data.

Similarly to the detection unit 21-1, the detection units 21-2 to 21-m also detect data regarding the states of the target portions of the target objects 12-2 to 12-m, respectively, and detect the detection results. The data is transmitted to the information processing unit 23.

Note that, hereinafter, when it is not necessary to individually distinguish the object 12-1 to the object 12-m and the detection unit 21-1 to the detection unit 21-m, they are simply referred to as the object 12 and the detection unit 21. ..

The image capturing unit 22 captures an image of the object 12 and a space including the non-objects 13-1 to 13-n (hereinafter, referred to as an imaging space). That is, the image capturing unit 22 captures an image (hereinafter, referred to as a captured image) including the target object 12-1 to the target object 12-m and the non-target object 13-1 to the non-target object 13-n in the shooting space. .. The image capturing unit 22 transmits captured image data, which is data including a captured image, to the information processing unit 23. Note that wired communication or wireless communication of any method can be used to transmit the captured image data.

The non-target object 13-1 to the non-target object 13-n are objects other than the target object 12 existing in the imaging space, and their types are not limited. Further, hereinafter, when it is not necessary to individually distinguish the non-target object 13-1 to the non-target object 13-n, they are simply referred to as the non-target object 13.

The information processing unit 23 is configured by an information processing device such as a personal computer, a smartphone, a tablet computer, a game terminal, or the like. The information processing unit 23 performs a recognition process of each target object 12 based on the detection data from each detection unit 21 and the captured image data from the imaging unit 22. More specifically, the information processing unit 23 performs a recognition process of a region (hereinafter, referred to as a target region) corresponding to a target portion of each target object 12 in the captured image. The information processing unit 23 outputs data indicating the recognition result of the target area of each target object 12.

The type of the target object 12 is not particularly limited as long as it is an object to which the detection unit 21 can be provided. The object 12 may be a moving object or a stationary object.

Furthermore, the state to be detected by the detection unit 21 is not particularly limited as long as it can be detected even in a captured image. For example, one or more of the position, orientation, movement, attribute, etc. of the target portion are detected. As the movement of the target portion, for example, velocity, acceleration, angular velocity, angular acceleration, etc. of the target portion are assumed. As the attribute of the target portion, for example, the shape, color, material, temperature, etc. of the target portion are assumed.

Further, the position where the detection unit 21 is installed is not particularly limited as long as the data regarding the state of the target portion can be detected. For example, the detection unit 21 may be provided on either the surface or the inside of the object 12. Further, for example, when the object 12 is a person, the detection unit 21 may be configured by a wearable device including various sensors, and the object 12 may be equipped with the detection unit 21.

Further, the captured image may be of any type as long as it can detect the same type of state as the state to be detected by each detection unit 21.

In addition, each target 12 does not necessarily have to be the same type of object. Further, the types of states to be detected by the detection units 21 do not necessarily have to be the same.

<Example of Configuration of Information Processing Unit 23>
FIG. 2 is a block diagram showing a configuration example of the information processing unit 23 of the information processing system 11 of FIG.

The information processing unit 23 includes a reception unit 51, a storage unit 52, a state detection unit 53, and a recognition unit 54. The state detection unit 53 includes a state detection unit 53A and a state detection unit 53B.

The reception unit 51 is capable of communication corresponding to the communication system of each detection unit 21 and the communication system of the imaging unit 22. The reception unit 51 receives the detection data transmitted from each detection unit 21 and the captured image data transmitted from the imaging unit 22, and stores the data in the storage unit 52.

The state detection unit 53A detects the state of each unit in the captured image based on the captured image data stored in the storage unit 52. The state detection unit 53A supplies data indicating the detection result to the recognition unit 54.

The state detection unit 53B detects the state of the target portion of each target object 12 based on the detection data stored in the storage unit 52. The state detection unit 53B supplies data indicating the detection result to the recognition unit 54.

The recognition unit 54 performs recognition processing of a target area corresponding to the target portion of each target 12 in the captured image based on the state of each unit in the captured image and the state of the target portion of each target 12. The recognition unit 54 outputs data indicating the recognition result of the target area of each target object 12.

<Structural Example and Installation Example of Imaging Unit 22>
FIG. 3 is a schematic diagram showing a specific configuration example and installation example of the imaging unit 22.

Here, an example in which the image capturing unit 22 includes the depth sensor 101 is shown.

The depth sensor 101 is installed, for example, on the ceiling in the shooting space, and shoots while looking down on the shooting space. Further, the depth sensor 101 includes an IMU (Inertial Measurement Unit), and the relationship between the optical axis direction of the depth sensor 101 and the installation direction of the IMU is known. The IMU includes, for example, an acceleration sensor and an angular velocity sensor, and further includes a geomagnetic sensor and the like as necessary. Therefore, the depth sensor 101 can detect the gravity direction with respect to the optical axis using the IMU. The depth sensor 101 transmits captured image data (depth image) obtained by capturing and captured image data including a detection result of the gravity direction to the information processing system 11.

Note that, hereinafter, a case where the image capturing unit 22 includes the depth sensor 101 as in this example will be described.

Further, hereinafter, a case will be described in which each detection unit 21 includes an IMU, the state detection unit 53A detects the posture of each unit in the captured image, and the state detection unit 53B detects the posture of the target portion of each target object 12. ..

<Recognition processing>
Next, the recognition processing executed by the information processing system 11 will be described with reference to the flowchart in FIG.

Note that this process starts, for example, when the information processing unit 23 is powered on, and ends when the information processing unit 23 is powered off.

In step S1, the information processing system 11 starts acquisition of captured image data and detection data.

Specifically, the depth sensor 101 starts photographing the photographing space and detecting the gravity direction. The depth sensor 101 transmits captured image data including the obtained captured image (depth image) and the detection result of the gravity direction to the information processing unit 23.

Each detection unit 21 starts detection of the acceleration and angular velocity of the target portion of each target object 12, and the direction of gravity. Each detection unit 21 transmits the detection data indicating the detection result to the information processing unit 23.

The receiving unit 51 of the information processing unit 23 receives the captured image data and the detection data, and stores them in the storage unit 52.

In step S2, the state detection unit 53A detects the state of each unit in the captured image.

Specifically, the state detection unit 53A divides the captured image into a plurality of areas.

Any method can be used to divide the area of the captured image.

For example, the state detection unit 53A detects a feature point in a captured image, tracks the detected feature point over a plurality of frames, and detects a region including a feature point that makes similar movements, thereby detecting a plurality of captured images. Divide into areas. The type of feature points is not particularly limited, but, for example, corners, human face parts, and the like are used.

For example, the state detection unit 53A divides the captured image into a plurality of regions by tracking each part in the captured image over a plurality of frames using an optical flow and detecting a region having the same movement.

For example, the state detection unit 53A divides the captured image into a plurality of segmentation regions by using an arbitrary method such as semantic segmentation.

For example, the state detection unit 53A uses bone tracking to track the movement of the skeleton of a person, a robot, or the like in the captured image, thereby dividing the person, the robot, or the like in the captured image into parts.

Next, the state detection unit 53A calculates a normal vector of each area (hereinafter, referred to as a divided area) in the captured image. Since the captured image is a depth image and each pixel has information in the depth direction, it is easy to calculate the normal vector of each divided area. The state detection unit 53A calculates the angle θc(t) of the calculated normal vector with respect to the direction of gravity using the following equation (1).

θc(t)=arccos((n·g)/|n||g|) (1)

In addition, n shows a normal vector and g has shown the gravity vector which shows a gravity direction.

In this way, the attitude (angle θc(t)) of each divided area at time t is detected.

The state detection unit 53A supplies the recognition unit 54 with data indicating the detection result of the posture of each divided area in the captured image.

In step S3, the state detection unit 53B detects the state of a part of each object 12.

For example, the state detection unit 53B calculates the posture of the target portion of each target 12 based on the detection result of the angular velocity and acceleration of the target portion that is a part of each target 12. For example, the state detection unit 53B calculates an angle θo(t) with respect to the gravity direction of the target portion of each target object 12 at time t.

The state detection unit 53B supplies the recognition unit 54 with data indicating the detection result of the posture of the target portion of each target object 12.

In step S4, the recognition unit 54 obtains the correlation between the state of each part in the captured image and the state of a part of each target object 12. Specifically, for example, the recognition unit 54 detects the posture (angle θc(t)) of each divided area in the captured image detected by the state detection unit 53A, and the target object 12 detected by the state detection unit 53B. Correlation coefficients are calculated for all combinations with the orientation (angle θo(t)) of the target portion of.

For example, the recognition unit 54 selects one of the divided areas in the captured image and one of the target portions of each target object 12. The recognition unit 54 calculates the cross-correlation coefficient r(m) between the posture of the selected divided area and the posture of the selected target portion using the following equation (2).

Note that f(n) represents a sample value (detection value) of the posture of the n-th target portion from the reference time t0. h(n+m) indicates a sample value (detection value) of the posture of the n+m-th divided area from the reference time t0. N indicates the total number of samples used for calculating the cross-correlation coefficient r(m). Therefore, the cross-correlation coefficient between the sample values of the N postures of the target portion and the sample values of the N postures of the divided area m samples after the target portion is calculated by the equation (1).

The recognizing unit 54 sets a time that is a predetermined time before the current time t to the reference time t0, sets the shift amount m to 0, and sets the mutual relationship between the posture of the selected divided region and the posture of the target portion. Calculate the number r(0). Then, the recognizing unit 54 calculates the cross-correlation coefficient r(m) between the posture of the selected divided area and the posture of the target portion while shifting the shift amount m one by one to a predetermined maximum value.

FIG. 5 shows an example of the calculation result of the cross-correlation coefficient r(m). The horizontal axis represents the shift amount m, the depth axis represents the time t, and the height axis represents the cross-correlation coefficient r(m).

The curve in the graph shows the waveform of the cross-correlation coefficient r(m) at each time t, and the circle on the waveform shows the peak of the cross-correlation coefficient r(m).

For example, even if there is a time difference between the timing of detecting the posture of the divided area in the captured image and the posture of the target portion of the target object 12, by adjusting the shift amount m as shown in FIG. The cross-correlation coefficient r(m) between the two can be accurately obtained.

Then, the recognition unit 54 obtains the maximum value (hereinafter, referred to as the maximum cross-correlation coefficient) of the calculated cross-correlation coefficient r(m), for example.

The recognition unit 54 calculates the maximum cross-correlation coefficient for all combinations of each divided region in the captured image and the target portion of each target 12 by the same process.

In step S5, the recognition unit 54 recognizes the target area.

For example, the recognition unit 54 extracts a divided area in which the maximum cross-correlation coefficient between the target portion of the target object 12-1 and the target portion continues for a predetermined time or more and is a predetermined threshold value or more from the captured image. Then, the recognition unit 54 recognizes the extracted region as a target region corresponding to the target portion of the target object 12-1. As a result, the divided area in which the posture and the target portion of the target object 12-1 are correlated is recognized as the target area. For example, a divided area having the same posture as the target portion of the object 12-1 or a target portion of the object 12-1 or a divided area having the same movement as the detection unit 21-1 is recognized as the target area. It

In addition, when a plurality of divided areas are extracted, that is, there are a plurality of divided areas in which the maximum cross-correlation coefficient with the target portion of the object 12-1 continues for a predetermined time or more and is a predetermined threshold or more. In this case, for example, the recognition unit 54 recognizes the divided area having the maximum average value of the maximum cross-correlation coefficients as the target area. At this time, when the divided area around the divided area where the average value of the maximum cross-correlation coefficient is the maximum is also extracted, for example, the recognition unit 54 sets the continuous area including the divided area around the divided area as the target area. You may recognize.

Further, when there is no divided area in which the maximum cross-correlation coefficient between the target portion of the target object 12-1 and the maximum cross-correlation coefficient continues for the predetermined time or longer, for example, the recognition unit 54 determines that the target object 12-1 It is recognized that the target portion of -1 does not exist in the captured image (exists outside the capturing space of the depth sensor 101).

The recognition unit 54 performs the same process on the objects 12-2 to 12-m. As a result, the target area corresponding to the target portion of each target 12 is recognized in the captured image.

The target area is not always fixed and may change depending on the movement of the target object 12.

For example, in FIG. 6, when a person is the target object 12, the ring type device 203 as the detection unit 21 including the IMU is provided between the second joint and the third joint of the index finger 202 of the hand 201 of the target object 12. An example of being attached is shown.

For example, when the joint of the index finger 202 is moved, the change in the posture detected based on the detection data from the ring-shaped device 203 is changed in the posture of the portion between the second joint and the third joint of the index finger 202. Almost matches. In this case, the portion between the second joint and the third joint of the index finger 202 is the target portion, and as shown in FIG. 7, the area 211 including the target portion is recognized as the target area.

On the other hand, for example, when the entire hand 201 is moved without moving the joint of the index finger 202, the change in the posture detected based on the detection data from the ring-shaped device 203 is substantially the same as the change in the posture of the entire hand 201. To do. In this case, the entire hand 201 is the target portion, and the region 212 including the entire hand 201 is recognized as the target region, as shown in FIG.

For example, FIG. 9 illustrates an example in which the earphone type device 233 as the detection unit 21 including the IMU is attached to the ear 232 of the head 231 of the target 12 when the person is the target 12. ..

For example, when only the head 231 is moved, the change in the posture detected based on the detection data from the earphone type device 233 substantially matches the change in the posture of the head 231. In this case, the head 231 is the target portion, and the area 211 including the head 231 is recognized as the target area.

On the other hand, for example, when the target object 12 walks and moves, the change in the posture detected based on the detection data from the earphone device 233 substantially matches the change in the posture of the part of the target object 12 excluding the arms and legs. To do. In this case, the part of the target object 12 excluding the arms and legs is the target part, and the region including the target part is recognized as the target region.

Note that, for example, when another object or a part of another object connected to the target portion of the target object 12 has the same posture or the same movement as the target portion, the target portion and the other object or another object A region including a part of is recognized as the target region. For example, when the target object 12 is a person and the target portion is a hand, an area including the hand and an object held in the hand or a part of the object held in the hand may be recognized as the target area.

The recognition unit 54 outputs data indicating the recognition result of the target area of each target object 12.

After that, the process returns to step S2, and the processes of steps S2 to S5 are repeatedly executed.

As described above, the recognition accuracy of a part (target part) of each target object 12 is improved.

For example, even if a marker (for example, a retroreflective marker, a color marker, an invisible marker, etc.) is not attached to a visually uncharacteristic portion of the object 12, the portion can be accurately recognized. For example, when the object 12 is an object without a monochrome pattern, it is possible to accurately recognize a part of the object 12. Specifically, for example, when a robot such as a manipulator having a plurality of joints is monochromatic and has no pattern, it is possible to accurately recognize a part of the skeleton of the robot. Further, as described above, the addition of the marker is unnecessary, and since the IMU can be built in the object 12, it is possible to recognize a part of the object 12 without spoiling the appearance of the object 12.

For example, it is possible to accurately recognize a part of the specific target object 12 from objects having a similar appearance. For example, when there are a plurality of objects 12 having the same appearance, it is possible to accurately recognize a part of the specific object 12 among them. Specifically, for example, when the same product is displayed, it is possible to accurately recognize a part of the specific product.

Further, even if the object 12 is an object that makes complicated movements such as an animal or an articulated robot, a part of the object 12 can be accurately recognized. Further, even when the object 12 is not a rigid body but deforms, a part of the object 12 can be accurately recognized.

Furthermore, for example, by including information about the object 12 or the target portion in the detection data, it becomes possible to recognize not only the target portion but also the information about the object 12 or the target portion. For example, it becomes possible to recognize the attribute such as the name of the target object 12 or the target portion.

Further, for example, even if the target portion is hidden by another object, the region in which the target portion exists in the captured image can be estimated based on the information about the target object 12 or the target portion. For example, in the above-described processing, when the hand is moved while grasping the target object 12, it is assumed that the area including the hand is recognized as the target area. In this case, for example, it is possible to presume that the target object 12 is present in the hand based on the information about the target object 12 or the target portion.

Furthermore, for example, when recognizing a target portion based on velocity or acceleration, it is difficult to recognize (the target area corresponding to) the target portion unless the target portion continuously moves for a certain period of time or longer. In addition, if the movement of the target portion is small, the recognition accuracy of the target portion decreases. Therefore, for example, when the target object 12 is an operation device such as a game controller, the target object 12 needs to be largely moved, which increases the burden on the user. Further, when many moving objects are present in the captured image, the load for detecting the movement of each moving object increases. On the other hand, if the detection accuracy of the movement of the moving body is reduced in order to reduce the load, the recognition accuracy of the target portion, which is one of the moving bodies, decreases. Further, depending on the position of the image capturing unit 22, a direction in which the movement of the target portion is difficult to detect occurs, and when the target portion moves in that direction, the recognition accuracy of the target portion may decrease.

On the other hand, when the target portion is recognized based on the orientation of the target portion as in the information processing system 11, the target portion can be accurately recognized even when the target portion is stationary. However, when there are many objects having the same posture as the target portion, for example, when many objects including the target object 12 are placed on the same desk, it may be difficult to recognize the target portion. However, even in this case, if the target portion moves even a little, the target portion can be accurately recognized. Further, the position of the image capturing unit 22 has little influence on the detection accuracy of the posture of the target portion. Therefore, regardless of the position of the imaging unit 22, the recognition accuracy of the target portion is kept good.

Further, as described above with reference to FIG. 5, even if the detection unit 21 and the image capturing unit 22 are not synchronized, the postures of the divided regions in the captured image and the postures of the target portion of the target object 12 are not changed. It is possible to calculate the correlation coefficient and recognize the target portion. Furthermore, it suffices to detect the orientation of the target portion in the gravity direction, and it is not necessary to detect the orientation of the target portion in the world coordinate system. Therefore, in the information processing unit 23, it is possible to reduce the equipment and load necessary for detecting the posture.

<<2. Modification>>
Hereinafter, modified examples of the embodiment of the present technology described above will be described.

In the above description, an example in which the depth image is used as a captured image to detect the orientation of each region in the captured image has been shown, but another method is used to detect the orientation of each region in the captured image, and other methods. The visual information may be used to detect the posture of each area in the visual information.

For example, the posture of each part of the object 12 may be detected based on the image of the object 12 registered in advance and the image of the object 12 in the captured image.

For example, the posture of each area in the visual information may be detected using three-dimensional visual information such as a point cloud.

Further, in the above description, an example in which the target region is recognized based on the correlation between the posture of each region in the captured image and the posture of the target portion of the target object 12 has been shown. If the state is detectable by both of the detection data of the unit 21, it is possible to use the correlation of any state.

For example, the IMU can be used to detect the velocity, acceleration, angular velocity, and position in addition to the angle of the object. Therefore, for example, it is possible to use a correlation of one or more states among an angle, a velocity, an acceleration, an angular velocity, and a position.

In addition, FIG. 10 illustrates an example of performing the recognition process of the object 12a that performs the rotational movement.

For example, the object 12a includes a rotating unit 251, and a rotary encoder (not shown) that detects the amount of rotation of the rotating unit 251.

The photographing unit 22 includes a camera 252, and the camera 252 can photograph the target object 12a.

The state detection unit 53A detects the angle and the angular acceleration of each area in the captured image from the camera 252.

The state detecting unit 53B detects the angle and the angular acceleration of the rotating unit 251 based on the detection data from the rotary encoder.

The recognition unit 54, based on the correlation between the angle and the angular acceleration of each region in the captured image detected by the state detection unit 53A, and the angle and the angular acceleration of the rotation unit 251 detected by the state detection unit 53B, The target area corresponding to the rotating unit 251 is recognized.

FIG. 11 illustrates an example of performing the recognition process of the object 12b that performs translational movement.

For example, the object 12b includes a movable portion 262 that moves in translation on the base 261 and a linear encoder (not shown) that detects the position of the movable portion 262.

The image capturing unit 22 includes a camera 263, and the camera 263 can capture the object 12b.

The state detection unit 53A detects the position, speed, and acceleration of each area of the captured image from the camera 263.

The state detection unit 53B detects the position, speed, and acceleration of the movable unit 262 based on the detection data from the linear encoder.

The recognition unit 54 recognizes the position, velocity, and acceleration of each area in the captured image detected by the state detection unit 53A and the position, velocity, and acceleration of the movable unit 262 detected by the state detection unit 53B. The target area corresponding to the movable portion 262 is recognized based on the correlation.

FIG. 12 shows an example in which the target region recognition processing is performed based on the temperature of the target object 12c.

The object 12c includes a temperature sensor 301.

The imaging unit 22 includes a thermal sensor 302. The thermal sensor 302 detects the temperature of the surface of the object 12c and generates a thermal image showing the distribution of the detected temperature.

The state detection unit 53A detects the temperature of each area in the thermal image from the thermal sensor 302.

The state detection unit 53B detects the temperature of the object 12c near the temperature sensor 301 based on the detection data from the temperature sensor 301.

The recognition unit 54 performs thermal detection based on the correlation between the temperature of each region in the thermal image detected by the state detection unit 53A and the temperature near the temperature sensor 301 of the object 12c detected by the state detection unit 53B. A target area including the vicinity of the temperature sensor 301 of the target object 12c is recognized in the image.

For example, when the object 12c is an object (for example, a drone) including a movable portion (for example, a motor), the temperature sensor 301 is provided near the movable portion. Accordingly, it is possible to recognize the area including the movable portion as the target area based on the temperature change due to the heat generated by the movement of the movable portion.

For example, when the object 12c is tableware, the temperature sensor 301 is built in the tableware. Then, for example, it is possible to recognize the region including the temperature sensor 301 of the tableware as the target region based on the temperature change when the dish is served with the dish.

For example, when the object 12c includes a thermoelectric element such as a Peltier element, the temperature sensor 301 is provided near the thermoelectric element. For example, when the object 12c is a tactile device such as a VR (Virtual Reality) glove, a thermoelectric element is provided to enhance the user experience by using the change in temperature. Then, for example, the region including the thermoelectric element can be recognized as the target region based on the temperature change generated by the thermoelectric element.

If the temperature change of the target portion is small, a thermoelectric element may be provided in the target portion to actively change the temperature of the target portion. This makes it possible to accurately recognize a portion of the object 12 in which the temperature change is small in the thermal image.

Further, for example, when the object 12 includes a deformable device such as a soft actuator, the part of the object 12 including the deformable device can be recognized.

For example, the deformation device detects its own shape and transmits detection data indicating the detection result.

The state detection unit 53A detects the shape of each area in the captured image.

The state detection unit 53B detects the shape of the deformable device based on the detection data.

The recognition unit 54, based on the correlation between the shape of each region in the captured image detected by the state detection unit 53A, and the shape of the portion including the deformation device of the object 12 detected by the state detection unit 53B, A target area including a portion of the object 12 including the deformation device is recognized in the captured image.

Further, for example, the posture of an object is generally represented by a quaternion, but it is possible to use the correlation of the posture change amount of the quaternion, for example. For example, it is possible to use the correlation based on the change in the angular velocity over time or the correlation based on the change in the direction of the outer product vector. In this case, the correlation may be obtained based on the angle in the three-dimensional direction, or the correlation may be obtained based on the angle in the two-dimensional direction.

Note that it is possible to use a correlation coefficient other than the above-described cross-correlation coefficient or a method other than the correlation coefficient for the calculation of the correlation.

Further, in the above description, the example in which the IMU is provided in the depth sensor 101 to detect the gravity direction has been described, but the gravity direction may be detected by a different method.

For example, the weight direction may be estimated based on the wall or floor in the captured image. Further, for example, when the depth sensor 101 is fixed, the gravity direction with respect to the optical axis of the depth sensor 101 may be measured in advance and the measurement result may be given to the information processing unit 23.

Further, the type and number of image sensors included in the image capturing unit 22 can be changed according to the state to be detected. For example, in addition to the depth sensor 101 described above, an image sensor that detects normal visible light, an infrared sensor that can detect infrared rays, an ultrasonic sensor, a polarization sensor that can detect a normal vector, and the like can be used. Is.

Furthermore, it is possible to use visual information other than the captured image. For example, it is possible to use the above-mentioned thermal image, point cloud, magnetic field image detected by a magnetic sensor, or the like.

Further, for example, the state detecting unit 53A may be provided in the image capturing unit 22, and the image capturing unit 22 may detect the state of each region in the captured image and transmit the data indicating the detection result to the information processing unit 23. ..

Further, for example, the state detection unit 53B may be provided in the detection unit 21 so that the detection unit 21 detects the state of the target portion based on the detection data and transmits the data indicating the detection result to the information processing unit 23. Good.

Further, for example, one target 12 may be provided with a plurality of detection units 21. Accordingly, it is possible to recognize a plurality of target portions of one target object 12.

Furthermore, the detection unit 21 and the target portion may be separated. That is, the detection unit 21 may detect the state of the portion of the object 12 that is distant from the detection unit 21, and the target region corresponding to the portion that is distant from the detection unit 21 may be recognized.

Further, for example, by recognizing the movement of the target portion of the target object 12, it is possible to realize a tangible user interface using the target portion.

Furthermore, for example, when the state of the entire target 12 can be detected based on the detection data of the detection unit 21, the present technology can be applied to the recognition processing of the entire target 12. For example, when the target object 12 has a simple configuration and the posture of the entire target object 12 is detected based on the detection data of the detection unit 21, it is possible to recognize the target area corresponding to the entire target object 12. ..

<Computer configuration example>
The series of processes described above can be executed by hardware or software. When the series of processes is executed by software, a program forming the software is installed in the computer. Here, the computer includes a computer incorporated in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 13 is a block diagram showing a configuration example of hardware of a computer that executes the series of processes described above by a program.

In a computer 1000, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are connected to each other by a bus 1004.

An input/output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a recording unit 1008, a communication unit 1009, and a drive 1010 are connected to the input/output interface 1005.

The input unit 1006 includes an input switch, a button, a microphone, an image sensor, and the like. The output unit 1007 includes a display, a speaker and the like. The recording unit 1008 includes a hard disk, a non-volatile memory, or the like. The communication unit 1009 includes a network interface or the like. The drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer 1000 configured as described above, the CPU 1001 loads the program recorded in the recording unit 1008 into the RAM 1003 via the input/output interface 1005 and the bus 1004 and executes the program. A series of processing is performed.

The program executed by the computer 1000 (CPU 1001) can be provided by being recorded in the removable medium 1011 as a package medium or the like, for example. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer 1000, the program can be installed in the recording unit 1008 via the input/output interface 1005 by mounting the removable medium 1011 in the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the recording unit 1008. In addition, the program can be installed in the ROM 1002 or the recording unit 1008 in advance.

The program executed by the computer may be a program in which processing is performed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

In this specification, the system means a set of a plurality of constituent elements (devices, modules (parts), etc.), and it does not matter whether or not all the constituent elements are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device housing a plurality of modules in one housing are all systems. ..

Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present technology.

For example, the present technology may have a configuration of cloud computing in which one device is shared by a plurality of devices via a network and processes jointly.

In addition, each step described in the above-described flowcharts can be executed by one device or shared by a plurality of devices.

Further, when one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

<Combination example of configuration>
The present technology may also be configured as below.

(1)
Based on the state of each area in the visual information, and the state of the part of the object based on the detection data detected by the detection unit included in the object included in the visual information, to the part of the object in the visual information An information processing apparatus including a recognition unit that recognizes a corresponding target area.
(2)
The information processing according to (1), wherein the recognition unit recognizes the target area based on a correlation between a state of each area in the visual information and a state of a part of the object based on the detection data. apparatus.
(3)
The information processing apparatus according to (2), wherein the recognition unit recognizes, as the target area, an area in the visual information that has the same attitude as the attitude of a part of the object detected based on the detection data.
(4)
The recognizing unit recognizes, as the target area, an area in the visual information that makes a movement similar to the movement of a part of the object detected based on the detection data. (2) or (3) Information processing device.
(5)
The information processing device according to any one of (1) to (4), wherein the part of the object is a part to be detected by the detection unit.
(6)
The information processing device according to (5), wherein a part of the object is a part including the detection unit.
(7)
The information processing apparatus according to (6), wherein the recognition unit recognizes, as the target region, a region in the visual information that moves in the same manner as the detection unit.
(8)
The information processing device according to any one of (1) to (7), wherein the state includes at least one of a posture, a velocity, an acceleration, an angular velocity, an angular acceleration, a position, a shape, and a temperature.
(9)
The detection unit includes an acceleration sensor and an angular velocity sensor,
The information processing device according to (8), wherein the state includes at least one of a posture, a velocity, an acceleration, an angular velocity, and an angular acceleration.
(10)
The visual information is a thermal image that is an image showing the temperature distribution,
The detection unit includes a temperature sensor,
The information processing device according to (8), wherein the state includes temperature.
(11)
The detection unit includes an encoder,
The information processing device according to (8), wherein the state includes at least one of velocity, acceleration, angular velocity, angular acceleration, and position.
(12)
A portion of the object comprises a deformable device,
The detection unit includes a sensor that detects the shape of the device,
The information processing device according to (8), wherein the state includes a shape.
(13)
The information processing apparatus according to any one of (1) to (12), further including a state detection unit that detects at least one of a state of each area in the visual information and a state of a part of the object.
(14)
Further comprising a receiving unit for receiving the detection data,
The information processing device according to (13), wherein the state detection unit detects the state of a part of the object based on the received detection data.
(15)
The information processing device
Based on the state of each area in the visual information, and the state of the part of the object based on the detection data detected by the detection unit included in the object included in the visual information, to the part of the object in the visual information An information processing method for recognizing a corresponding target area.
(16)
Based on the state of each area in the visual information, and the state of the part of the object based on the detection data detected by the detection unit included in the object included in the visual information, to the part of the object in the visual information A computer-readable recording medium in which a program for executing processing for recognizing a corresponding target area is recorded.

It should be noted that the effects described in the present specification are merely examples and are not limited, and may have other effects.

11 information processing system, 12, 12a, 12b, 12c target object, 13 non-target object, 21 detection unit, 22 imaging unit, 23 information processing unit, 51 reception unit, 53, 53A, 53B state detection unit, 54 recognition unit, 101 depth sensor, 251 rotating part, 252 camera, 262 movable part, 263 camera, 301 temperature sensor, 302 thermal sensor

Claims (16)

Based on the state of each area in the visual information, and the state of the part of the object based on the detection data detected by the detection unit included in the object included in the visual information, to the part of the object in the visual information An information processing apparatus including a recognition unit that recognizes a corresponding target area. The information processing apparatus according to claim 1, wherein the recognition unit recognizes the target area based on a correlation between a state of each area in the visual information and a state of a part of the object based on the detection data. .. The information processing apparatus according to claim 2, wherein the recognition unit recognizes, as the target area, an area in the visual information that has the same attitude as the attitude of a part of the object detected based on the detection data. The information processing apparatus according to claim 2, wherein the recognition unit recognizes, as the target area, an area in the visual information that makes a movement similar to the movement of the part of the object detected based on the detection data. The information processing apparatus according to claim 1, wherein a part of the object is a part to be detected by the detection unit. The information processing device according to claim 5, wherein a part of the object is a part including the detection unit. The information processing apparatus according to claim 6, wherein the recognition unit recognizes an area in the visual information that moves in the same manner as the detection unit as the target area. The information processing apparatus according to claim 1, wherein the state includes at least one of a posture, a velocity, an acceleration, an angular velocity, an angular acceleration, a position, a shape, and a temperature. The detection unit includes an acceleration sensor and an angular velocity sensor,
The information processing apparatus according to claim 8, wherein the state includes at least one of a posture, a velocity, an acceleration, an angular velocity, and an angular acceleration. The visual information is a thermal image that is an image showing the distribution of temperature,
The detection unit includes a temperature sensor,
The information processing apparatus according to claim 8, wherein the state includes temperature. The detection unit includes an encoder,
The information processing apparatus according to claim 8, wherein the state includes at least one of velocity, acceleration, angular velocity, angular acceleration, and position. A portion of the object comprises a deformable device,
The detection unit includes a sensor that detects the shape of the device,
The information processing apparatus according to claim 8, wherein the state includes a shape. The information processing apparatus according to claim 1, further comprising a state detection unit configured to detect at least one of a state of each area in the visual information and a state of a part of the object. Further comprising a receiver for receiving the detection data,
The information processing device according to claim 13, wherein the state detection unit detects a state of a part of the object based on the received detection data. The information processing device
Based on the state of each area in the visual information, and the state of the part of the object based on the detection data detected by the detection unit included in the object included in the visual information, to the part of the object in the visual information An information processing method for recognizing a corresponding target area. Based on the state of each area in the visual information, and the state of the part of the object based on the detection data detected by the detection unit included in the object included in the visual information, to the part of the object in the visual information A computer-readable recording medium in which a program for executing processing for recognizing a corresponding target area is recorded.

Images