JP7647032B2 - DETECTION APPARATUS, PROCESSING APPARATUS, AND PROCESSING PROGRAM - Google Patents

Description

The present invention relates to a detection device, a processing device, a detection method, and a processing program.

For example, as described in Patent Document 1, a technology has been proposed in which an object is imaged using multiple imaging devices, and the resulting images are input to a computer to obtain the three-dimensional shape of the object; however, it is desirable to detect the shape or orientation of all or part of the object (object) with high accuracy.

JP 2010-134546 A

According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives positional information of each point on the surface of an object, and determines feature points of the object from normals at the points using the positional information; and a posture calculation unit that determines the posture of the object from the multiple feature points, wherein the posture calculation unit determines the posture of the object by fitting skeletal information to connecting lines connecting the multiple feature points.
According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives point cloud data indicating positional information of each point on a surface of an object, and determines feature points of the object from normals at points on the point cloud data; a posture calculation unit that determines the posture of the object from the plurality of feature points; and a model generation unit that generates model information including shape information of the object based on the point cloud data, wherein the posture calculation unit determines the posture of the object by fitting skeletal information to the plurality of feature points, and the model generation unit distinguishes between a first partial point cloud and a second partial point cloud of the point cloud data based on the posture of the object determined by the posture calculation unit.

According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives position information indicating a surface position of an object and calculates feature points of the object from normals at points on the position information; and a posture calculation unit that calculates the posture of the object based on a plurality of feature points and skeletal information.
According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives position information indicating a position of a surface of an object, and determines feature points related to the object from normals at points on the position information, and an attitude calculation unit that calculates the attitude of the object, wherein the calculation unit determines a plurality of feature points inside the object, and the attitude calculation unit calculates the attitude of the object using the plurality of feature points set inside the object.
According to an aspect of the present invention, there is provided a detection device including a detection section that detects position information of each point on a surface of an object, and the above-mentioned processing device.

According to an aspect of the present invention, there is provided a processing program that causes a computer to receive positional information of each point on the surface of an object, determine feature points of the object from normals at the points based on the positional information, and determine the orientation of the object from the multiple feature points, where the process of determining the orientation of the object involves fitting skeletal information to connecting lines connecting the multiple feature points to determine the orientation of the object.
According to an aspect of the present invention, a processing program is provided that causes a computer to receive point cloud data indicating positional information of each point on the surface of an object, determine feature points related to the object from normals at points on the point cloud data, determine the orientation of the object from the plurality of feature points, and generate model information including shape information of the object based on the point cloud data, wherein in the process of determining the orientation of the object, the orientation of the object is determined by fitting skeletal information to the plurality of feature points, and in the process of generating the model information including the shape information of the object, a first partial point cloud and a second partial point cloud of the point cloud data are distinguished based on the determined orientation of the object.

According to an aspect of the present invention, there is provided a processing program that causes a computer to receive position information indicating a position of a surface of an object, determine feature points of the object from normals at points on the position information, and determine the posture of the object based on a plurality of feature points and skeletal information.
According to an aspect of the present invention, there is provided a processing program that causes a computer to receive position information indicating a position of a surface of an object, determine feature points of the object from normals at points on the position information, and calculate the orientation of the object, wherein in the process of determining the feature points, a plurality of feature points are determined within the object, and in the process of calculating the orientation of the object, the orientation of the object is calculated using the plurality of feature points set within the object.

FIG. 1 is a diagram showing a detection device according to a first embodiment. FIG. 2 is a diagram showing an information acquisition unit according to the first embodiment. 5A to 5C are diagrams showing processing of a point cloud data generating unit according to the first embodiment. 5A to 5C are diagrams showing processing of a point cloud data generating unit according to the first embodiment. 5A to 5C are diagrams showing processing of a point cloud data generating unit according to the first embodiment. FIG. 4 is a diagram showing processing of a calculation unit according to the first embodiment. FIG. 4 is a diagram showing processing of a calculation unit according to the first embodiment. 5 is a diagram showing the processing of an attitude calculation unit according to the first embodiment. FIG. FIG. 4 is a diagram showing predetermined skeletal information according to the first embodiment. 5 is a diagram showing the processing of an attitude calculation unit according to the first embodiment. FIG. 4 is a flowchart showing a detection method according to the first embodiment. FIG. 13 is a diagram showing a detection device according to a second embodiment. FIG. 11 is a diagram showing the processing of a calculation unit according to the second embodiment. FIG. 11 is a diagram showing the processing of a calculation unit according to the second embodiment. 10 is a flowchart showing a detection method according to a second embodiment. FIG. 13 is a diagram showing a detection device according to a third embodiment. FIG. 13 is a diagram showing a detection device according to a fourth embodiment. FIG. 13 is a diagram showing a detection device according to a fifth embodiment.

The following describes the embodiments with reference to the drawings. In order to explain the embodiments, the drawings are appropriately scaled, with some parts enlarged or emphasized, and may differ in size and shape from the actual product. In the following description, directions may be explained using the XYZ Cartesian coordinate system shown in the drawings.

[First embodiment]
The first embodiment will be described. FIG. 1 is a diagram showing an example of a detection device according to the first embodiment. The detection device 1 (detection system) is, for example, a motion capture device, a motion detection system, an exercise support system, etc. The detection device 1 is also used for posture analysis or three-dimensional modeling, etc. In these cases, the detection device 1 detects an object M moving in one direction or in multiple directions in a predetermined time range. The detection device 1 may detect an object M moving linearly, or may detect an object M moving in a meandering manner. That is, the movement path (e.g., a trajectory) of the object M when the detection device 1 detects the object M may include a straight line, may include a curved line, or may include a straight line and a curved line.

The object M is movable within a target area AR that is the subject of detection by the detection device 1. The target area AR corresponds to, for example, an area or range of an object that can be detected by the detection device 1, or a field of view. The object M is, for example, a human body or a non-human animal, a humanoid or non-human animal-type robot, or a non-animal robot. The object M changes, for example, one or both of its posture and shape as it moves. For example, if the object M is a human body, a first part (for example, the left foot) and a second part (for example, the right foot) move alternately in the movement direction MD when it moves (for example, walking, running, exercising, moving). At this time, at least a part of the shape (for example, posture) of the object M changes.

The moving human body is, for example, a human body moving in sports such as fencing, baseball, soccer, golf, kendo, American football, ice hockey, or gymnastics. The moving human body is, for example, a human body moving while walking or posing in running, exercise, yoga, bodybuilding, fashion shows, etc. The moving human body is, for example, a human body moving in games, person authentication, or work. Note that the target M may include the moving human body and objects attached to the moving human body (for example, clothing, wearables, exercise equipment, protective gear).

In the following, the XYZ Cartesian coordinate system shown in FIG. 1 and the like will be referred to. In this XYZ Cartesian coordinate system, the X direction is the moving direction MD of the object M, the Y direction is the vertical direction of the moving direction MD of the object M, and the Z direction is the intersecting direction (orthogonal direction) that intersects (is perpendicular to) the moving direction MD of the object M and the vertical direction. When the object M is a human body, the Z direction includes the direction of the shoulder width of the human body. In addition, in the following, in each of the X, Y, and Z directions, the side indicated by the arrow is appropriately referred to as the + side (e.g., the +Z side), and the opposite side to the side indicated by the arrow is appropriately referred to as the - side (e.g., the -Z side). For example, the first part (e.g., the left foot) is on the -Z side of the second part (e.g., the right foot), and the second part is on the +Z side of the first part.

The detection device 1 includes a detection unit 10 and a processing device 100. The detection unit 10 detects position information of each point on the surface of the object M. The detection unit 10 detects point cloud data including three-dimensional coordinates of each point on the surface of the object M as the position information of the object M, as described below. In this embodiment, the movement direction MD of the object M is predetermined with respect to the object area AR. The vertical direction in the object area AR and the movement direction MD of the object M are given to the detection device 1 as known information. In the detection result (point cloud data) by the detection unit 10, the X, Y, and Z directions are set based on the above known information. The detection unit 10 includes an information acquisition unit 11 and a point cloud data generation unit 110.

The information acquisition unit 11 is, for example, at least a part of a portable device (mobile equipment). The information acquisition unit 11 may be at least a part of a stationary device. The information acquisition unit 11 may be provided inside the processing device 100. Furthermore, the point cloud data generation unit 110 may be provided in a device external to the processing device 100 and connected to the processing device 100. For example, the detection unit 10 may be a device external to the processing device 100 and may incorporate the information acquisition unit 11 and the point cloud data generation unit 110. A part or all of the detection device 1 may be a portable device (for example, an information terminal, a smartphone, a tablet, a mobile phone with a camera, a wearable terminal). A part or all of the detection device 1 may be a stationary device (for example, a fixed camera).

Figure 2 is a diagram showing an example of an information acquisition unit according to the first embodiment. The information acquisition unit 11 detects the depth of the target object M. The information acquisition unit 11 includes, for example, a depth sensor (depth camera). The information acquisition unit 11 detects the depth (distance, depth, depth) from a predetermined point to each point on the surface of an object placed in the target area AR. The predetermined point is, for example, a point at a position that serves as a reference for detection by the information acquisition unit 11 (for example, a viewpoint, a point of detection origin, a point representing the position of the information acquisition unit 11, or a pixel position of the image sensor 14 described later).

The information acquisition unit 11 includes an irradiation unit 12, an optical system 13, and an image sensor 14. The irradiation unit 12 irradiates (projects) light La (e.g., pattern light, irradiation light) onto the target area AR. The optical system 13 includes, for example, an imaging optical system (image sensor). The image sensor 14 includes, for example, a CMOS image sensor or a CCD image sensor. The image sensor 14 has a plurality of pixels arranged two-dimensionally. The image sensor 14 images the target area AR via the optical system 13. The image sensor 14 detects light Lb (infrared light, return light) emitted from an object placed in the target area AR by irradiation with light La.

The information acquisition unit 11 detects the depth from a point on the target area AR corresponding to each pixel of the image sensor 14 to each pixel of the image sensor 14 based on, for example, the pattern (e.g., intensity distribution) of light La irradiated from the irradiation unit 12 and the pattern (e.g., intensity distribution, captured image) of light Lb detected by the image sensor 14. The information acquisition unit 11 outputs a depth map (depth image, depth information, distance image) representing the distribution of depth in the target area AR as the detection result to the processing device 100. That is, the information acquisition unit 11 outputs the depth map of the object M to the processing device 100.

The information acquisition unit 11 may be a device that detects depth using a time-of-flight (TOF) method. The information acquisition unit 11 may also be a device that detects depth using a method other than the TOF method. For example, the information acquisition unit 11 may include a laser scanner (laser range finder) and detect depth using laser scanning. For example, the information acquisition unit 11 may include a phase difference sensor and detect depth using a phase difference method. For example, the information acquisition unit 11 may detect depth using a depth-from-defocus (DFD) method.

The information acquisition unit 11 may irradiate the object M with light other than infrared light (e.g., visible light) and detect the light (e.g., visible light) emitted from the object M. For example, the information acquisition unit 11 may include a stereo camera or the like, and detect (image) the object M from multiple viewpoints. For example, the information acquisition unit 11 may detect the depth by triangulation using captured images of the object M captured from multiple viewpoints. For example, the information acquisition unit 11 may detect the depth by a method other than an optical method (e.g., ultrasonic scanning).

Returning to the explanation of FIG. 1, the point cloud data generation unit 110 generates point cloud data equivalent to information representing the surface position of the object M and the shape of the object M as position information of the object M based on the depth detected by the information acquisition unit 11. In FIG. 1, the information acquisition unit 11 is provided outside the processing device 100. Also in FIG. 1, the point cloud data generation unit 110 is provided inside the processing device 100. The processing device 100 is communicatively connected to the information acquisition unit 11. The information acquisition unit 11 outputs the detection result to the processing device 100. The processing device 100 executes processing based on the detection result output from the information acquisition unit 11.

The processing device 100 includes a point cloud data generation unit 110, a calculation unit 120, an attitude calculation unit 130, and a storage unit 140. The storage unit 140 is, for example, a non-volatile memory, a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 140 stores original data to be processed by the processing device 100 and data processed (generated) by the processing device 100. The storage unit 140 stores a depth map output from the information acquisition unit 11 as original data of the point cloud data.

The point cloud data generating unit 110 executes point cloud processing to generate point cloud data as position information of the object M. The point cloud data includes three-dimensional coordinates of multiple points on the object M in the target area AR. In addition, the point cloud data may include three-dimensional coordinates of multiple points on objects (e.g., walls, floors) surrounding the object M. The point cloud data generating unit 110 reads out the depth map stored in the storage unit 140 to calculate the point cloud data of the object M. The point cloud data generating unit 110 may generate point cloud data as model information (shape information) of the object M.

Figures 3 to 5 are diagrams showing an example of processing by the point cloud data generation unit according to the first embodiment. D1 shown in Figure 3 is a depth map corresponding to the detection result of the information acquisition unit 11. The depth map D1 is information (image) that represents the spatial distribution of the depth measurement values by the information acquisition unit 11. The depth map D1 is a grayscale image that represents the depth at each point in the target area AR as a gradation value. Parts in the depth map D1 with relatively high gradation values (e.g., white parts, bright parts) are parts with relatively small depths (e.g., parts relatively close to the detection unit 10). Parts in the depth map D1 with relatively low gradation values (e.g., black parts, dark parts) are parts with relatively large depths (e.g., parts relatively far from the detection unit 10).

The point cloud data generator 110 reads out the data of the depth map D1 from the storage unit 140, calculates the three-dimensional coordinates of points in real space corresponding to each pixel based on the gradation value (depth measurement value) of each pixel of the depth map D1, and generates point cloud data D2 (see FIG. 4). Hereinafter, the three-dimensional coordinates of one point may be referred to as point data. The point cloud data D2 is a set of multiple point data.

The point cloud data generation unit 110 extracts point cloud data D2 of the target object M, for example by performing segmentation, pattern recognition, etc. on the point cloud data of the entire target area AR. In other words, the point cloud data generation unit 110 executes an extraction process (segmentation) to extract position information of a part of an object (target object M) from the position information of the objects placed in the target area AR. Here, the objects placed in the target area AR include the target object M and objects around the target object M (for example, floors, walls, and background objects).

Prior to the above extraction process, the detection unit 10 detects position information of an object in the target area AR in each of a first state in which the target object M is not located in the target area AR and a second state in which the target object M is located in the target area AR. In the extraction process, the point cloud data generation unit 110 extracts the position information of the target object M by calculating the difference between the detection result of the detection unit 10 in the first state and the detection result of the detection unit 10 in the second state.

The first state may be a state in which the object M is placed in the target area AR in a state different from the second state. For example, the first state may be a state in which the object M is placed in a first position in the target area AR, and the second state may be a state in which the object M is placed in a second position different from the first position in the target area AR. The extraction process may be performed by a processing unit other than the point cloud data generation unit 110. Such a processing unit may be provided in the processing device 100, or may be provided in a device external to the processing device 100. The extraction process may not be performed.

The processing device 100 may also remove noise from the detection result of the detection unit 10. The noise removal process for removing noise includes, for example, a process for removing spatial noise from the depth map D1 generated by the detection unit 10 (information acquisition unit 11). For example, when the difference in depth between a first region (first pixel) and a second region (second pixel) adjacent to each other in the depth map D1 exceeds a threshold, the processing device 100 determines that the depth in the first region or the depth in the second region is noise. When the processing device 100 determines that the depth in the first region is noise, it estimates the depth of the first region by interpolation or the like using the depths of the regions (third pixel, fourth pixel) surrounding the first region. Then, the processing device 100 removes spatial noise by updating (replacing, correcting) the depth of the first region with the estimated depth. The point cloud data generation unit 110 may generate point cloud data based on the depth map D1 from which spatial noise has been removed.

The noise removal process also includes, for example, a process of removing temporal noise (noise that changes over time) from the depth map D1 generated by the detection unit 10 (information acquisition unit 11). For example, the detection unit 10 repeats detection at a predetermined sampling frequency. Then, the processing device 100 compares the detection results (depth maps D1 generated by the information acquisition unit 11) of the detection unit 10 that have different detection timings. For example, the processing device 100 calculates the amount of change in depth (e.g., the amount of change over time) for each pixel between the depth map D1 in the first frame and the depth map D1 in the second frame that follows the first frame.

When the amount of change in depth for each pixel satisfies a predetermined condition, the processing device 100 determines that the depth of this pixel in the first frame or the depth in the second frame is noise. When the processing device 100 determines that the depth in the second frame is noise, it estimates the depth of the second frame by, for example, interpolation using the depth in the first frame and the depth in the third frame that follows the second frame. The processing device 100 removes temporal noise by updating (replacing, correcting) the depth of the pixel corresponding to the noise with the estimated depth. The point cloud data generation unit 110 may generate point cloud data based on the depth map D1 from which temporal noise has been removed.

The noise removal process does not have to include a process for removing spatial noise or a process for removing temporal noise. The noise removal process may be performed by a part other than the processing device 100 (e.g., the detection unit 10). The detection device 1 does not have to perform the noise removal process.

The point cloud data generating unit 110 determines the normal line H at the point based on the point cloud data D2 (see FIG. 5). Here, a "point" is any point among the points on the surface S1 of the object M. Note that the normal line H at the point does not have to be determined by the point cloud data generating unit 110, and may be determined by a processing unit other than the point cloud data generating unit 110. The processing unit may be provided in the processing device 100, or in a device external to the processing device 100.

The point cloud data generating unit 110 selects, for example, each of the other points (D2a 2 , D2a ₃ , D2a ₄ , D2a ₅ ) existing within a range DR1 of a predetermined distance from an arbitrary point D2a ₁ among the points on the surface S1 of the object M in the point _cloud data D2. The range DR1 indicates, for example, a range included in a sphere (or an ellipsoid) centered on the point D2a ₁ , and may be set in advance by a user, or may be a variable range set according to the number and density of the point cloud data D2. The point cloud data generating unit 110 obtains and normalizes a vector V from the point D2a ₁ to each of the points (D2a ₂ , D2a ₃ , D2a ₄ , D2a ₅ ). With these, the point cloud data generating unit 110 obtains a normal line H at the point D2a _1. The point cloud data generating unit 110 obtains each of the normal lines H from each of the arbitrary points D2a ₁ . When the object M is a human body, the normal H becomes a vector pointing inwardly toward the inside of the human body because the surface S1 of the human body is rounded.

Returning to the explanation of FIG. 1, the calculation unit 120 determines feature points related to the object M from the normal H at a point (a point on the surface S1 of the object M) based on the position information detected by the detection unit 10. In this embodiment, the feature points are used to determine the posture of the object M.

FIG. 6 is a diagram showing an example of processing of the calculation unit according to the first embodiment. The calculation unit 120, for example, selects nearby points from each point on the surface of the object M, and obtains a surface constituted by the selected points. The surface is obtained over the entire object M. In FIG. 6, one of the surfaces constituted by nearby points is S2. The surface constituted by nearby points may be obtained by the point cloud data generation unit 110, or may be obtained by a processing unit other than the calculation unit 120. The processing unit may be provided in the processing device 100, or may be provided in an external device to the processing device 100. When the normal line H obtained by the point cloud data generation unit 110 is extended toward the inside of the object M, and the normal line H' collides with the surface S2 (the surface facing the surface S1 of the object M), the calculation unit 120 obtains the distance between the point _D2a1 and the surface S2, and obtains the midpoint of the obtained distance as the feature point Q. The feature point Q is obtained based on the normal line H toward the inside of the object M, and is therefore a point that exists inside the object M.

Figure 7 is a diagram showing an example of processing by the calculation unit according to the first embodiment. The multiple circles (ellipses) shown in Figure 7 are feature points Q. Figure 7 shows an enlarged view of the inside of the object M (more specifically, the location where the determined feature point Q exists).

As shown in FIG. 7, feature points Q may be concentrated in a local area (range DR2). When there are multiple feature points Q included in range DR2, the calculation unit 120 groups the multiple feature points Q into one feature point Qt. The range DR2 may be changed depending on the number of point cloud data D2 or the degree of accuracy of posture detection described below, or may be set by the user.

The calculation unit 120 may obtain the feature point Qt from, for example, the average value of the coordinates of the multiple feature points Q. Alternatively, the calculation unit 120 may obtain the feature point Qt from, for example, the median value of the coordinates of the multiple feature points Q. Note that the method for obtaining the feature point Qt is not limited to the above, and various methods may be used to combine points (feature points Q) scattered within a certain range (for example, range DR2) into one point (feature point Qt).

When the calculation unit 120 obtains the feature point Qt, the processing device 100 may not use the feature point Q included in the range DR2 at this time in the subsequent processing. Note that, when the number of feature points Q included in the range DR2 is less than a predetermined number, the calculation unit 120 may not obtain the feature point Qt in the range DR2 at this time. Also, the calculation unit 120 may adopt any one point (or several points) of the feature points Q included in the range DR2 and not use the other feature points Q. Also, the calculation unit 120 may not obtain the feature point Qt (as long as the feature point Q is used). Also, the information on the feature points Qt and Q obtained by the calculation unit 120 may be output to any device (an external device different from the processing device 100). The processing device 100 obtains the calculation of the feature point Qt (feature point Q) described above while changing the position inside the object M. For example, the processing device 100 calculates the feature point Qt (feature point Q) from each position while shifting the position by a predetermined distance inside the object M. In other words, the processing device 100 acquires multiple feature points Qt (feature points Q).

Returning to the explanation of FIG. 1, the posture calculation unit 130 calculates the posture of the object M from the multiple feature points Qt (feature points Q) calculated by the calculation unit 120. As described above, the processing result of the calculation unit 120 may be only the feature points Q, may be only the feature points Qt, or may be a mixture of the feature points Qt and Q. The posture calculation unit 130 calculates the posture of the object M using these feature points Qt and Q. For example, the posture calculation unit 130 fits predetermined model information to the multiple feature points Qt and Q to calculate the posture of the object M. Note that the predetermined model information may be stored in advance in the storage unit 140, or may be obtained from outside the processing device 100. Furthermore, if the object M is not a human body, the predetermined model information may be prepared accordingly.

Here, an example of a method for determining the posture of the object M will be described. FIG. 8 is a diagram showing an example of processing by the posture calculation unit according to the first embodiment. In FIG. 8, the feature points Qt and Q determined by the calculation unit 120 are shown as circles. Note that in FIG. 8, the object M is shown by a two-dot chain line for the purpose of explanation, but the object M (two-dot chain line) is not actually represented (as shown). The posture calculation unit 130 connects the multiple feature points Qt and Q determined by the calculation unit 120. FIG. 8 shows a connecting line CL connecting the feature points Qt and Q. The posture calculation unit 130 may then fit predetermined skeletal information (e.g., skeleton information) as model information to the connecting line CL connecting the multiple feature points Qt and Q to determine the posture of the object M.

Figure 9 is a diagram showing an example of predetermined skeletal information according to the first embodiment. That is, the posture calculation unit 130 uses skeleton information Si, which is an example of model information, to determine the posture of the object M. The skeleton information Si holds information on parts of the object M, such as the head, as well as the joints of the object M. As described above, if the object M is not a human body, the skeleton information Si may be prepared accordingly.

10 is a diagram showing an example of processing by the posture calculation unit according to the first embodiment. Note that, for the sake of explanation, FIG. 10 shows a circle filled in on the connecting line CL, but this circle is not actually expressed (as shown in the figure). The posture calculation unit 130 fits the skeleton information Si to the connecting line CL as shown by the arrow in FIG. 10. The shape of the skeleton information Si obtained by fitting can be the posture of the target M. In addition, the skeleton information Si obtained by fitting and the processing result obtained by performing a predetermined process (for example, a process of adding an external shape) on the skeleton information Si may be output (displayed) to an arbitrary device (an external device different from the processing device 100). The processing result obtained by performing a predetermined process on the skeleton information Si is, for example, data including lines, points, etc. that represent the posture of the target M. Note that the detection device 1 does not need to be equipped with the posture calculation unit 130.

Next, a detection method according to an embodiment will be described based on the configuration of the detection device 1 described above. FIG. 11 is a flowchart showing an example of the detection method according to the first embodiment. For the configuration of the detection device 1 and the processing by each unit, refer to FIG. 1 to FIG. 10 as appropriate.

In step S101, the detection unit 10 detects position information of each point on the surface of the object M. The detection unit 10 detects the depth of the object M (see FIG. 3) and generates point cloud data D2 as a detection result (see FIG. 4). For example, the information acquisition unit 11 detects the depth from a predetermined point (viewpoint) to each point in the object area AR. The information acquisition unit 11 stores a depth map (spatial distribution of depth) representing the detection result in the storage unit 140. The point cloud data generation unit 110 generates the point cloud data D2 based on the depth detection result. The point cloud data generation unit 110 stores the generated point cloud data D2 in the storage unit 140. The point cloud data generation unit 110 obtains a normal H at a point D2a1 on the surface S1 of the object M based on the point cloud data _D2 (see FIG. 5).

In step S102, the calculation unit 120 obtains a feature point Q of the object M from the normal line H at the point _D2a1 (see FIG. 6). For example, the calculation unit 120 obtains the feature point Q that is the midpoint of the distance from the point _D2a1 to another surface S2 through which a normal line H' obtained by extending the normal line H passes. The calculation unit 120 may also obtain the feature point Qt from the average value or median value of the coordinates of the feature points Q included in the range DR2 (see FIG. 7).

In step S103, the posture calculation unit 130 determines the posture of the object M from the multiple feature points Qt, Q (see Figures 8, 9, and 10). For example, the posture calculation unit 130 connects the multiple feature points Qt, Q to generate a connecting line CL (see Figure 8). The posture calculation unit 130 fits predetermined skeleton information Si (see Figure 9) to the connecting line CL (see Figure 10). From these, the posture calculation unit 130 determines the posture of the object M.

As described above, the detection device 1 detects position information of each point on the surface of the object M, determines feature points Q of the object M from normals at the points based on the position information, and determines the posture of the object M from the multiple feature points Q. As a result, the detection device 1 determines the posture of the object M based on feature points Q assumed to be inside the object M, and can determine the posture of the object M with high accuracy. In other words, when setting feature points on the surface of the object (for example, the hands and feet, which can be more distinctive than other parts), the detection device 1 can determine the posture of the object M with high accuracy compared to techniques that have a large range of motion and therefore produce large noise.

[Second embodiment]
Next, a second embodiment will be described. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof may be omitted or simplified. In the above-mentioned embodiment, the feature points Qt, Q are calculated based on the distance from a point to another surface through which the normal line at the point passes. In the second embodiment, a different method for calculating the feature points Qt, Q will be described.

Figure 12 is a diagram showing an example of a detection device according to the second embodiment. The processing device 200 included in the detection device 1a has a calculation unit 220 that calculates a feature point Q (Qt) from multiple normals (normals at a point) based on the detection results of the detection unit 10.

Figure 13 is a diagram showing an example of processing by the calculation unit according to the second embodiment. The rectangle shown in Figure 13 represents a portion of the point cloud data D2. In Figure 13, arbitrary points on the surface of the object M are designated as points D2a, D2b, and D2c. The points are points on surfaces S1a, S1b, and S1c, respectively. As in the first embodiment, the calculation unit 120 determines normals at points D2a, D2b, and D2c, respectively. The normals are designated as Ha, Hb, and Hc, and the normals obtained by extending the normals are designated as normal Ha', normal Hb', and normal Hc'.

The calculation unit 220 finds the feature point Q from the intersection of multiple normal lines Ha'-Hc'. Since the feature point Q is found from the intersection of multiple normal lines Ha'-Hc', it is a point that is (assumed to be) inside the object M. Note that the calculation of the normal at the point does not have to be performed over the entire surface of the object M.

In other words, in this embodiment, the feature point Q is found based on the intersection of multiple normal lines Ha'-Hc', and other surfaces through which the normal lines pass are not used in the processing, so it is not necessary to use points over the entire surface of the object M. In other words, in this embodiment, it is sufficient to determine at least the intersection of multiple normal lines, so normal lines from all directions on the surface of the object M are not required, and it is not necessary to use points over the entire surface of the object M. The area in which points are used may be changed depending on the type (or size, shape, etc.) of the object M, or may be set by the user.

Fig. 14 is a diagram showing an example of processing by the calculation unit according to the second embodiment. In Fig. 14, multiple normals (multiple extended normals) are represented by H'. Fig. 14 also shows an enlarged view of the vicinity of the intersection of the multiple normals H' (the interior side of the object M, where the determined feature point Q exists).

As shown in FIG. 14, feature points Q may be concentrated in a local area (range DR3). When there are multiple feature points Q included in range DR3, the calculation unit 220 groups the multiple feature points Q into one feature point Qt. Range DR3 may be changed depending on the number of point cloud data D2 or the degree of accuracy of posture detection, or may be set by the user.

The calculation unit 220 may obtain the feature point Qt from, for example, the average value of the coordinates of the multiple feature points Q. Alternatively, the calculation unit 220 may obtain the feature point Qt from, for example, the median value of the coordinates of the multiple feature points Q. Note that, as in the first embodiment, the method for obtaining the feature point Qt is not limited to these. When the calculation unit 220 obtains the feature point Qt, the processing device 200 may not use the feature point Q included in the range DR3 at this time in subsequent processing.

When the number of feature points Q included in range DR3 is less than a predetermined number, calculation unit 220 does not need to find feature points Qt in range DR3 at this time. Also, calculation unit 220 does not need to find feature points Qt (feature points Q may be used). Also, information on feature points Qt and Q found by calculation unit 220 may be output to any device (external device different from processing device 200).

Next, a detection method according to an embodiment will be described based on the configuration of the detection device 1a described above. FIG. 15 is a flowchart showing an example of a detection method according to the second embodiment. For the configuration of the detection device 1a and the processing by each unit, refer to FIG. 1 to FIG. 14 as appropriate.

In step S201, the detection unit 10 detects position information of each point on the surface of the object M. The detection unit 10 detects the depth of the object M (see FIG. 3) and generates point cloud data D2 as the detection result (see FIG. 4). For example, the information acquisition unit 11 detects the depth from a predetermined point (viewpoint) to each point in the object area AR. The information acquisition unit 11 stores a depth map (spatial distribution of depth) representing the detection result in the storage unit 140. The point cloud data generation unit 110 generates point cloud data D2 based on the depth detection result. The point cloud data generation unit 110 stores the generated point cloud data D2 in the storage unit 140. The point cloud data generation unit 110 calculates normals Ha, Hb, Hc at points D2a, D2b, D2c on the surface S1a, S1b, S1c of the object M based on the point cloud data D2.

In step S202, the calculation unit 220 finds the feature point Q based on the intersection of multiple normal lines (see FIG. 13). For example, the calculation unit 220 finds the feature point Q based on the intersection of a normal line Ha' obtained by extending the normal line Ha at point D2a, a normal line Hb' obtained by extending the normal line Hb at point D2b, and a normal line Hc' obtained by extending the normal line Hc at point D2c (see FIG. 13). The calculation unit 220 may also find the feature point Qt from the average or median of the coordinates of the feature points Q included in the range DR3 (see FIG. 14).

In step S203, the posture calculation unit 130 determines the posture of the object M from the multiple feature points Qt, Q (see Figures 8, 9, and 10). For example, the posture calculation unit 130 connects the multiple feature points Qt, Q to generate a connecting line CL (see Figure 8). The posture calculation unit 130 fits the skeleton information Si (see Figure 9) to the connecting line CL (see Figure 10). From these, the posture calculation unit 130 determines the posture of the object M.

As described above, the detection device 1a detects position information of each point on the surface of the object M, finds feature points Q from the intersections of multiple normals at the points based on the position information, and finds the posture of the object M from the multiple feature points Q. As a result, the detection device 1a finds feature points Q assumed to be inside the object M using points in any area, not the entire area, on the surface of the object M, and can therefore find the posture of the object M with high accuracy while reducing the processing load and increasing the processing speed.

[Third embodiment]
Next, a third embodiment will be described. In this embodiment, the same reference numerals are used for the same configurations as those in the above-mentioned embodiment, and the description thereof may be omitted or simplified. In the above-mentioned embodiment, the moving direction MD of the object M is predetermined with respect to the target area AR. In the third embodiment, the case where the moving direction MD of the object M is detected will be described.

FIG. 16 is a diagram showing an example of a detection device according to the third embodiment. The processing device 300 included in the detection device 1b includes a direction detection unit 350 that calculates the moving direction of the object M based on the detection result of the detection unit 10. In this embodiment, the information acquisition unit 11 repeatedly detects the object M. In addition, the point cloud data generation unit 110 generates point cloud data of the object M for each detection timing as position information. The direction detection unit 350 calculates the moving direction MD of the object M based on the time change of the position information of the object M detected by the detection unit 10. For example, the direction detection unit 350 calculates the trajectory of the object M (for example, the time history of a predetermined position of the object M) and sets the direction along the trajectory as the moving direction MD of the object M. Such a predetermined position is a position on the object M determined by default settings or user settings.

The direction detection unit 350 calculates the center of gravity of multiple points included in the point cloud data as a predetermined position of the object M. The direction detection unit 350 calculates the movement direction MD based on the change over time of the calculated center of gravity. For example, the direction detection unit 350 sets the start point to a first predetermined position of the object M obtained from the detection result of the information acquisition unit 11 at a first time. Then, the direction detection unit 350 calculates a vector whose end point is a second predetermined position of the object M obtained from the detection result of the information acquisition unit 11 at a second time that is later than the first time. Next, the direction detection unit 350 determines a direction vector (e.g., a unit vector) parallel to the calculated vector as the movement direction MD of the object M at the first time.

The direction detection unit 350 may calculate the moving direction MD of the object M based on the position information of the object M acquired from a global positioning system (GPS). In addition, if an acceleration sensor is provided on the object M, the direction detection unit 350 may calculate the moving direction MD of the object M based on the detection result of the acceleration sensor. For example, the object M may move accompanied by a mobile terminal (e.g., a smartphone) equipped with either or both of a receiving unit that receives information from a GPS and an acceleration sensor. In such a case, the direction detection unit 350 may acquire the position information of the object M (e.g., GPS information, acceleration information) from the mobile terminal and calculate the moving direction MD of the object M.

[Fourth embodiment]
Next, a fourth embodiment will be described. In this embodiment, the same components as those in the above-described embodiments are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

Figure 17 is a diagram showing an example of a detection device according to the fourth embodiment. The processing device 400 included in the detection device 1c has a model generation unit 460 that generates model information of the object M. The model information is, for example, three-dimensional CG (Computer Graphics) model data, and includes shape information of the object M. As described in the first embodiment, the point cloud data generation unit 110 generates point cloud data as shape information based on the depth of the object M detected by the information acquisition unit 11. The model generation unit 460 executes surface processing to calculate surface information as shape information.

Surface information includes, for example, at least one of polygon data, vector data, and draw data. Surface information includes the coordinates of multiple points on the surface of an object, and connectivity information (attribute information) between the multiple points. The connectivity information includes, for example, information that associates with each other the points at both ends of a line that corresponds to a ridge (e.g., an edge) on the object surface. The connectivity information also includes, for example, information that associates with each other multiple lines that correspond to the contour of the object surface.

In surface processing, the model generation unit 460 estimates a surface between a point selected from a plurality of points included in the point cloud data and a point in the vicinity of the selected point. The model generation unit 460 segments the point cloud data when estimating a surface between a point and a point in the vicinity of the point. For example, the model generation unit 460 may segment the point cloud data using the processing result of the posture calculation unit 130 during estimation. For example, when generating surface information of the left and right feet of the object M, the model generation unit 460 distinguishes between the point cloud data of the left foot and the point cloud data of the right foot based on the processing result of the posture calculation unit 130. In this way, the model generation unit 460 can avoid estimating a surface that spans the left knee and the right knee, for example, when the left knee and the right knee are close to each other (in contact), and generate highly accurate surface information.

In addition, in surface processing, the model generation unit 460 converts the point cloud data into polygon data having plane information between points. The model generation unit 460 converts the point cloud data into polygon data using an algorithm that uses, for example, the least squares method. This algorithm may be, for example, an algorithm that is published in a point cloud processing library. The model generation unit 460 stores the calculated surface information in the storage unit 140.

The model information may include texture information of the object M. The model generation unit 460 may generate texture information of a surface defined by three-dimensional point coordinates and related information. The texture information includes at least one of information defining characters, figures, designs, textures, patterns, and unevenness on the object surface, a specific image, and color (e.g., chromatic color, achromatic color). The model generation unit 460 may store the generated texture information in the storage unit 140.

The model information may also include spatial information of the image (e.g., lighting conditions, light source information). The light source information includes information on at least one of the following: the position of the light source that irradiates the object M with light, the direction in which light is irradiated from this light source to the object M, the wavelength of the light irradiated from this light source, and the type of this light source. The model generation unit 460 may calculate the light source information using, for example, a model that assumes Lambertian reflection, a model that includes albedo estimation, or the like. The model generation unit 460 does not need to generate either or both of the texture information and the spatial information.

[Fifth embodiment]
Next, a fifth embodiment will be described. In this embodiment, the same components as those in the above-described embodiments are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

FIG. 18 is a diagram showing an example of a detection device according to the fifth embodiment. A processing device 500 included in the detection device 1d includes a model generation unit 460 and a rendering processing unit 570. The processing device 500 is also connected to an input device 21 and a display device 22. The rendering processing unit 570 includes, for example, a GPU (Graphics Processing Unit). Note that the rendering processing unit 570 may be configured such that a CPU (Central Processing Unit) and a memory execute each process according to an image processing program. The rendering processing unit 570 executes, for example, at least one process among a drawing process, a texture mapping process, and a shading process.

In the drawing process, the rendering processing unit 570 can calculate an estimated image (reconstructed image) of a shape defined in the shape information of the model information, for example, viewed from an arbitrary viewpoint. Hereinafter, the shape indicated by the shape information may be referred to as the model shape. The rendering processing unit 570 can reconstruct a model shape (estimated image) from the model information (shape information), for example, by the drawing process. The rendering processing unit 570 stores data of the calculated estimated image in the storage unit 140, for example.

In addition, in the texture mapping process, the rendering processing unit 570 can calculate an estimated image in which, for example, an image indicated by the texture information of the model information is pasted onto the surface of an object in the estimated image. The rendering processing unit 570 can calculate an estimated image in which a texture different from that of the target is pasted onto the surface of an object in the estimated image.

In the shading process, the rendering processing unit 570 can calculate an estimated image in which a shadow formed by a light source indicated by the light source information in the model information is added to an object in the estimated image. In the shading process, the rendering processing unit 570 can calculate an estimated image in which a shadow formed by an arbitrary light source is added to an object in the estimated image.

The input device 21 is used to input various information (e.g., data, commands) to the processing device 500. A user can input various information to the processing device 500 by operating the input device 21. The input device 21 includes at least one of a keyboard, a mouse, a trackball, a touch panel, and an audio input device (e.g., a microphone), for example.

The display device 22 displays the image based on the image data output from the processing device 500. For example, the processing device 500 outputs the estimated image data generated by the rendering processing unit 570 to the display device 22. The display device 22 displays the estimated image based on the estimated image data output from the processing device 500. The display device 22 includes, for example, a liquid crystal display. The input device 21 and the display device 22 may be configured with a touch panel or the like.

The input device 21 and the display device 22 may be connected to a detection device according to the embodiment described above. For example, the display device 22 may display information on the feature points Qt and Q calculated by the feature point calculation unit 122 or the like. For example, the display device 22 may display skeleton information Si calculated by the posture calculation unit 130 or the like, or a processing result obtained by performing a predetermined process on the skeleton information Si (for example, data including lines, points, etc. that represent the posture of the target object M).

The detection device 1d may not include the input device 21. For example, the detection device 1d may be configured such that various commands and information are input via communication. The detection device 1d may not include the display device 22. For example, the detection device 1d may output data of an estimated image generated by the rendering process to a display device via communication, and cause the display device to display the estimated image. The rendering processing unit 570 may be provided in a device external to the processing device 500. Such an external device may be provided by cloud computing that is communicatively connected to the processing device 500.

In the above-described embodiment, the processing device 100 etc. includes, for example, a computer system. The processing device 100 reads a processing program stored in the storage unit 140 and executes various processes in accordance with the read processing program. Such a processing program causes a computer to execute, for example, determining feature points of an object from normals at points based on position information of each point on the surface of the object. This processing program may be provided by being recorded on a computer-readable storage medium (for example, a non-transitory tangible media).

The technical scope is not limited to the aspects described in the above-mentioned embodiments. One or more of the requirements described in the above-mentioned embodiments may be omitted. The requirements described in the above-mentioned embodiments may be combined as appropriate. In addition, to the extent permitted by law, the disclosures of all documents cited in the above-mentioned embodiments are incorporated by reference and made part of the description in this text.

1: Detection device, 10: Detection unit, 11: Information acquisition unit, 100: Processing unit, 110: Point cloud data generation unit, 120: Calculation unit, 130: Attitude calculation unit, 140: Storage unit

Claims (15)

a calculation unit that receives point cloud data indicating position information of each point on a surface of an object, and calculates feature points related to the object from normals at points on the point cloud data ;
a posture calculation unit that calculates a posture of the object from a plurality of the feature points;
a model generation unit that generates model information including shape information of the object based on the point cloud data ,
the posture calculation unit obtains a posture of the object by fitting skeleton information to the plurality of feature points ;
The model generation unit distinguishes between a first partial point cloud and a second partial point cloud of the point cloud data based on the posture of the object calculated by the posture calculation unit . The processing device according to claim 1 , wherein the model generation unit generates shape information of the first portion of the object based on the first partial point cloud. The calculation unit determines the feature points within the object.
The processing device according to claim 1 or 2 . the calculation unit determines the feature point from a normal at the point that is determined based on the position information between the point and each of other points that exist within a predetermined distance from the point;
The processing device according to any one of claims 1 to 3 . The calculation unit determines the feature point based on a distance from the point to a plane through which a normal line at the point passes.
The processing device according to any one of claims 1 to 4 . The calculation unit determines the feature point from a midpoint of a distance to a surface through which a normal line at the point passes.
The processing device according to claim 5 . The calculation unit calculates the feature point from an average value of coordinates of the plurality of midpoints.
The processing device of claim 6 . The calculation unit determines the feature point from a median value of coordinates of the plurality of midpoints.
The processing device of claim 6 . The calculation unit determines the feature points from normals at the plurality of points.
The processing device according to any one of claims 1 to 4 . The calculation unit determines the feature point based on an intersection of normals at the plurality of points.
The processing device of claim 9 . the calculation unit determines the feature point from an average value of coordinates of intersections of normal lines at a plurality of the points;
The processing device of claim 10 . the calculation unit determines the feature point from a median value of coordinates of intersections of normal lines at a plurality of the points;
The processing device of claim 10 . The processing device according to claim 1 , wherein the orientation calculation unit calculates the orientation of the target object using a connecting line that connects a plurality of the feature points. A detection unit that detects position information of each point on a surface of an object;
A processing device according to any one of claims 1 to 13 ;
A detection device comprising: On the computer,
receiving point cloud data indicating positional information of each point on a surface of an object, and determining feature points related to the object from normals at points on the point cloud data ;
determining a posture of the object from a plurality of the feature points;
generating model information including shape information of the object based on the point cloud data;
In the process of determining the posture of the object, the posture of the object is determined by fitting skeleton information to the plurality of feature points ;
A processing program in which, in a process of generating model information including shape information of the object, a first partial point cloud and a second partial point cloud of the point cloud data are distinguished based on the determined posture of the object .

Images