JP2021125194A - Shape detection method, shape detection system, and program - Google Patents

Abstract

To provide a shape detection method, a shape detection system, and a program that can easily and quickly detect a predetermined shape on a target object.SOLUTION: The shape detection method for detecting a shape of an object includes the steps of: acquiring three-dimensional point group data from the object by a sensor (SQ101); displaying the three-dimensional point group data in a display region of a terminal; acquiring the display region showing the three-dimensional point group data as two-dimensional image data (SQ102); and conducting an image analysis to determine a predetermined shape in the two-dimensional image data and detecting the predetermined shape of the object (SQ103).SELECTED DRAWING: Figure 7

Description

The present invention relates to a shape detection method, a shape detection system, and a program.

Conventionally, as a method of detecting the shape of an object, a contact type or a non-contact type shape detection method has existed (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-135276

However, with the conventional shape detection method, although it is easy to detect the entire shape, it is difficult to find a predetermined shape on the object. For example, the predetermined shape is visually observed using a three-dimensional model. Although it is possible to confirm with or directly analyze the 3D model itself, the former has a large load in terms of labor and time of the operator, and there is a possibility that confirmation of the predetermined shape may be omitted. In the latter case, the load of the analysis processing of the terminal used for the analysis is heavy, so that they cannot be said to be a simple and quick method.

The present invention has been made in view of such a background, and an object of the present invention is to provide a technique capable of easily and quickly detecting a predetermined shape on an object.

The main invention of the present invention for solving the above problems is a shape detection method for detecting the shape of an object, which includes a step of acquiring three-dimensional point group data from the object by a sensor and the three-dimensional point group. A step of displaying the data in the display area of the terminal, a step of acquiring the display area in which the three-dimensional point group data is displayed as two-dimensional image data, and a predetermined shape in the two-dimensional image data are determined. The shape detection method includes a step of performing image analysis and detecting the predetermined shape of the object.

Other problems disclosed in the present application and solutions thereof will be clarified in the columns and drawings of the embodiments of the invention.

According to the present invention, a predetermined shape on an object can be detected easily and quickly.

It is a figure which shows the whole structure example of one shape detection system 100 of this embodiment. It is a figure which shows the hardware configuration example of the terminal 1 which concerns on this embodiment. It is a figure which shows the functional structure example of the terminal 1 which concerns on this embodiment. The figure which shows the display example of the display area V which concerns on this embodiment is shown. It is a figure which shows the display example of the display area V which concerns on this embodiment. It is a figure explaining the feature point of the 3D point cloud data which concerns on this embodiment. It is a figure which shows the flowchart example of the shape detection method which concerns on this embodiment. It is a figure explaining the function of another shape detection system of this embodiment.

The contents of the embodiments of the present invention will be described in a list. The present invention includes, for example, the following configuration.

[Item 1]
A shape detection method that detects the shape of an object.
The step of acquiring 3D point cloud data from the object with a sensor,
The step of displaying the three-dimensional point cloud data in the display area of the terminal, and
A step of acquiring the display area in which the three-dimensional point cloud data is displayed as two-dimensional image data, and
A step of performing image analysis to determine a predetermined shape in the two-dimensional image data and detecting the predetermined shape in the object, and a step of detecting the predetermined shape.
A shape detection method comprising.
[Item 2]
The shape detection method according to item 1.
The two-dimensional coordinates of the two-dimensional image data are associated with the three-dimensional coordinates of the three-dimensional point cloud data.
Further, the step including displaying the position of the detected predetermined shape in the display area so as to be graspable on the three-dimensional data of the object is included.
A shape detection method characterized by this.
[Item 3]
The shape detection method according to item 1 or 2.
Further including a step of storing the size of the display area at the time of acquiring the two-dimensional image data.
A shape detection method characterized by this.
[Item 4]
The shape detection method according to items 1 to 3.
Further including a step of prohibiting the change of the size of the display area at the time of acquiring the two-dimensional image data.
A shape detection method characterized by this.
[Item 5]
The shape detection method according to items 1 to 4.
The predetermined shape is a concave shape of the object.
A shape detection method characterized by this.
[Item 6]
The shape detection method according to items 1 to 4.
The predetermined shape is the convex shape of the object.
A shape detection method characterized by this.
[Item 7]
The shape detection method according to items 1 to 4.
The object is a stack of two or more components.
A shape detection method characterized in that the predetermined shape is a boundary shape between the components.
[Item 8]
The shape detection method according to items 1 to 7.
Measuring the length between arbitrary positions on the two-dimensional image data,
A shape detection method characterized by this.
[Item 9]
A shape detection system that detects the shape of an object
A three-dimensional point cloud data acquisition unit that acquires three-dimensional point cloud data from the object with a sensor,
A three-dimensional point cloud data display unit that displays the three-dimensional point cloud data in the display area of the terminal,
A two-dimensional image data acquisition unit that acquires the display area in which the three-dimensional point cloud data is displayed as two-dimensional image data, and
An image analysis unit that performs image analysis to determine a predetermined shape in the two-dimensional image data and detects the predetermined shape in the object, and an image analysis unit.
A shape detection system characterized by including.
[Item 10]
A program for causing a computer to execute a shape detection method for detecting the shape of an object.
The program is used as the shape detection method.
The step of acquiring 3D point cloud data from the object with a sensor,
The step of displaying the three-dimensional point cloud data in the display area of the terminal, and
A step of acquiring the display area in which the three-dimensional point cloud data is displayed as two-dimensional image data, and
A step of performing image analysis to determine a predetermined shape in the two-dimensional image data and detecting the predetermined shape in the object, and a step of detecting the predetermined shape.
A program characterized by having a computer execute.

<Details of the embodiment>
A specific example of the shape detection system 100 according to the embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims. In the following description, the same or similar elements are given the same or similar reference numerals and names in the accompanying drawings, and duplicate description of the same or similar elements may be omitted in the description of each embodiment. In addition, the features shown in each embodiment can be applied to other embodiments as long as they do not contradict each other.

FIG. 1 is a diagram showing an example of the shape detection system 100 of the present embodiment. As shown in FIG. 1, the shape detection system 100 of the present embodiment includes a terminal 1 and a working robot 2. The working robot 2 has at least an arm 21, a tool 22, and a sensor 23. The terminal 1 and the working robot 2 are connected to each other so as to be able to communicate with each other by wire or wirelessly.

<Terminal 1>
FIG. 2 is a diagram showing a hardware configuration of the terminal 1. The terminal 1 may be a general-purpose computer such as a personal computer, or may be logically realized by cloud computing. The illustrated configuration is an example, and may have other configurations.

The terminal 1 has at least a processor 10, a memory 11, a storage 12, and a transmitter / receiver 13.
, Input / output units 14, etc., which are electrically connected to each other through the bus 15.

The processor 10 is an arithmetic unit that controls the operation of the entire terminal 1, at least controls the transmission and reception of data and the like with the working robot 2, and performs information processing and the like necessary for executing an application and performing authentication processing. For example, the processor 10 is a CPU (Central Process Unit).
ng Unit) and / or GPU (Graphics Processing U)
Nit), and each information processing is performed by executing a program or the like for this system stored in the storage 12 and expanded in the memory 11.

The memory 11 is a DRAM (Dynamic Random Access Memory).
Main memory composed of volatile storage devices such as y), flash memory and HDD (Hard)
Includes auxiliary storage configured with a non-volatile storage device such as Disk Drive). The memory 11 is used as a work area or the like of the processor 10, and also stores a BIOS (Basic Input / Output System) executed when the terminal 1 is started, various setting information, and the like.

The storage 12 stores various programs such as application programs. A database storing data used for each process may be built in the storage 12.

The transmission / reception unit 13 connects the terminal 1 to at least the work robot 2 and transmits / receives data and the like according to the instructions of the processor. The transmission / reception unit 13 is configured by wire or wireless, and in the case of wireless, for example, WiFi, Bluetooth (registered trademark) and B.
It may be configured by a short-range communication interface of LE (Bluetooth Low Energy).

The input / output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.

<Working robot 2>
Returning to FIG. 1, the working robot 2 according to the present embodiment will be described.

As described above, the working robot 2 has an arm 21, a tool 22, and a sensor 23. The illustrated configuration is an example, and may have other configurations.

The operation of the arm 21 is controlled by the terminal 1 based on the three-dimensional robot coordinate system. Further, the arm 21 may further include a controller (not shown) connected to the working robot 2 by wire or wirelessly, thereby controlling its operation.

The operation of the tool 22 is controlled by the terminal 1 based on the three-dimensional tool coordinate system. Further, the configuration of the tool 22 may include any tool according to the application, and may be, for example, a welding torch, a paint injection device for painting, a gripping device, an excavating device, a polishing device, or the like.

The sensor 23 senses an object based on a three-dimensional sensor coordinate system. Sensor 2
Reference numeral 3 denotes a laser sensor that operates as, for example, a three-dimensional scanner, and acquires three-dimensional point cloud data of an object by sensing. The three-dimensional point cloud data is as shown in FIG. 4, for example, and each point data has sensor coordinates, and the shape of the object can be grasped by the point cloud.

More specifically, the arm is subjected to a predetermined calibration before the work, the robot coordinate system, the tool coordinate system, and the sensor coordinate system are associated with each other, and the user specifies a position based on the sensor coordinate system, for example. The operation of the 21 and the tool 22 may be controlled based on the corresponding positions.

<Function of terminal 1>
FIG. 3 is a block diagram illustrating the functions implemented in the terminal 1. In the present embodiment, the processor 10 of the terminal 1 has a three-dimensional point cloud data acquisition unit 101, a three-dimensional point cloud data display unit 102, a two-dimensional image data acquisition unit 103, and an image analysis unit 104. Further, the storage 12 of the terminal 1 has a three-dimensional point cloud data storage unit 121 and a two-dimensional image data storage unit 122.
, It has an image analysis result information storage unit 123.

The three-dimensional point cloud data acquisition unit 101 controls the working robot 2 according to an instruction from the input / output unit 14 of the terminal 1, and acquires the three-dimensional point cloud data of the object by the sensor 23. The acquired three-dimensional point cloud data is, for example, three-dimensional coordinate information data based on the sensor coordinate system, and is stored in the three-dimensional point cloud data storage unit 121.

The three-dimensional point cloud data display unit 102 displays the three-dimensional point cloud data acquired by the three-dimensional point cloud data acquisition unit 101 in the display area V as illustrated in FIG. 4, for example. The user can visually recognize the three-dimensional point cloud data displayed in the display area V from any direction, and the user can use an information input device such as a keyboard or mouse connected to the input / output unit 14 to view the data. The direction can be specified.

Further, the three-dimensional point cloud data display unit 102 is based on the coordinate data (for example, two-dimensional coordinates on the two-dimensional image data) that constitutes a predetermined shape detected by the image analysis unit 104 described later, for example, in FIG. As shown, the predetermined shape is displayed in the display area V so that the predetermined shape can be recognized (for example, the coordinate data portion constituting the predetermined shape on the display area V is colored with red or the like in the white portion). .. The user can visually recognize the three-dimensional point cloud data displayed in the display area V from any direction, and the user can use an information input device such as a keyboard or mouse connected to the input / output unit 14 to view the data. The direction can be specified.

The two-dimensional image data acquisition unit 103 acquires the display of the display area V as two-dimensional image data as in a so-called screenshot. The acquired two-dimensional image data is general-purpose image data such as bitmap data, and is stored in the two-dimensional image data storage unit 122.

The image analysis unit 104 analyzes the two-dimensional image data acquired by the two-dimensional image data acquisition unit 103 under a predetermined condition, and detects a predetermined shape designated by the predetermined condition. Then, the coordinate data (for example, the two-dimensional coordinates on the two-dimensional image data) constituting the predetermined shape is stored in the image analysis result information storage unit 123. The predetermined condition is, for example, to determine that the size and length of the feature points of a predetermined color generated by the concentration of point clouds of a predetermined color are equal to or larger than a predetermined value.

Here, regarding a state in which point clouds of a predetermined color are densely packed, for example, in a metal processing field or a resin molding field, for an object that may have a minute concave shape (so-called sink mark) caused by molding shrinkage of a material or the like. The explanation will be based on the case of confirming the three-dimensional point cloud data obtained by performing the three-dimensional scan. For example, when confirming the three-dimensional point cloud data 41 from the virtual user position 42 in FIG. 6, since about one point data can be seen as a whole, the degree of density is almost uniform and there are no feature points. equal. On the other hand, the three-dimensional point cloud data 4 from the virtual user position 43 in FIG.
When 1 is confirmed, since there is a portion where two or more point data are concentrated, such a portion appears as a feature point. As a result, for example, a white feature portion such as the concave shape 31 and the convex shape 32 in FIG. 4 can be detected by image analysis.

Further, the image analysis unit 104 can detect the boundary shape between the components even when the object is a stack of two or more components under the same or similar conditions. That is, when a three-dimensional scan is performed on an overlapping structure of two or more components, point cloud data such as a kind of concave shape can be obtained as the boundary shape. Therefore, the components can be analyzed by image analysis in the same manner as described above. It becomes possible to detect the boundary shape.

Further, the image analysis unit 104 can also detect the end face and the edge of the object under the same or similar conditions. That is, since point cloud data such as a kind of convex shape can be acquired for the end face and edge of the object, it is possible to detect the end face and edge of the object by image analysis in the same manner as described above.

The image analysis unit 104 learns the relationship between the three-dimensional point group data and the portion where a predetermined shape (feature portion) such as a concave shape, a boundary shape, or a convex shape is likely to appear by machine learning or deep learning. , The user may be able to detect a predetermined shape on the two-dimensional image data by image analysis only by specifying a predetermined shape.

<Flowchart of shape detection method>
FIG. 7 is a flowchart of a shape detection method in the shape detection system 100 of the present embodiment.

First, the user operates the work robot 2 by the terminal 1 or a controller (not shown), and based on the control by the three-dimensional point cloud data acquisition unit 101 of the terminal 1, the sensor 23, for example, an object located on the work table. Acquire three-dimensional point cloud data of an object (SQ101).

Next, the three-dimensional point cloud data acquired by the three-dimensional point cloud data display unit 102 is displayed in the display area V, and the image data of the display area V is acquired by the two-dimensional image data acquisition unit 103 (
SQ102).

Next, the two-dimensional image data acquired by the image analysis unit 104 is image-analyzed under predetermined conditions to detect a predetermined shape specified by the user (SQ103).

Next, the three-dimensional point cloud data display unit 102 displays the predetermined shape on the display area V based on the detected coordinate data (for example, two-dimensional coordinates on the two-dimensional image data) constituting the predetermined shape. It is displayed so that it can be recognized (SQ104).

Therefore, since the predetermined shape on the object can be detected only by analyzing the two-dimensional image data, the shape detection system 100 of the present embodiment can easily and quickly detect the predetermined shape on the object. It is a thing.

Considering that the image analysis result is reflected based on the two-dimensional coordinates on the two-dimensional image data acquired at the step of acquiring the two-dimensional image data (SQ102), for example, at the time of acquiring the two-dimensional image data. By storing the size of the display area V, even if the size of the display area V is changed after the image analysis, the current size of the display area V (and / or its two-dimensional coordinates) and , The image analysis result can be reflected by associating the size (and / or its two-dimensional coordinates) of the display area V at the time of acquiring the two-dimensional image data. In addition, the size of the display area V may be set so as not to be changeable, at least after the image data is acquired.

<Other Embodiment 1>
In the shape detection system 100 described above, the predetermined shape is recognized on the display area V based on the coordinate data (for example, the two-dimensional coordinates on the two-dimensional image data) that constitutes the predetermined shape obtained as the image analysis result. It is displayed as possible. In addition to this (or instead), which direction the user is viewing the three-dimensional point cloud data from, and which three-dimensional point cloud data is in the display area V. By grasping whether or not it is displayed at the position of, the predetermined position of the two-dimensional coordinate system on the display area V and the predetermined position of the three-dimensional coordinate system of the three-dimensional point cloud data are associated with each other and are three-dimensional. The image analysis result may be reflected on the point cloud data.

More specifically, for example, referring to FIG. 8, the arbitrary direction specified by the user is a direction from the virtual user position 61, and a position advanced by a predetermined length from the virtual user position 61 in the arbitrary direction. The plane area of the predetermined range in is designated as the virtual display area 62, and this is displayed in the display area V. Then, for example, a virtual straight line (one-point chain line in FIG. 8) is drawn from the virtual user position 61 to the two-dimensional position 63 where the point N of the virtual display area 62 is located, and the virtual straight line is extended to obtain the three-dimensional point group data. Of these, the point M that intersects first may be obtained, and the correspondence between the coordinates of the point M and the point N may be stored.

After that, when the coordinate data constituting the predetermined shape on the two-dimensional image data is obtained by the image analysis of the image analysis unit 104, the corresponding relationship in the three-dimensional point cloud data is used by using the above-mentioned correspondence relationship. The point data may be detected and displayed so that the point data can be confirmed by being colored on the display area V.

With such a configuration, the analysis result displayed on the display area V is not limited to the two-dimensional image data at the time of image analysis, but is information such as keyboards and mice connected to the input / output unit 14. Even when the arbitrary direction is specified by the input device, the predetermined shape can be confirmed on the display area V.

In addition, the processor 10 can grasp the length between the analyzed arbitrary points on the three-dimensional point cloud data or the display area V in particular. This makes it possible to grasp the length between the positions that the user wants to recognize. For example, when the predetermined shape is a concave shape such as a groove, the length and width between the concave shapes can also be confirmed. It is possible.

<Other Embodiment 2>
The above-mentioned shape detection method is particularly useful for determining whether or not a shape does not satisfy the criteria for a plurality of objects. That is, when the point cloud data is confirmed from any direction (virtual user position) for an object having a shape that does not satisfy the standard (for example, the concave shape 31 appears in a predetermined portion), the feature portion is image-analyzed. By memorizing whether or not it can be detected in, it is possible to easily confirm whether or not the shape does not satisfy the same criteria for all the objects.

<Other Embodiment 3>
The above-mentioned shape detection method is particularly useful for determining whether or not a shape satisfies a standard for a plurality of objects. That is, the virtual user positions are set and stored in multiple directions around the object so that it can be confirmed whether the shape meets the criteria, and the analysis results of the images acquired from those directions are the image analysis results of those that meet the criteria. In comparison with, for example, the presence or absence of shape deformation, scratches, etc. may be determined.

Although the present embodiment has been described above, the above embodiment is for facilitating the understanding of the present invention, and is not for limiting and interpreting the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention also includes an equivalent thereof.

1 Terminal 2 Working robot 21 Arm 22 Tool 23 Sensor

Claims (10)

A shape detection method that detects the shape of an object.
The step of acquiring 3D point cloud data from the object with a sensor,
The step of displaying the three-dimensional point cloud data in the display area of the terminal, and
A step of acquiring the display area in which the three-dimensional point cloud data is displayed as two-dimensional image data, and
A step of performing image analysis to determine a predetermined shape in the two-dimensional image data and detecting the predetermined shape in the object, and a step of detecting the predetermined shape.
A shape detection method comprising. The shape detection method according to claim 1.
The two-dimensional coordinates of the two-dimensional image data are associated with the three-dimensional coordinates of the three-dimensional point cloud data.
Further, the step including displaying the position of the detected predetermined shape in the display area so as to be graspable on the three-dimensional data of the object is included.
A shape detection method characterized by this. The shape detection method according to claim 1 or 2.
Further including a step of storing the size of the display area at the time of acquiring the two-dimensional image data.
A shape detection method characterized by this. The shape detection method according to claims 1 to 3.
Further including a step of prohibiting the change of the size of the display area at the time of acquiring the two-dimensional image data.
A shape detection method characterized by this. The shape detection method according to claims 1 to 4.
The predetermined shape is a concave shape of the object.
A shape detection method characterized by this. The shape detection method according to claims 1 to 4.
The predetermined shape is the convex shape of the object.
A shape detection method characterized by this. The shape detection method according to claims 1 to 4.
The object is a stack of two or more components.
A shape detection method characterized in that the predetermined shape is a boundary shape between the components. The shape detection method according to claims 1 to 7.
Measuring the length between arbitrary positions on the two-dimensional image data,
A shape detection method characterized by this. A shape detection system that detects the shape of an object
A three-dimensional point cloud data acquisition unit that acquires three-dimensional point cloud data from the object with a sensor,
A three-dimensional point cloud data display unit that displays the three-dimensional point cloud data in the display area of the terminal,
A two-dimensional image data acquisition unit that acquires the display area in which the three-dimensional point cloud data is displayed as two-dimensional image data, and
An image analysis unit that performs image analysis to determine a predetermined shape in the two-dimensional image data and detects the predetermined shape in the object, and an image analysis unit.
A shape detection system characterized by including. A program for causing a computer to execute a shape detection method for detecting the shape of an object.
The program is used as the shape detection method.
The step of acquiring 3D point cloud data from the object with a sensor,
The step of displaying the three-dimensional point cloud data in the display area of the terminal, and
A step of acquiring the display area in which the three-dimensional point cloud data is displayed as two-dimensional image data, and
A step of performing image analysis to determine a predetermined shape in the two-dimensional image data and detecting the predetermined shape in the object, and a step of detecting the predetermined shape.
A program characterized by having a computer execute.

Images