WO2022244172A1

WO2022244172A1 - Information processing device, information processing system, information processing method, and information processing program

Info

Publication number: WO2022244172A1
Application number: PCT/JP2021/019117
Authority: WO
Inventors: 壮平大澤
Original assignee: 三菱電機株式会社
Priority date: 2021-05-20
Filing date: 2021-05-20
Publication date: 2022-11-24
Also published as: TW202247094A; JP7330420B2; JPWO2022244172A1

Abstract

An information processing device (100) comprises: an acquisition unit (120) that acquires position information (111) indicating a first position, point cloud information (112), an image obtained by imaging an object, and specified position information indicating a specified position; a detection unit (130) that detects a first point cloud on the basis of an image, detects a plurality of edges of an object on the basis of the first point cloud, detects a first edge from among the plurality of detected edges, detects a plurality of edges of an object on the basis of point cloud information (112) indicating a second point cloud, which serves as a plurality of feature points of the object, and detects a second edge from among the detected plurality of edges; and a process execution unit (140) that executes a first process for matching a specified position and a first position, and executes a second process for matching the first edge and the second edge so as to match the first point cloud and the second point cloud.

Description

Information processing device, information processing system, information processing method, and information processing program

The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program.

AR (Augmented Reality) technology is known. For example, in AR, a point group based on an image obtained by imaging an object is matched with a previously acquired point group. Matching involves global registration and local registration. For example, Non-Patent Document 1 describes global registration.

By the way, there are cases where devices installed in factories and the like are targeted. Then, matching is performed between the point group based on the image obtained by imaging the object and the point group acquired in advance. A method using the technology of Non-Patent Document 1 is conceivable for global registration of matching.

Here, the target object does not move, but there are cases where things that exist in factories etc. are moved. For example, objects existing around the object (eg, shelves, desks, etc.) are moved. Also, for example, an object is moved. For example, if the object is a belt conveyor, the belt conveyor moves the object on the belt conveyor. Thus, objects that exist in an environment such as a factory may not exist in a fixed position. That is, the variation in the position of an object existing in an environment such as a factory is large. The method using the technique of Non-Patent Document 1 is effective when the variation is small. Therefore, if the variation is large, using the technique of Non-Patent Document 1 may not be desirable.

The purpose of this disclosure is to perform appropriate registration even for information based on an environment that fluctuates greatly.

An information processing device according to one aspect of the present disclosure is provided. The information processing device acquires position information indicating a first position, point cloud information, an image obtained by imaging an object, and specified position information indicating a specified position specified by a user in the image. an acquisition unit that detects a plurality of feature points of the object based on the image as a first point group, detects a plurality of edges of the object based on the first point group, and detects detecting a first edge existing within a preset range from the specified position from among the plurality of edges obtained, and detecting the point group information indicating a second point group that is a plurality of characteristic points of the object; a detection unit that detects a plurality of edges of the object based on the specified executing a first process of aligning a position with the first position; executing a second process of aligning the first edge with the second edge; and a processing execution unit that aligns the first point group and the second point group by executing a process.

According to the present disclosure, appropriate alignment can be performed even with information based on an environment with large fluctuations.

4 is a diagram showing an overview of matching according to Embodiment 1; FIG. 2 is a diagram showing hardware included in the terminal device according to the first embodiment; FIG. 2 is a block diagram showing functions of the terminal device according to Embodiment 1; FIG. 4 is a flow chart showing an example of processing executed by the terminal device according to the first embodiment; FIG. 3 is a diagram showing an example of global registration according to the first embodiment; FIG. FIG. 10 is a block diagram showing functions of a terminal device according to Embodiment 2; FIG. 10 is a diagram showing a specific example of detection processing according to the second embodiment; 10 is a flow chart showing an example of processing executed by the terminal device according to the second embodiment; FIG. 11 is a block diagram showing functions of a terminal device according to Embodiment 3; FIG. 13 is a diagram illustrating an example of detection processing according to the third embodiment; FIG. 13 is a flow chart showing an example of processing executed by a terminal device according to Embodiment 3; FIG. 13 is a flow chart showing an example of processing executed by a terminal device according to a modification of the third embodiment; FIG. FIG. 12 is a block diagram showing functions of a terminal device according to Embodiment 4; FIG. 12 is a diagram showing an example of detection processing according to the fourth embodiment; FIG. FIG. 13 is a flow chart showing an example of processing executed by a terminal device according to a fourth embodiment; FIG. FIG. 12 illustrates an example of an information processing system according to a fifth embodiment; FIG.

Embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and various modifications are possible within the scope of the present disclosure.

Embodiment 1.
FIG. 1 is a diagram showing an overview of matching according to the first embodiment. FIG. 1 shows a terminal device 100 and a belt conveyor 200. As shown in FIG.
For example, the terminal device 100 is a smart phone, a tablet terminal, or the like. The terminal device 100 is also called an information processing device. The terminal device 100 is a device that executes an information processing method. Also, the terminal device 100 is a device used by a user.

The belt conveyor 200 is an example of an object. Objects present on the belt conveyor 200 are moved. That is, the objects existing on the belt conveyor 200 do not exist at fixed positions.

A user uses the terminal device 100 to capture an image of the belt conveyor 200, which is an object, in the factory. A point group based on an image obtained by imaging the belt conveyor 200 is matched with a previously obtained point group. Matching involves global registration and local registration. A point group is a collection of a plurality of feature points. For example, if the matching is successful, the CG (Computer Graphics) of the belt conveyor 200 is displayed on the display of the terminal device 100 .

In the following description, the global registration performed by the terminal device 100 will be mainly described.

Next, hardware included in the terminal device 100 will be described.
FIG. 2 illustrates hardware included in the terminal device according to the first embodiment. The terminal device 100 has a processor 101 , a volatile memory device 102 , a nonvolatile memory device 103 , an imaging device 104 and a display 105 .

The processor 101 controls the terminal device 100 as a whole. For example, the processor 101 is a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), or the like. Processor 101 may be a multiprocessor. Moreover, the terminal device 100 may have a processing circuit. The processing circuit may be a single circuit or multiple circuits.

The volatile memory device 102 is the main memory device of the terminal device 100 . For example, the volatile memory device 102 is RAM (Random Access Memory). The nonvolatile storage device 103 is an auxiliary storage device of the terminal device 100 . For example, the nonvolatile memory device 103 is a HDD (Hard Disk Drive) or an SSD (Solid State Drive).
The imaging device 104 images an object. For example, imaging device 104 is a camera. A display 105 displays information.

Next, functions of the terminal device 100 will be described.
FIG. 3 is a block diagram showing functions of the terminal device according to the first embodiment. The terminal device 100 has a storage unit 110 , an acquisition unit 120 , a detection unit 130 and a process execution unit 140 .

The storage unit 110 may be implemented as a storage area secured in the volatile storage device 102 or the nonvolatile storage device 103 .
A part or all of the acquisition unit 120, the detection unit 130, and the processing execution unit 140 may be realized by a processing circuit. Also, part or all of the acquisition unit 120, the detection unit 130, and the processing execution unit 140 may be implemented as modules of a program executed by the processor 101. FIG. For example, a program executed by the processor 101 is also called an information processing program. For example, the information processing program is recorded on a recording medium.

The storage unit 110 may store the position information 111 and the point group information 112. The position information 111 is information indicating a position. This position is also referred to as the first position. The positions are two-dimensional coordinates or three-dimensional coordinates. The point cloud information 112 is information indicating a point cloud that is a plurality of feature points of the object. The point group is also called a second point group. Point cloud information 112 may include location information 111 . Therefore, the position indicated by the position information 111 and the point group indicated by the point group information 112 can be expressed in the same coordinate system.

Also, the position information 111 may be information indicating the position of the AR object. This sentence can be expressed as follows. The position information 111 may be information indicating the position of the AR marker.

The acquisition unit 120 acquires the position information 111 and the point group information 112. For example, the acquisition unit 120 acquires the position information 111 and the point group information 112 from the storage unit 110 . Here, the position information 111 and the point cloud information 112 may be stored in an external device (for example, cloud server). When the position information 111 and the point group information 112 are stored in an external device, the acquisition unit 120 acquires the position information 111 and the point group information 112 from the external device.

Other functions of the acquisition unit 120 will be described later. Also, functions of the detection unit 130 and the processing execution unit 140 will be described later. Also, in the following description, the object is the belt conveyor 200 .

Next, processing executed by the terminal device 100 will be described using a flowchart.
4 is a flowchart illustrating an example of processing executed by the terminal device according to Embodiment 1. FIG.
(Step S11 ) The acquisition unit 120 acquires an image obtained by imaging the belt conveyor 200 with the imaging device 104 . The image may be a two-dimensional image or a three-dimensional image.
Also, the image is displayed on the display 105 . A user taps an arbitrary position. The tapped position is a specified position specified by the user in the image. Information indicating the specified position is called specified position information. The designated position information may be called tap position information.
Note that, for example, when the position information 111 indicates the position of the AR object, the user taps the position of the AR object.

(Step S12) The acquisition unit 120 acquires designated position information.
(Step S13) Based on the image, the detection unit 130 detects a plurality of characteristic points of the belt conveyor 200 as a point group. The point group is also called a first point group.

(Step S14) The detection unit 130 detects a plurality of edges of the belt conveyor 200 based on the detected point group. For example, the detection unit 130 uses the method described in Non-Patent Document 2 to detect the plurality of edges.
The detection unit 130 detects one edge existing within a preset range from the specified position indicated by the specified position information from among the plurality of edges. In other words, the detection unit 130 detects one edge existing in the vicinity of the designated position from among the plurality of edges. The edge is also called the first edge.

(Step S15 ) The detection unit 130 detects a plurality of edges of the belt conveyor 200 based on the point group information 112 . A method of detecting a plurality of edges is the same as in step S14.
The detection unit 130 detects one edge existing within a preset range from the position indicated by the position information 111 from among the plurality of edges. In other words, the detection unit 130 detects one edge existing near the position indicated by the position information 111 . This edge is also called a second edge.

(Step S16 ) The process executing unit 140 executes a process of aligning the designated position indicated by the designated position information with the position indicated by the position information 111 . This processing is also referred to as first processing.
The process executing unit 140 executes a process of combining the edges detected in step S14 and the edges detected in step S15. This process is also called a second process.
The process execution unit 140 performs the first process and the second process to match the point cloud detected in step S13 with the point cloud indicated by the point cloud information 112 .
Further, when the coordinate systems are different between the first process and the second process, the process execution unit 140 adjusts them to the same coordinate system.

(Step S17) The process executing unit 140 executes local registration.
(Step S18) The process execution unit 140 determines whether or not the point group detected in step S13 and the point group indicated by the point group information 112 match. In other words, the processing execution unit 140 determines whether or not the point group detected in step S13 and the point group indicated by the point group information 112 match. Note that the match does not have to be a perfect match. For example, when the matching rate is 80%, the processing execution unit 140 determines that the two point groups match.
If the condition is satisfied, the process proceeds to step S19. If the condition is not met, the process ends.

(Step S19 ) The processing execution unit 140 executes processing for displaying the display information associated with the point cloud information 112 on the display 105 . In addition, for example, the display information is the CG of the belt conveyor 200 .
Thereby, the display information is displayed on the display 105 .

Next, global registration will be specifically described.
FIG. 5 is a diagram showing an example of global registration according to the first embodiment. Point cloud information 112 in FIG. 5 shows a state in which position 113 indicated by position information 111 is included. Also, the point cloud information 112 may not include the point cloud of the object existing on the belt conveyor 200 .
The acquisition unit 120 acquires an image 300 obtained by imaging the belt conveyor 200 with the imaging device 104 . Acquisition unit 120 acquires specified position information.

Based on the image 300, the detection unit 130 detects a plurality of characteristic points of the belt conveyor 200 as a point group. Based on the detected point group, the detection unit 130 detects an edge 302 existing in the vicinity of the designated position 301 indicated by the designated position information.
Based on the point group information 112 , the detection unit 130 detects edges 114 that exist near the position 113 indicated by the position information 111 .

The processing execution unit 140 executes processing for aligning the specified position 301 indicated by the specified position information with the position 113 indicated by the position information 111 . The processing execution unit 140 executes processing for combining the edge 302 and the edge 114 . By executing these processes, the processing execution unit 140 matches the point cloud detected based on the image 300 with the point cloud indicated by the point cloud information 112 .

According to Embodiment 1, the terminal device 100 can perform appropriate alignment even for an image based on an environment with large fluctuations.

In addition, the above explained the case where the point cloud is detected based on the image. The point cloud may be detected as follows. First, the acquisition unit 120 acquires information from the infrared sensor. The detection unit 130 detects the point cloud based on the information acquired from the infrared sensor. Note that the point group is also referred to as a first point group.

Embodiment 2.
Next, Embodiment 2 will be described. In Embodiment 2, mainly matters different from Embodiment 1 will be described. In the second embodiment, descriptions of items common to the first embodiment are omitted.
In the first embodiment, the case of acquiring the specified position information indicating the position specified by the user has been described. In the second embodiment, a case of automatically detecting a position to be aligned with the position information 111 will be described.

FIG. 6 is a block diagram showing functions of the terminal device according to the second embodiment. The terminal device 100a has an acquisition unit 120a, a detection unit 130a, and a process execution unit 140a.
First, the position information 111 indicates the position of the characteristic part of the belt conveyor 200 . This position is also referred to as the first position.

The obtaining unit 120a differs from the obtaining unit 120 in that it does not obtain the designated position information.
The detection unit 130a detects the position of the characteristic part of the belt conveyor 200 based on the image. 4 illustrates the detection process.

FIG. 7 is a diagram illustrating a specific example of detection processing according to the second embodiment. The terminal device 100a generates an image 310 by capturing an image of the belt conveyor 200. FIG. Image 310 includes switch 311 and signal 312 .
The detection unit 130 a detects the position of the characteristic part of the belt conveyor 200 . For example, the detection unit 130a detects the position of the characteristic part of the belt conveyor 200 using general object recognition technology. For example, the detected feature is the switch 311 . Also, for example, the detected feature location is the signal 312 .

Other functions of the detection unit 130a will be described later. The function of the processing execution unit 140a will be described later.

Next, processing executed by the terminal device 100a will be described using a flowchart.
8 is a flowchart illustrating an example of processing executed by the terminal device according to the second embodiment; FIG.
(Step S21 ) The acquisition unit 120 a acquires an image obtained by imaging the belt conveyor 200 with the imaging device 104 . The image may be a two-dimensional image or a three-dimensional image.
(Step S22) The detection unit 130a detects one characteristic position of the belt conveyor 200 based on the image.
(Step S23) Based on the image, the detection unit 130a detects a plurality of characteristic points of the belt conveyor 200 as a point group. The point group is also called a first point group.

(Step S24) The detection unit 130a detects a plurality of edges of the belt conveyor 200 based on the detected point group.
The detection unit 130a detects one edge existing within a preset range from the position of the characteristic location from among the plurality of edges. In other words, the detection unit 130a detects one edge existing in the vicinity of the position of the characteristic location from among the plurality of edges. The edge is also called the first edge.

(Step S25 ) The detection unit 130 a detects a plurality of edges of the belt conveyor 200 based on the point group information 112 .
The detection unit 130a detects one edge existing within a preset range from the position indicated by the position information 111 (that is, the position of the characteristic part of the belt conveyor 200) from among the plurality of edges. In other words, the detection unit 130a detects one edge that exists in the vicinity of the position indicated by the position information 111. FIG. This edge is also called a second edge.

(Step S26 ) The process execution unit 140 a executes a process of aligning the position of the feature location with the position indicated by the position information 111 . This processing is also referred to as first processing.
The process executing unit 140a executes a process of combining the edges detected in step S24 and the edges detected in step S25. This process is also called a second process.
The process execution unit 140a matches the point cloud detected in step S23 with the point cloud indicated by the point cloud information 112 by performing the first process and the second process.

(Step S27) The process executing unit 140a executes local registration.
(Step S28) The process execution unit 140a determines whether or not the point group detected in step S23 and the point group indicated by the point group information 112 match. If the condition is satisfied, the process proceeds to step S29. If the condition is not met, the process ends.
(Step S29 ) The processing execution unit 140 a executes processing for displaying display information associated with the point cloud information 112 on the display 105 .

According to Embodiment 2, the terminal device 100a automatically detects the position of the feature location. Therefore, in the second embodiment, unlike in the first embodiment, the user does not have to perform the designated operation. Therefore, according to Embodiment 2, the terminal device 100a can reduce the burden on the user.

Embodiment 3.
Next, Embodiment 3 will be described. In the third embodiment, mainly matters different from the first embodiment will be described. In the third embodiment, descriptions of matters common to the first embodiment are omitted.
In the first embodiment, the case of detecting edges using an image has been described. In the third embodiment, a case of detecting lines using tracking technology will be described.

FIG. 9 is a block diagram showing functions of the terminal device according to the third embodiment. The terminal device 100b has an acquisition unit 120b, a detection unit 130b, and a process execution unit 140b.
Main functions of the detection unit 130b will be described with reference to FIG. Detailed functions of the acquisition unit 120b, the detection unit 130b, and the processing execution unit 140b will be described later.
The detection unit 130b detects the trajectory of the object. In other words, the detection unit 130b detects the movement trajectory of the object. 4 illustrates the detection process.

FIG. 10 is a diagram showing an example of detection processing according to the third embodiment. FIG. 10 shows conveyor belt 200 moving object 320 . The detection unit 130b detects the trajectory of the object 320 using tracking technology. For example, the detection unit 130 b detects the trajectory of the object 320 by tracking the corner 321 of the object 320 . The detection unit 130b detects the straight line 322 based on the trajectory.

Also, the detection unit 130b may detect the straight line 322 based on the trajectories of a plurality of objects. Furthermore, when the trajectory is a curve, the detection unit 130b detects the curve based on the trajectory.

Next, processing executed by the terminal device 100b will be described using a flowchart.
11 is a flowchart illustrating an example of processing executed by the terminal device according to the third embodiment; FIG.
(Step S31) The acquisition unit 120b acquires an image obtained by imaging the belt conveyor 200 that moves the object.
Also, the image is displayed on the display 105 . A user taps an arbitrary place. The tapped place is the specified position specified by the user. Information indicating the specified position is called specified position information.

(Step S32) The acquisition unit 120b acquires designated position information.
(Step S33) Based on the image, the detection unit 130b detects a plurality of characteristic points of the belt conveyor 200 as a point group. Specifically, the detection unit 130b detects a plurality of characteristic points of the belt conveyor 200 as a point group based on one of the plurality of images that are the videos. The point group is also called a first point group.
(Step S34) The detection unit 130b detects the trajectory of the object on the belt conveyor 200 based on the image. The detection unit 130b detects lines based on the trajectory. The line is also called the first line.

(Step S35 ) The detection unit 130 b detects a plurality of edges of the belt conveyor 200 based on the point group information 112 .
The detection unit 130b detects one edge existing within a preset range from the position indicated by the position information 111 from among the plurality of edges. In other words, the detection unit 130b detects one edge that exists in the vicinity of the position indicated by the position information 111. FIG. This edge is also called a second edge.

(Step S36) The process executing unit 140b executes a process of matching the specified position indicated by the specified position information and the position indicated by the position information 111. FIG. This processing is also referred to as first processing.
The process executing unit 140b executes a process of matching the line detected in step S34 and the edge detected in step S35. This process is also called a second process.
The process execution unit 140b matches the point cloud detected in step S33 with the point cloud indicated by the point cloud information 112 by executing the first process and the second process.

(Step S37) The process executing unit 140b executes local registration.
(Step S38) The process execution unit 140b determines whether or not the point group detected in step S33 and the point group indicated by the point group information 112 match. If the condition is satisfied, the process proceeds to step S39. If the condition is not met, the process ends.
(Step S39 ) The processing execution unit 140 b executes processing for displaying display information associated with the point cloud information 112 on the display 105 .

According to Embodiment 3, the terminal device 100b can perform appropriate alignment even for images based on an environment with large fluctuations.

Modification of Embodiment 3.
A modification of the third embodiment describes a case where the second embodiment and the third embodiment are combined. In addition, the code|symbol of Embodiment 2 is used for a code|symbol.

FIG. 12 is a flowchart showing an example of processing executed by a terminal device according to a modification of the third embodiment. The process of FIG. 12 differs from the process of FIG. 8 in that steps S21a to 24a and 26a are executed. Therefore, in FIG. 12, steps S21a to 24a and 26a will be explained. Further, description of processes other than steps S21a to 24a and 26a is omitted.

(Step S21a) The acquisition unit 120a acquires an image obtained by imaging the belt conveyor 200 that moves the object.
(Step S22a) The detection unit 130a detects the position of the characteristic part of the belt conveyor 200 based on the image.

(Step S23a) Based on the image, the detection unit 130a detects a plurality of feature points of the belt conveyor 200 as a point group. Specifically, the detection unit 130a detects a plurality of feature points of the belt conveyor 200 as a point group based on one of the plurality of images that are the video. The point group is also called a first point group.
(Step S24a) The detection unit 130a detects the trajectory of the object on the belt conveyor 200 based on the image. The detection unit 130a detects a line based on the trajectory. The line is also called the first line.

(Step S26 a ) The process execution unit 140 a executes a process of matching the position of the characteristic location and the position indicated by the position information 111 . This processing is also referred to as first processing.
The process executing unit 140a executes a process of matching the line detected in step S24a and the edge detected in step S25. This process is also called a second process.
The process execution unit 140a matches the point cloud detected in step S23a with the point cloud indicated by the point cloud information 112 by executing the first process and the second process.

According to the modified example of Embodiment 3, appropriate alignment is possible even for images based on an environment with large fluctuations.

Embodiment 4.
Next, Embodiment 4 will be described. In Embodiment 4, mainly matters different from Embodiment 1 will be described. In the fourth embodiment, descriptions of items common to the first embodiment are omitted.
In Embodiment 1, alignment between points is performed. In the fourth embodiment, a case in which points are not aligned will be described.

FIG. 13 is a block diagram showing functions of the terminal device according to the fourth embodiment. The terminal device 100c has an acquisition unit 120c, a detection unit 130c, and a process execution unit 140c.
The acquisition unit 120 c differs from the acquisition unit 120 in that it does not acquire the position information 111 . Therefore, the storage unit 110 does not have to store the position information 111 .
Main functions of the detection unit 130c will be described. The detection unit 130c detects a plane area of the object based on the point group. 4 illustrates the detection process.

FIG. 14 is a diagram showing an example of detection processing according to the fourth embodiment. FIG. 14 shows an image 330 generated by the terminal device 100c. The detection unit 130c detects a point cloud based on the image 330. FIG. The detection unit 130c detects the plane area 331 of the belt conveyor 200 based on the point group.

It is desirable that the planar area be a characteristic planar area. For example, the planar regions may be triangles, hexagons, circles, and the like. Further, the detection unit 130c may detect a three-dimensional area in the object based on the point group. The three-dimensional shape is desirably a characteristic shape. Note that the plane area detected based on the point group is also called a first area. A three-dimensional region detected based on the point group is also referred to as a first region.

The detection unit 130c detects a plurality of planar regions of the object based on the point group information 112 when planar regions are detected based on the image. Further, when a three-dimensional region is detected based on the image, the detection unit 130 c detects a plurality of three-dimensional regions in the object based on the point group information 112 .

Other functions of the detection unit 130c will be described later. The function of the processing execution unit 140c will be described later.

Next, processing executed by the terminal device 100c will be described using a flowchart.
15 is a flowchart illustrating an example of processing executed by the terminal device according to the fourth embodiment; FIG.
(Step S41 ) The acquisition unit 120 c acquires an image obtained by imaging the belt conveyor 200 with the imaging device 104 . The image may be a two-dimensional image or a three-dimensional image.
(Step S42) Based on the image, the detection unit 130c detects a plurality of characteristic points of the belt conveyor 200 as a point group. The point group is also called a first point group.
(Step S43) The detection unit 130c detects the plane area of the belt conveyor 200 based on the detected point group.

(Step S44 ) Based on the point group information 112 , the detection unit 130 c detects multiple areas that are multiple plane areas of the belt conveyor 200 .
(Step S45) The process execution unit 140c identifies the planar area detected in step S43 from among the plurality of areas (that is, the plurality of planar areas). This processing is also referred to as first processing.

(Step S46) The process execution unit 140c executes a process of matching the plane area detected in step S43 and the plane area specified in step S45. This process is also called a second process.
The processing execution unit 140c matches the point cloud detected in step S42 with the point cloud indicated by the point cloud information 112 by executing the second processing.

(Step S47) The process executing unit 140c executes local registration.
(Step S48) The process execution unit 140c determines whether or not the point group detected in step S42 and the point group indicated by the point group information 112 match. If the condition is satisfied, the process proceeds to step S49. If the condition is not met, the process ends.
(Step S49 ) The processing execution unit 140 c executes processing for displaying display information associated with the point group information 112 on the display 105 .

According to Embodiment 4, the terminal device 100c can perform appropriate alignment even for an image based on an environment with large fluctuations.

Embodiment 5.
Next, Embodiment 5 will be described. In Embodiment 5, mainly matters different from Embodiments 1 to 4 will be described. Further, in the fifth embodiment, descriptions of matters common to the first to fourth embodiments are omitted.

FIG. 16 is a diagram illustrating an example of an information processing system according to a fifth embodiment; The information processing system includes an information processing device 400 and a terminal device 500 . The information processing device 400 and the terminal device 500 communicate via a network.
The information processing device 400 is a device that executes an information processing method. The information processing device 400 has a processor, a volatile memory device, and a non-volatile memory device. Further, the information processing device 400 may have a processing circuit.
The terminal device 500 has an imaging device for imaging an object and a display.

In Embodiments 1 to 4, cases where

terminal devices

100, 100a, 100b, and 100c perform processing have been described. The information processing device 400 has the functions of the

terminal devices

100, 100a, 100b, and 100c.

For example, the information processing apparatus 400 includes position information 111 indicating a first position, point cloud information 112, an image obtained by capturing an image of a target object by the imaging device, and designation specified by the user in the image. Acquire the specified position information indicating the position. For example, the information processing device 400 acquires the position information 111 indicating the first position and the point group information 112 from a volatile or nonvolatile storage device. Also, for example, the information processing device 400 acquires the image and the designated position information from the terminal device 500 . The information processing device 400 detects a plurality of feature points of the object as a first point group based on the image. The information processing device 400 detects multiple edges of the object based on the first point group. The information processing apparatus 400 detects a first edge existing within a preset range from the specified position from among the plurality of detected edges. The information processing device 400 detects multiple edges of the object based on the point group information 112 indicating the second point group, which is the multiple feature points of the object. The information processing device 400 detects a second edge existing within a range from the first position from among the plurality of detected edges. The information processing device 400 executes a first process of aligning the specified position with the first position. The information processing device 400 executes a second process of aligning the first edge and the second edge. The information processing apparatus 400 aligns the first point group and the second point group by executing the first process and the second process.
In addition, for example, the information processing apparatus 400 includes position information 111 indicating a first position, which is a position of a characteristic part of an object, point cloud information 112, and an image obtained by imaging the object with the imaging device. to get The information processing device 400 detects the first position based on the image. The information processing device 400 detects a plurality of feature points of the object as a first point group based on the image. The information processing device 400 detects multiple edges of the object based on the first point group. The information processing apparatus 400 detects a first edge existing within a preset range from the detected first position from among the detected edges. The information processing device 400 detects multiple edges of the object based on the point group information 112 indicating the second point group, which is the multiple feature points of the object. The information processing device 400 detects a second edge existing within a range from the first position indicated by the position information 111 from among the detected edges. The information processing device 400 executes a first process of aligning the detected first position with the first position indicated by the position information 111 . The information processing device 400 executes a second process of aligning the first edge and the second edge. The information processing apparatus 400 aligns the first point group and the second point group by executing the first process and the second process.

Also, for example, the information processing device 400 acquires the point cloud information 112 and an image obtained by imaging the target object. The information processing device 400 detects a plurality of feature points of the object as a first point group based on the image. The information processing apparatus 400 detects a first area, which is a planar area of the object or a three-dimensional area in the object, based on the first point group. Based on the point group information 112 indicating the second point group, which is a plurality of characteristic points of the object, the information processing apparatus 400 detects a plurality of planar regions of the object or a plurality of three-dimensional regions of the object. to detect the region of The information processing device 400 executes a first process of identifying a first area from among the plurality of areas. The information processing device 400 executes a second process of matching the first region of the first point group and the first region of the second point group. The information processing apparatus 400 aligns the first point group and the second point group by executing the first process and the second process.

Further, for example, the information processing apparatus 400 executes local registration, and if the first point group and the second point group match, the information processing apparatus 400 converts the display information associated with the point group information 112 to the relevant point group. Perform processing for displaying on the display. Specifically, the information processing device 400 transmits display information and an instruction to display the display information to the terminal device 500 . Thereby, the display information is displayed on the display of the terminal device 500 .

Thus, the information processing device 400 has the functions of the

terminal devices

100, 100a, 100b, and 100c.

According to the fifth embodiment, the information processing device 400 can achieve the same effects as those described in the first to fourth embodiments.

The features of each embodiment described above can be combined as appropriate.

100, 100a, 100b, 100c terminal device, 101 processor, 102 volatile storage device, 103 non-volatile storage device, 104 imaging device, 105 display, 110 storage unit, 111 position information, 112 point group information, 113 position, 114 edge , 120, 120a, 120b, 120c acquisition unit, 130, 130a, 130b, 130c detection unit, 140, 140a, 140b, 140c processing execution unit, 200 belt conveyor, 300 image, 301 designated position, 302 edge, 310 image, 311 Switch, 312 signal, 320 object, 321 angle, 322 straight line, 330 image, 331 plane area, 400 information processing device, 500 terminal device.

Claims

an acquisition unit that acquires position information indicating a first position, point cloud information, an image obtained by imaging an object, and specified position information indicating a specified position specified by a user in the image;
detecting a plurality of feature points of the object based on the image as a first point group; detecting a plurality of edges of the object based on the first point group; and detecting a plurality of detected edges. A first edge existing within a preset range from the specified position is detected from among, and based on the point group information indicating a second point group that is a plurality of feature points of the object, a detection unit that detects a plurality of edges of the object and detects a second edge existing within the range from the first position from among the plurality of detected edges;
executing a first process of aligning the specified position with the first position; executing a second process of aligning the first edge with the second edge; a processing execution unit that aligns the first point group and the second point group by executing the processing of 2;
Information processing device having
wherein the first position is the position of an AR object;
The information processing device according to claim 1 .
The acquisition unit acquires an image obtained by imaging the object that moves the object,
The detection unit detects the first point group based on the image, detects a trajectory of the object based on the image, detects a first line based on the trajectory,
The processing execution unit aligns the first line and the second edge in the second processing.
The information processing apparatus according to claim 1 or 2.
an acquisition unit that acquires position information indicating a first position, which is the position of a characteristic part of an object, point cloud information, and an image obtained by imaging the object;
detecting the first position based on the image; detecting a plurality of characteristic points of the object based on the image as a first point group; and detecting the object based on the first point group detecting a plurality of edges of the object, detecting a first edge existing within a preset range from the detected first position from among the plurality of detected edges, and detecting a plurality of edges of the object A plurality of edges of the object are detected based on the point group information indicating a second point group that is feature points, and the first position indicated by the position information is selected from the plurality of detected edges. a detection unit that detects a second edge existing within the range from
executing a first process of aligning the detected first position with the first position indicated by the position information, and executing a second process of aligning the first edge with the second edge; and a processing execution unit that combines the first point group and the second point group by executing the first processing and the second processing;
Information processing device having
The acquisition unit acquires an image obtained by imaging the object that moves the object,
The detection unit detects the first position based on the image, detects the first point group based on the image, detects a trajectory of the object based on the image, and detects the trajectory of the object based on the image. detecting the first line based on
The processing execution unit aligns the first line and the second edge in the second processing.
The information processing apparatus according to claim 4.
an acquisition unit that acquires point cloud information and an image obtained by imaging an object;
detecting a plurality of feature points of the object based on the image as a first point group, and detecting a planar area of the object or a three-dimensional shape in the object based on the first point group; A plurality of plane regions of the object or a plurality of plane regions of the object are detected based on the point group information indicating a second point group that is a plurality of feature points of the object. a detection unit that detects a plurality of areas that are three-dimensional areas of
performing a first process of specifying the first area from among the plurality of areas, and identifying the first area of the first point group and the first area of the second point group; a processing execution unit that performs a second process of matching, and performs the second process to match the first point group and the second point group;
Information processing device having
The acquisition unit acquires information from an infrared sensor,
The detection unit detects the first point group based on information acquired from the infrared sensor.
The information processing apparatus according to any one of claims 1 to 6.
an imaging device that images the object;
a display;
further having
The processing execution unit
perform local registration,
When the first point group and the second point group match, performing processing for displaying display information associated with the point group information on the display;
The information processing apparatus according to any one of claims 1 to 7.
a terminal device having an imaging device for imaging an object and a display;
an information processing device;
including
The information processing device is
Acquiring position information indicating a first position, point cloud information, an image obtained by imaging the object with the imaging device, and specified position information indicating a specified position specified by a user in the image. ,
Detecting a plurality of feature points of the object based on the image as a first point group,
detecting a plurality of edges of the object based on the first point cloud;
Detecting a first edge existing within a preset range from the specified position from among the plurality of detected edges;
Detecting a plurality of edges of the object based on the point group information indicating a second point group that is a plurality of feature points of the object;
Detecting a second edge existing within the range from the first position from among the plurality of detected edges;
performing a first process of aligning the specified position with the first position;
performing a second process of aligning the first edge and the second edge;
By performing the first process and the second process, the first point group and the second point group are combined,
Information processing system.
a terminal device having an imaging device for imaging an object and a display;
an information processing device;
including
The information processing device is
Acquiring position information indicating a first position, which is the position of a characteristic part of the object, point cloud information, and an image obtained by imaging the object with the imaging device,
detecting the first position based on the image;
Detecting a plurality of feature points of the object based on the image as a first point group,
detecting a plurality of edges of the object based on the first point cloud;
Detecting a first edge existing within a preset range from the detected first position from among the plurality of detected edges;
Detecting a plurality of edges of the object based on the point group information indicating a second point group that is a plurality of feature points of the object;
detecting a second edge existing within the range from the first position indicated by the position information from among the detected edges;
performing a first process of aligning the detected first position with the first position indicated by the position information;
performing a second process of aligning the first edge and the second edge;
By performing the first process and the second process, the first point group and the second point group are combined,
Information processing system.
a terminal device having an imaging device for imaging an object and a display;
an information processing device;
including
The information processing device is
Acquiring point cloud information and an image obtained by imaging the object,
Detecting a plurality of feature points of the object based on the image as a first point group,
Detecting a first area, which is a planar area of the object or a three-dimensional area in the object, based on the first point group;
Based on the point group information indicating a second point group that is a plurality of characteristic points of the object, a plurality of areas that are a plurality of planar areas of the object or a plurality of areas that are a plurality of three-dimensional shape areas of the object are determined. detect and
performing a first process of identifying the first area from among the plurality of areas;
performing a second process of matching the first region of the first point group and the first region of the second point group;
By performing the first process and the second process, the first point group and the second point group are combined,
Information processing system.
The information processing device
Acquiring position information indicating a first position, point cloud information, an image obtained by imaging an object, and specified position information indicating a specified position specified by a user in the image,
Detecting a plurality of feature points of the object based on the image as a first point group,
detecting a plurality of edges of the object based on the first point cloud;
Detecting a first edge existing within a preset range from the specified position from among the plurality of detected edges;
Detecting a plurality of edges of the object based on the point group information indicating a second point group that is a plurality of feature points of the object;
Detecting a second edge existing within the range from the first position from among the plurality of detected edges;
performing a first process of aligning the specified position with the first position;
performing a second process of aligning the first edge and the second edge;
By performing the first process and the second process, the first point group and the second point group are combined,
Information processing methods.
The information processing device
Acquiring position information indicating a first position, which is the position of a characteristic part of an object, point cloud information, and an image obtained by imaging the object,
detecting the first position based on the image;
Detecting a plurality of feature points of the object based on the image as a first point group,
detecting a plurality of edges of the object based on the first point cloud;
Detecting a first edge existing within a preset range from the detected first position from among the plurality of detected edges;
Detecting a plurality of edges of the object based on the point group information indicating a second point group that is a plurality of feature points of the object;
detecting a second edge existing within the range from the first position indicated by the position information from among the detected edges;
performing a first process of aligning the detected first position with the first position indicated by the position information;
performing a second process of aligning the first edge and the second edge;
By performing the first process and the second process, the first point group and the second point group are combined,
Information processing methods.
The information processing device
Acquiring point cloud information and an image obtained by imaging an object,
Detecting a plurality of feature points of the object based on the image as a first point group,
Detecting a first area, which is a planar area of the object or a three-dimensional area in the object, based on the first point group;
Based on the point group information indicating a second point group that is a plurality of characteristic points of the object, a plurality of areas that are a plurality of planar areas of the object or a plurality of areas that are a plurality of three-dimensional shape areas of the object are determined. detect and
performing a first process of identifying the first area from among the plurality of areas;
performing a second process of matching the first region of the first point group and the first region of the second point group;
By executing the second process, the first point group and the second point group are combined,
Information processing methods.
information processing equipment,
Acquiring position information indicating a first position, point cloud information, an image obtained by imaging an object, and specified position information indicating a specified position specified by a user in the image,
Detecting a plurality of feature points of the object based on the image as a first point group,
detecting a plurality of edges of the object based on the first point cloud;
Detecting a first edge existing within a preset range from the specified position from among the plurality of detected edges;
Detecting a plurality of edges of the object based on the point group information indicating a second point group that is a plurality of feature points of the object;
Detecting a second edge existing within the range from the first position from among the plurality of detected edges;
performing a first process of aligning the specified position with the first position;
performing a second process of aligning the first edge and the second edge;
By performing the first process and the second process, the first point group and the second point group are combined,
An information processing program that causes a process to be executed.
information processing equipment,
Acquiring position information indicating a first position, which is the position of a characteristic part of an object, point cloud information, and an image obtained by imaging the object,
detecting the first position based on the image;
Detecting a plurality of feature points of the object based on the image as a first point group,
detecting a plurality of edges of the object based on the first point cloud;
Detecting a first edge existing within a preset range from the detected first position from among the plurality of detected edges;
Detecting a plurality of edges of the object based on the point group information indicating a second point group that is a plurality of feature points of the object;
detecting a second edge existing within the range from the first position indicated by the position information from among the detected edges;
performing a first process of aligning the detected first position with the first position indicated by the position information;
performing a second process of aligning the first edge and the second edge;
By performing the first process and the second process, the first point group and the second point group are combined,
An information processing program that causes a process to be executed.
information processing equipment,
Acquiring point cloud information and an image obtained by imaging an object,
Detecting a plurality of feature points of the object based on the image as a first point group,
Detecting a first area, which is a planar area of the object or a three-dimensional area in the object, based on the first point group;
Based on the point group information indicating a second point group that is a plurality of characteristic points of the object, a plurality of areas that are a plurality of planar areas of the object or a plurality of areas that are a plurality of three-dimensional shape areas of the object are determined. detect and
performing a first process of identifying the first area from among the plurality of areas;
performing a second process of matching the first region of the first point group and the first region of the second point group;
By executing the second process, the first point group and the second point group are combined,
An information processing program that causes a process to be executed.