JP2022037856A - Program, method and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the burden of a worker in visual inspection.

SOLUTION: A program causes a computer including a processor and a memory to execute processing. The program causes the processor to execute the steps of: acquiring a three-dimensional shape of an object based on sensing data obtained by sensing using electromagnetic waves to the object; detecting abnormality of the object based on the sensing data; specifying a three-dimensional position of the detected abnormality; moving a robot arm including a device capable of resolving the abnormality, mounted thereon, in order to deliver the device to the specified three-dimensional position; and when delivering the device to the three-dimensional position, activating the device.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

This disclosure relates to programs, methods, and systems.

Conventionally, the manufacturing process at a manufacturing site includes various processes such as a process of assembling an object and a process of painting the body of an object.
In the painting process, for example, painting of an object is advanced through a plurality of procedures using a robot arm or the like. Then, when the painting is completed, the final confirmation of the object as a manufactured object is performed, and it is confirmed that there is no abnormality in the manufactured object. If, for example, an abnormality such as discoloration, cracks, adhesion of dust, or uneven coating is found during the visual inspection of the object, repair is performed to eliminate the abnormality. For example, when uneven coating is found, the area with uneven coating is polished with a polisher to make the area painted uniform.

Japanese Unexamined Patent Publication No. 2000-205846

In the visual inspection, the detection of the abnormality and the elimination of the found abnormality are performed by the workers. However, the detection of anomalies and the elimination of the discovered anomalies require high concentration, which is a heavy burden on the workers.
Patent Document 1 discloses an inspection method / apparatus for optically detecting coating unevenness, but does not describe detection other than coating unevenness.

An object of the present disclosure is to reduce the burden on workers in visual inspection.

One aspect of the present disclosure is a program for causing a computer including a processor and a memory to execute. The program has a step of acquiring the three-dimensional shape of the object based on the sensing data obtained by sensing the object using electromagnetic waves to the object, a step of detecting the abnormality of the object based on the sensing data, and a step of detecting the detected abnormality. A step to specify a 3D position, a step to move a robot arm equipped with a device capable of eliminating an abnormality so that the device reaches the specified 3D position, and a step to move the device to the 3D position. , Steps to get the device up and running.

According to the program of the present invention, the burden on the worker can be reduced.

It is a figure which shows the whole structure of the system which concerns on this embodiment. It is a block diagram which shows the structure of a control device. It is a figure which shows the data structure of the 3D position information of an abnormality part. It is a figure for demonstrating the flow of detecting an abnormality on a painted surface of an object by a system, and eliminating the detected abnormality. It is a schematic diagram which shows the abnormal part in the painted surface of an object. It is a figure for demonstrating the operation of an industrial robot. It is a figure which shows the structure of the system which concerns on the modification 1. It is a figure which shows the structure of the system which concerns on the modification 2. It is a figure which shows the structure of the system which concerns on the modification 3. It is a figure which shows the structure of the system which concerns on the modification 4.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiments, the same components are designated by the same reference numerals in principle, and the repeated description thereof will be omitted.

<Overview>
The system 1 according to the present embodiment is, for example, a system for inspecting a painted surface of a painted object and performing work for eliminating the abnormality when an abnormality occurs on the painted surface. The system 1 is used, for example, in an automobile production line.

<Overall configuration>
FIG. 1 is a diagram showing an overall configuration of the system 1 according to the present embodiment. The system 1 shown in FIG. 1 inspects the painted surface of the object 100. Although the illustration of the object 100 is simplified, it is a work such as an outer panel of an automobile body, which has finished the painting process and is not to be visually inspected. The object 100 is not limited to the outer panel of the body of an automobile, and various workpieces that refrain from visual inspection are assumed.

The system 1 includes a sensing unit 10, an industrial robot 20, a control device 30, and a display device 40. The sensing unit 10, the industrial robot 20, and the control device 30 are connected to the network 50 so as to be able to communicate with each other. The connection of the sensing unit 10, the industrial robot 20, and the control device 30 to the network 50 may be wired or wireless. Further, the sensing unit 10 and the industrial robot 20 may be connected to the control device 30 without going through the network 50. The control device 30 is connected to the display device 40 by wire or wirelessly. The control device 30 and the display device 40 may be connected via the network 50.

The sensing unit 10 senses the object 100 using electromagnetic waves. The sensing unit 10 includes a sensing module (sensing means) 11 for sensing and an actuator (not shown) for controlling the position of the sensing module 11.

The sensing module 11 senses the object 100 using electromagnetic waves in a preset band. For the sensing module 11, for example, any of the following devices realized by using electromagnetic waves in a preset band is assumed.
・ Visible light camera ・ Infrared camera ・ Ultraviolet camera ・ Ultrasonic sensor ・ RGB-D camera ・ LiDAR (Light Detection and Ranging)

A plurality of sensing modules 11 are arranged around the object 100 so that the three-dimensional positions of abnormalities that may occur on the painted surface of the object 100 can be acquired. For example, the sensing module 11 is arranged at a position where a predetermined surface of the object 100 can be simultaneously sensed from different directions. In the example shown in FIG. 1, two sensing modules 11 are arranged around the object 100, but the number of sensing modules 11 arranged around the object 100 may be three or more.

The sensing unit 10 transmits the sensing data acquired by the sensing module 11 to the control device 30.
As the actuator of the sensing unit 10, for example, a Cartesian robot capable of moving the position of the sensing module 11 along the XY axis is used. The behavior of the orthogonal robot is controlled by the control device 30. The actuator may be, for example, a single-axis robot capable of moving the sensing module 11 along one axis.

The industrial robot 20 is, for example, a vertical articulated robot. The industrial robot 20 includes, for example, an articulated robot arm 21 and an end effector 22 mounted on the tip of the robot arm 21. The industrial robot 20 is communicatively connected to the control device 30.

The robot arm 21 is driven according to the control from the control device 30. Specifically, in the robot arm 21, a motor is provided in the joint, and the arm portion is moved by operating the joint by the force of the motor.

The end effector 22 is a device that performs a predetermined process. In the present embodiment, it is a device for eliminating an abnormality that has occurred on the painted surface of the object 100. The end effector 22 is moved to a desired position by the robot arm 21. Examples of the end effector 22 include an electric polisher that eliminates coating unevenness (abnormality) generated on the outer panel after painting by polishing. The end effector 22 is not limited to the electric polisher as long as it is a device that can eliminate the abnormality generated on the painted surface of the object 100.

The control device 30 controls the operation of each device in the system 1.

The display device 40 presents the information processed by the control device 30 to the user.

<Control device configuration>
Next, the configuration of the control device 30 will be described.
FIG. 2 is a block diagram showing the configuration of the control device 30. The control device 30 shown in FIG. 2 includes a processor 31, a storage device 32, a communication interface 33, and an input / output interface 34.

The storage device 32 is realized by a non-volatile storage circuit such as an HDD (hard disk drive) or SSD (solid state drive) that stores various information. The storage device 32 stores various programs executed by the processor 31, data processed by the programs, and the like. For example, the storage device 32 stores three-dimensional position information described later.

Further, the storage device 32 stores, for example, a plurality of trained models generated by machine learning. The first trained model is, for example, a model for detecting an abnormality in the painted surface of the object 100. The second trained model is, for example, a model for controlling the operation of the industrial robot 20. The first trained model and the second trained model may be stored in advance at the time of shipment of the control device 30, and may be installed via the network 50 or via a storage medium after shipment.

The first trained model and the second trained model are obtained by causing a machine learning model to perform machine learning according to a model learning program based on training data.

For example, in the present embodiment, the first trained model is trained to output an abnormality on the painted surface with respect to the input sensing data. At this time, for the learning data, for example, a plurality of sensing data about the object are used as input data, and a judgment about an abnormality that can be included in the input data is used as correct output data.

Further, for example, the second trained model is trained to output the control parameters of the industrial robot 20 with respect to the input three-dimensional position information. At this time, for the learning data, for example, the position information about the object having a predetermined three-dimensional shape is used as the input data, and the control parameter of the robot arm 21 that the end effector 22 can access to the input data is the correct output data. And. As a result, the parameters that the end effector 22 can reach the input three-dimensional position are set in consideration of the shape of the robot arm 21 and the three-dimensional shape of the object 100. That is, in the process of reaching the three-dimensional position of the end effector 22, the parameter for driving the robot arm 21 is set without the robot arm 21 and the object 100 coming into contact with each other.

The communication interface 33 is realized by, for example, a circuit connected to the network 50. Specifically, for example, the communication interface 33 is a wireless base station compatible with communication standards such as 5G, 4G, and LTE (Long Term Evolution), and a wireless LAN such as IEEE (Institute of Electrical and Electronics Engineers) 802.11 ( It is realized by a circuit that connects to a communication device such as a wireless LAN router that supports the Local Area Network) standard. When the control device 30 directly communicates with the sensing unit 10 or the industrial robot 20, it is realized by a circuit capable of communicating with the sensing unit 10 or the industrial robot 20.

The input / output interface 34 is an interface for connecting to an input device that receives input from a user and a display device 40. The input device is realized by, for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, or the like.

The processor 31 functions as the center of the control device 30. The processor 31 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like. The processor 31 realizes various functions corresponding to the program by reading and executing the program stored in the storage device 32. For example, the processor 31 realizes the functions as the first control unit 31A, the acquisition unit 31B, the detection unit 31C, the position identification unit 31D, and the second control unit 31E by executing the program stored in the storage device 32. ..

The first control unit 31A controls the sensing unit 10. For example, the first control unit 31A transmits a drive signal to the actuator of the sensing unit 10 so as to move the sensing module 11 to an appropriate position.

The first control unit 31A transmits, for example, a drive signal to the sensing module 11 to start sensing by the sensing module 11. The first control unit 31A senses the object 100 by the sensing module 11 and acquires the sensing data.

Specifically, for example, the first control unit 31A drives two sensing modules 11 in FIG. 1 substantially simultaneously to sense the painted surface of the object 100 from different directions. At this time, the first control unit 31A may cause the sensing module 11 to sense while driving the actuator. That is, the first control unit 31A may control the sensing unit 10 so as to acquire the sensing data as a moving image. Further, the first control unit 31A may stop the sensing when driving the actuator, and may stop the actuator when sensing. That is, the first control unit 31A may control the sensing unit 10 so as to acquire sensing data as a plurality of still images.

The acquisition unit 31B acquires the three-dimensional shape of the object 100 based on the sensing data. Specifically, the acquisition unit 31B is three-dimensionally based on the sensing data obtained by sensing the object 100 from a plurality of directions, the positions of the plurality of sensing modules 11 that have acquired the sensing data, and the positions of the object 100. Calculate the shape. In the example shown in FIG. 1, since two sensing modules 11 are arranged, it is possible to acquire sensing data from a plurality of directions without moving the sensing module 11 and the object 100. The acquisition unit 31B stores the acquired information about the three-dimensional shape in the storage device 32.

The detection unit 31C detects an abnormality occurring on the painted surface of the object 100 based on the sensing data. In the present embodiment, the abnormality means, for example, discoloration, cracks, adhesion of dust, uneven coating, and the like. In the following description, a portion where an abnormality occurs on the painted surface of the object 100 is referred to as an abnormal portion.

Specifically, the detection unit 31C detects an abnormality by using, for example, the first trained model. The detection unit 31C inputs the sensing data acquired by the sensing unit 10 to the first trained model, and outputs the abnormal unit included in the sensing data.

The position specifying unit 31D specifies the position of the detected abnormal portion in the three-dimensional space. Specifically, for example, the position specifying portion 31D identifies the three-dimensional position of the abnormal portion in the object 100 by comparing it with the three-dimensional shape of the object 100. The three-dimensional position of the anomalous portion is represented by, for example, coordinates in a predetermined coordinate system with respect to a predetermined position of the object 100. The predetermined coordinate system is, for example, an orthogonal coordinate system or a polar coordinate system. The three-dimensional position of the abnormal portion may be specified, for example, based on a predetermined position in the factory, a predetermined position of the industrial robot 20, or the like. The position specifying unit 31D stores information about the specified three-dimensional position in the storage device 32.

The second control unit 31E controls the industrial robot 20. For example, the second control unit 31E transmits a drive signal to the servomotor of the robot arm 21 so that the end effector 22 reaches a predetermined position. Further, the second control unit 31E transmits a drive signal to the end effector 22, for example, and starts processing by the end effector 22.

Specifically, the second control unit 31E sets a parameter to be input to the robot arm 21 in order to bring the end effector 22 to the three-dimensional position of the abnormal unit specified by the position specifying unit 31D.

At this time, the second control unit 31E inputs the three-dimensional position information of the abnormal unit to the second trained model and outputs the control parameters of the industrial robot 20. The second control unit 31E transmits the parameters output from the second trained model to the robot arm 21. The second control unit 31E operates the end effector 22 when the end effector 22 reaches the specified three-dimensional position.

For example, when the abnormality is coating unevenness and the end effector 22 is an electric polisher, the electric polisher rotates and comes into contact with the abnormal portion, so that the area where the coating unevenness exists is polished and the coating unevenness is eliminated. To.

<Data structure>
Next, the data structure of the information stored in the storage device 32 will be described.
FIG. 3 is a diagram showing a data structure of three-dimensional position information of an abnormal portion. Note that FIG. 3 is an example and does not exclude data not described.

As shown in FIG. 3, each of the records in the position information DB includes an item "abnormal ID", an item "object ID", an item "date", an item "position information", and the like.

The item "abnormal ID" stores an ID for identifying an abnormality that has occurred.

The item "object ID" stores an ID for identifying the object 100 in which the abnormality has occurred.

The item "date" stores the date when the abnormality occurred.

The item "position information" stores the three-dimensional position of the generated abnormality. FIG. 3 shows an example in which a three-dimensional position in a Cartesian coordinate system is stored, but the position information stored in the position information DB is not limited to that in the Cartesian coordinate system.

<Operation>
Next, the processing of the system 1 will be described.
FIG. 4 is a diagram for explaining an example of a flow in which an abnormality on a painted surface of an object 100 is detected by the system 1 and the detected abnormality is eliminated.

As shown in FIG. 4, first, the first control unit 31A of the control device 30 controls the sensing unit 10 (step S301). For example, the first control unit 31A transmits a drive signal to the actuator of the sensing unit 10 and the sensing module 11.

The actuator of the sensing unit 10 moves the sensing module 11 to an appropriate position according to the drive signal from the control device 30. The sensing module 11 starts sensing according to the drive signal from the sensing unit 10 (step S101). The two sensing modules 11 shown in FIG. 1 sense the object 100 from different directions. The sensing unit 10 transmits the sensing data acquired by the sensing of the sensing module 11 to the control device 30 (step S102).

Upon receiving the sensing data from the sensing unit 10, the acquisition unit 31B of the control device 30 acquires the three-dimensional shape of the object 100 (step S302). Specifically, the acquisition unit 31B calculates the three-dimensional shape of the object 100 based on the sensing data acquired by the sensing module 11, the position where the sensing module 11 acquires the sensing data, and the position of the object 100. .. The acquisition unit 31B stores the acquired information about the three-dimensional shape in the storage device 32.

After step S302, the detection unit 31C detects an abnormality on the painted surface of the object 100 (step S303). Specifically, the detection unit 31C inputs the sensing data acquired by the sensing module 11 into the first trained model, and outputs the abnormal unit included in the sensing data.

When the detection unit 31C detects the abnormal portion, the position specifying unit 31D identifies the position of the abnormal portion (step S304). For example, the position specifying unit 31D identifies the three-dimensional position of the abnormal portion in the object 100 by comparing it with the three-dimensional shape of the object 100. At this time, the position specifying unit 31D represents, for example, the three-dimensional position of the abnormal portion by the coordinates in the orthogonal coordinate system with respect to the predetermined position of the object 100.

When the position of the abnormal portion is specified by the position specifying unit 31D, the second control unit 31E controls the industrial robot 20 (step S305). For example, the second control unit 31E inputs the three-dimensional position information of the abnormal portion to the second trained model, and outputs the control parameters for reaching the end effector 22 of the industrial robot 20 to the abnormal portion. The second control unit 31E transmits a drive signal for instructing the control parameters to the robot arm 21 of the industrial robot 20. Further, the second control unit 31E transmits a drive signal for starting the processing of the end effector 22 to the end effector 22.

The robot arm 21 causes the end effector 22 to reach a position according to the drive signal from the control device 30 (step S201). The end effector 22 starts processing at the reached position according to the drive signal from the sensing unit 10 (step S202).

FIG. 5 is a schematic view showing an example of an abnormal portion on the painted surface of the object 100. FIG. 6 is a schematic diagram for explaining the operation of the industrial robot 20. When the abnormal portion B is detected on the painted surface of the object 100 as shown in FIG. 5, the position specifying portion 31D specifies the three-dimensional position of the abnormal portion B. The second control unit 31E calculates the control parameters of the robot arm 21 based on the specified three-dimensional position, and controls the robot arm 21 based on the calculated control parameters, so that the end effector is as shown in FIG. 22 is made to reach the abnormal portion B. When the second control unit 31E reaches the abnormal portion B, the second control unit 31E operates the electric polisher, which is the end effector 22, and polishes the abnormal portion B. As a result, uneven coating is eliminated by polishing with the electric polisher which is the end effector 22.

When the processing to the abnormal portion B is completed by the end effector 22, the control device 30 performs the processing of steps S301 and S303 shown in FIG. 4 to confirm whether or not the abnormality on the painted surface of the object 100 has been resolved. If the abnormality is not resolved, the control device 30 performs the processes of steps S304 and S305 shown in FIG. 4 to eliminate the abnormality. The processes of steps S301 and S303 to S305 may be repeated, for example, until appropriate repair work is confirmed.

Not limited to the above processing, the steps of each processing can be arbitrarily changed as long as the flow of each processing does not contradict.

For example, in the example shown in FIG. 4, the case where the acquisition of the three-dimensional shape of the object 100 in step S302 and the detection of the abnormality in the object 100 in step S303 are performed in a series of flows has been described. However, the control device 30 may perform the acquisition of the three-dimensional shape of the object 100 in step S302 and the detection of the abnormality in the object 100 in step S303 in separate flows.

As described above, in the present embodiment, the control device 30 senses the appearance of the object 100 by the sensing module 11 and acquires the three-dimensional shape of the object 100. Further, the control device 30 detects an abnormality based on the sensing data and identifies the three-dimensional position of the detected abnormality. Then, the control device 30 is designed to repair the abnormality that has occurred in the appearance of the object 100 by reaching the end effector 22 at the position where the abnormality has occurred and driving the end effector 22. As a result, the control device 30 can automatically detect the abnormality in the object 100 and automatically repair the detected abnormality.

Therefore, according to the system 1 according to the present embodiment, it is possible to reduce the burden on the worker in the visual inspection.

Further, in the present embodiment, the acquisition unit 31B acquires the three-dimensional shape of the object 100 based on the sensing data obtained by sensing the object 100 from a plurality of directions. As a result, the control device 30 can accurately acquire the three-dimensional shape of the object 100.

Further, in the present embodiment, the detection unit 31C inputs the sensing data for the object and detects the abnormality by using the trained model trained to output the presence or absence of the abnormality. This makes it possible to detect abnormalities with high accuracy regardless of the operator.

Further, in the present embodiment, the second control unit 31E uses a trained model trained to input the abnormal three-dimensional position and output the position of the end effector 22 with respect to the three-dimensional position, and uses the robot arm. I am trying to move 21. This makes it possible to bring the end effector 22 to the position where the abnormality has occurred with high accuracy regardless of the operator.

Further, in the present embodiment, after the end effector 22 is operated, it is confirmed whether or not appropriate repair work has been performed, so that it is possible to eliminate the generated abnormality with high accuracy.

<Modification 1>
Next, a modification 1 of the system 1 will be described. In the above embodiment, the case where two sensing modules 11 are arranged has been described. In the first modification, a case where one sensing module 11 is provided will be described.

FIG. 7 is a diagram showing the configuration of the system 1 according to the modified example 1. Note that FIG. 7 omits the illustration of the control device 30, the display device 40, and the network 50 in the system 1. The system 1 shown in FIG. 7 includes one sensing unit 10. Further, the object 100 is placed on, for example, a rotatable stage 60.

In the first modification, the first control unit 31A controls the sensing unit 10 and the stage 60. Specifically, for example, the first control unit 31A controls the actuator of the sensing unit 10 or rotates the stage 60 so as to move the sensing module 11 relative to the object 100.

The first control unit 31A causes the sensing module 11 to sense the painted surface of the object 100 before and after moving the sensing module 11 relative to the object 100. At this time, since the first control unit 31A acquires the three-dimensional shape of the object 100, the first control unit 31A has a predetermined angle with respect to the object 100 between the first sensing and the second sensing. I try to open the interval only. As a result, sensing data in different directions regarding the painted surface of the object 100 are acquired.

The acquisition unit 31B obtains a three-dimensional shape of the object 100 based on the sensing data obtained by sensing the object 100 from a plurality of directions, the position of the sensing module 11 with respect to the object 100 when the sensing data is acquired, and the position of the object 100. calculate.

The control device 30 in the first modification carries out the operations of steps S301 to S305 in the same manner as the operations shown in FIG. However, in step S301, the first control unit 31A may rotate the stage 60 instead of controlling the actuator of the sensing unit 10. As a result, in step S101, sensing data from two directions will be acquired.

As described above, the control device 30 according to the modification 1 uses one sensing module 11 and moves the sensing module 11 with respect to the object 100 to obtain sensing data from different directions on the painted surface of the object 100. get. Then, the control device 30 detects an abnormality on the painted surface of the object 100 based on the acquired sensing data, and eliminates the detected abnormality by using the industrial robot 20. As a result, even when there is only one sensing module 11, it is possible to accurately detect the abnormality of the object 100 and eliminate the abnormality.

<Modification 2>
Next, a modification 2 of the system 1 will be described. In the above embodiment, the case of inspecting the object 100 without using special lighting has been described as an example. In the second modification, a case of using an illumination that emits light having a predetermined wavelength will be described.

FIG. 8 is a diagram showing the configuration of the system 1 according to the modified example 2. Note that FIG. 8 omits the illustration of the control device 30, the display device 40, and the network 50 in the system 1.

The system 1 shown in FIG. 8 includes an illumination 70 which is a light source. The illumination 70 generates light having a predetermined wavelength. The light emitted from the illumination 70 is, for example, white light including light of all wavelengths. The illumination 70 generates light with an amount of light that is unlikely to cause diffuse reflection of the reflected light reflected on the surface of the object 100. The light emitted by the illumination 70 irradiates the object 100.

In the second modification, the sensing module 11 senses light having a wavelength emitted from the illumination 70. When the light emitted from the illumination 70 is white light, the sensing module 11 senses visible light, for example.

The control device 30 in the second modification carries out the operations of steps S301 to S305 in the same manner as the operations shown in FIG.

As described above, in the system 1 according to the modification 2, the illumination 70 irradiates light having a predetermined wavelength, and the sensing module 11 senses the reflected light from the object 100. Then, the control device 30 detects an abnormality on the painted surface of the object 100 based on the acquired sensing data, and eliminates the detected abnormality by using the industrial robot 20. This makes it possible to accurately detect abnormalities such as slight coating unevenness, for which differences are difficult to confirm by sensing in natural light.

In the second modification, the illumination 70 may be moved relative to the object 100 when irradiating the light. At this time, the sensing module 11 may or may not be moved relative to the object 100. By moving the illumination 70 relative to the object 100, the reflection of light changes, so that the accuracy of detecting an abnormality is further improved.

Further, in the second modification, the illumination 70 may switch the wavelength of the irradiated light when irradiating the object 100 with the light. At this time, the sensing unit 10 may include a plurality of sensing modules 11 corresponding to the wavelength of light.

As a result, the sensing module 11 can also be switched according to the wavelength of the emitted light, so that various sensing can be performed.

<Modification 3>
Next, a modification 3 of the system 1 will be described. In the third modification, the case where the three-dimensional position of the identified abnormal portion is shared by the terminal owned by the user will be described.

FIG. 9 is a diagram showing the configuration of the system 1 according to the modified example 3. The system 1 shown in FIG. 9 has a terminal device 80. The terminal device 80 is realized by a portable computer such as a head-mounted display, a smartphone, or a tablet terminal. The terminal device 80 has a function of expanding a perceptible real environment by superimposing a desired image on a real image or a landscape. The terminal device 80 as a head-mounted display is generally referred to as an AR (Augmented Reality) glass. The terminal device 80 has a communication function and a display function of superimposing and displaying the information acquired by the communication function on an actual image or a landscape.

The control device 30 transmits information about the three-dimensional position of the abnormal portion of the object 100 stored in the storage device 32 to the terminal device 80 via the network 50. That is, the control device 30 transmits information regarding the three-dimensional position of the abnormality identified by the processing of steps S301 to S304 shown in FIG. 4 to the terminal device 80.

The terminal device 80 determines whether or not an abnormal portion is included in the viewing direction via the terminal device 80 based on the acquired information on the three-dimensional position. When the abnormal portion is caught in the viewing direction, the terminal device 80 superimposes and displays information indicating that an abnormality has occurred at a position corresponding to the abnormal portion on the display. Even in a situation where the abnormal part cannot be seen directly, that is, even when the abnormal part is hidden behind a predetermined structure, if the direction is grasped, information indicating the occurrence of the abnormality is superimposed. The information indicating that an abnormality has occurred includes information indicating the position of the abnormal part, information indicating the date and time when the abnormality was detected, information indicating the degree of the abnormality, information indicating the line where the abnormality occurred, and an object in which the abnormality occurred. Includes information that represents.

Specifically, for example, when the terminal device 80 is an AR glass, the terminal device 80 determines whether or not an abnormal portion is included in the viewing direction via the lens. When the abnormal portion is captured in the viewing direction, the terminal device 80 superimposes and displays information indicating that an abnormality has occurred at a position corresponding to the abnormal portion on the lens. As a result, when the user of the terminal device 80 visually recognizes the object 100 through the terminal device 80, an image showing the indication to that effect can be visually recognized as augmented reality in the portion where the abnormality of the object 100 occurs. It will be like.

As described above, the control device 30 according to the modification 3 causes the terminal device 80 to display the information regarding the three-dimensional position of the identified abnormal portion as augmented reality. As a result, the position of the abnormal portion in the object 100 can be immediately grasped, and it becomes possible to efficiently confirm the degree of abnormality, examine the repair work, estimate the man-hours, and the like.

<Modification example 4>
Next, a modification 4 of the system 1 will be described. In the fourth modification, the case where the actuator of the sensing unit 10 is the robot arm 12 will be described.

FIG. 10 is a diagram showing the configuration of the system 1 according to the modified example 4. The system 1 shown in FIG. 10 has two sensing units 10a in which the sensing module 11 is arranged at the tip of the robot arm 12. The number of sensing units 10a included in the system 1 may be one or three or more.

In the modified example 4, the first control unit 31A controls the sensing unit 10a. Specifically, for example, the first control unit 31A controls the servomotor of the robot arm 12 so as to move the sensing module 11 relative to the object 100. More specifically, the first control unit 31A controls the robot arm 12 so that the sensing module 11 captures the motion of the object 100 as a regular motion, for example, a constant velocity linear motion, a constant acceleration motion, or the like. do.

The stage 61 on which the object 100 is arranged may transport the object 100 in a predetermined direction while repeating stopping and moving. The first control unit 31A controls and senses the robot arm 12 so that the sensing module 11 can capture the motion of the object 100 as a regular motion even when the stage 61 repeats stopping and moving. Move the module 11.

The control device 30 in the modified example 4 carries out the operations of steps S301 to S304 in the same manner as the operations shown in FIG.

As described above, the control device 30 according to the modified example 4 senses the painted surface of the object 100 by the sensing module 11 attached to the robot arm 12. Then, the control device 30 detects an abnormality on the painted surface of the object 100 based on the acquired sensing data. By attaching the sensing module 11 to the robot arm 12, the position of the sensing module 11 can be precisely controlled. As a result, the accuracy of detecting the abnormality of the object 100 is improved.

Further, in the modification 4, the sensing module 11 controls the movement of the robot arm 12 so that the movement of the object 100 can be regarded as a regular movement, for example, a constant velocity linear motion, a constant acceleration motion, or the like. There is. As a result, the abnormality of the object 100 also moves regularly, so that erroneous detection can be suppressed and the abnormality detection accuracy can be improved.

<Other variants>
Next, other modification examples will be described.

In the above embodiment, the case where the control device 30 includes the detection unit 31C and the detection unit 31C detects an abnormality of the object 100 based on the sensing data has been described as an example. However, the abnormality of the object 100 may be detected by the sensing unit 10. The sensing unit 10 stores a trained model for detecting an abnormality on the painted surface of the object 100. The trained model is trained to output an abnormality on the painted surface with respect to the input sensing data. At this time, for the learning data, for example, a plurality of sensing data about the object are used as input data, and a judgment about an abnormality that can be included in the input data is used as correct output data.

When the sensing data is acquired by the sensing module 11, the sensing unit 10 inputs the acquired sensing data to the trained model and outputs a determination of an abnormality included in the sensing data. The sensing unit 10 transmits the sensing data and the determination of abnormality to the control device 30.

In the above embodiment, for example, a case where a device realized by using electromagnetic waves of a preset band is adopted for the sensing module 11 has been described, but electromagnetic waves of different bands are used for the sensing module 11. A plurality of types of devices realized by the above may be adopted. At this time, the detection unit 31C of the control device 30 detects the abnormality of the object 100 based on the sensing data using the electromagnetic waves of the respective bands of the sensing module 11, for example. This makes it possible to improve the accuracy of detecting abnormalities.

Although the preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to such specific embodiments, and the present disclosure includes the inventions described in the claims and the equivalent scope thereof. Is done.
Further, the configurations of the devices described in the above-described embodiments and modifications can be appropriately combined as long as there is no technical contradiction.

<Additional Notes>
The matters described in each of the above embodiments will be added below.

(Appendix 1)
A program for causing a computer having a processor 31 and a memory to execute the program.
The step (S302) of acquiring the three-dimensional shape of an object based on the sensing data obtained by sensing the object using electromagnetic waves, and
Step (S303) to detect anomalies in an object based on sensing data,
Step (S304) to identify the three-dimensional position of the detected anomaly,
A step (S305) of moving the robot arm 21 equipped with a device capable of resolving an abnormality so that the device 22 reaches a specified three-dimensional position.
A program that executes the step (S202) of operating the device when the device reaches a three-dimensional position.

(Appendix 2)
The visual inspection program according to (Appendix 1), wherein in the step (S302) of acquiring a three-dimensional shape of an object, the three-dimensional shape is acquired based on sensing data obtained by sensing the object from a plurality of directions.

(Appendix 3)
The program according to (Appendix 2), wherein the sensing means 11 is moved so as to acquire the sensing data obtained by sensing the object from a plurality of directions, and the processor is made to execute the step (S301) of causing the sensing means to sense the object.

(Appendix 4)
Described in (Appendix 2) or (Appendix 3), wherein the processor executes the step (S301) of moving the object and causing the sensing means to sense the object so as to acquire sensing data obtained by sensing the object from a plurality of directions. Program.

(Appendix 5)
The program according to (Appendix 2), which causes a processor to execute a step (S301) of causing a plurality of sensing means to sense an object so as to acquire sensing data obtained by sensing the object from a plurality of directions.

(Appendix 6)
The program according to any one of (Appendix 1) to (Appendix 5), which causes a processor to execute a step (S101) of sensing an object by a sensing means using electromagnetic waves in a plurality of bands.

(Appendix 7)
In the step of detecting anomalies (S303), anomalies are detected from (Appendix 1) to (Appendix 6) by using a trained model trained to input sensing data for an object and output the presence or absence of anomalies. The program described in any of.

(Appendix 8)
In the step of moving the robot arm (S305), the robot arm is moved by using a trained model trained to input the abnormal 3D position and output the position of the device with respect to the 3D position (Appendix). The program described in any of 1) to (Appendix 7).

(Appendix 9)
A program to be executed by a computer equipped with a processor and a memory, and the program is a program to be executed by the processor.
A step of moving the sensing means 11 with respect to the object and causing the sensing means to sense the object using electromagnetic waves so that the motion of the object is regarded as a regular motion.
The step (S303) of detecting an abnormality of an object based on the sensing data obtained by sensing, and
Step (S304) to identify the three-dimensional position of the detected anomaly,
A program that runs.

(Appendix 10)
The step of irradiating an object with light,
The program according to any one of (Appendix 1) to (Appendix 9), which causes a processor to execute a step of sensing an object by a sensing means for detecting light.

(Appendix 11)
The program according to (Appendix 10), wherein in the step of irradiating light, the light source 70 of light is moved relative to an object.

(Appendix 12)
The program according to (Appendix 10) or (Appendix 11), wherein in the step of irradiating light, light of a plurality of wavelengths is switched and irradiated.

(Appendix 13)
The program according to any one of (Appendix 1) to (Appendix 12), which causes a processor to execute a step of transmitting the specified three-dimensional position to the terminal device 80.

(Appendix 14)
A method performed by a computer equipped with a processor and memory, wherein the processor
The step (S302) of acquiring the three-dimensional shape of an object based on the sensing data obtained by sensing the object using electromagnetic waves, and
The step (S303) of detecting an abnormality of an object based on the sensing data,
Step (S304) to identify the three-dimensional position of the detected anomaly,
A step (S305) of moving a robot arm equipped with a device capable of resolving an abnormality so that the device reaches a specified three-dimensional position.
When the device reaches the three-dimensional position, the step (S202) of operating the device and
How to do it.

(Appendix 15)
A means for acquiring a three-dimensional shape of an object based on sensing data obtained by sensing the object using electromagnetic waves, and
A means of detecting anomalies in an object based on sensing data,
A means to identify the three-dimensional position of the detected anomaly,
A robot arm equipped with a device capable of resolving an abnormality is moved so that the device reaches a specified three-dimensional position, and when the device reaches the three-dimensional position, a means for operating the device is used.
A system equipped with.

(Appendix 16)
A method performed by a computer equipped with a processor and memory, wherein the processor
A step of moving a sensing means to an object and causing the sensing means to sense the object using electromagnetic waves so that the motion of the object is regarded as a regular motion.
The step (S303) of detecting an abnormality of an object based on the sensing data obtained by sensing, and
Step (S304) to identify the three-dimensional position of the detected anomaly,
How to get it done.

(Appendix 17)
A means of moving a sensing means with respect to an object and causing the sensing means to sense the object using electromagnetic waves so that the motion of the object is regarded as a regular motion.
A means to detect anomalies in an object based on the sensing data obtained by sensing,
A means to identify the three-dimensional position of the detected anomaly,
A system equipped with.

1 ... System 10, 10a ... Sensing unit 100 ... Object 11 ... Sensing module 12 ... Robot arm 20 ... Industrial robot 21 ... Robot arm 22 ... End effector 30 ... Control device 31 ... Processor 31A ... First control unit 31B ... Acquisition unit 31C ... Detection unit 31D ... Position identification unit 31E ... Second control unit 32 ... Storage device 33 ... Communication interface 34 ... Input / output interface 40 ... Display device 50 ... Network 60 ... Stage 61 ... Stage 70 ... Lighting 80 ... Terminal device

Claims (17)

A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
A step of acquiring the three-dimensional shape of the object based on the sensing data obtained by sensing the object using electromagnetic waves, and
A step of detecting an abnormality of the object based on the sensing data,
The step of identifying the three-dimensional position of the detected abnormality and
A step of moving a robot arm equipped with a device capable of resolving the abnormality so that the device reaches the specified three-dimensional position.
A program that executes a step of operating the device when the device reaches the three-dimensional position. The program according to claim 1, wherein in the step of acquiring the three-dimensional shape of the object, the three-dimensional shape is acquired based on the sensing data obtained by sensing the object from a plurality of directions. The program according to claim 2, wherein the sensing means is moved so as to acquire sensing data obtained by sensing the object from a plurality of directions, and the processor is made to perform a step of causing the sensing means to sense the object. The program according to claim 2 or 3, wherein the processor executes a step of moving the object and causing the sensing means to sense the object so as to acquire sensing data obtained by sensing the object from a plurality of directions. The program according to claim 2, wherein the processor executes a step of causing a plurality of sensing means to sense the object so as to acquire sensing data obtained by sensing the object from a plurality of directions. The program according to any one of claims 1 to 5, wherein the processor executes a step of causing a sensing means using electromagnetic waves of a plurality of bands to sense the object. In the step of detecting the abnormality, any one of claims 1 to 6 for detecting the abnormality by using a trained model trained to input the sensing data for the object and output the presence or absence of the abnormality. The program described in the section. In the step of moving the robot arm, the robot arm is moved by using a trained model trained to input the abnormal three-dimensional position and output the position of the device with respect to the three-dimensional position. The program according to any one of claims 1 to 7. A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
A step of moving the sensing means with respect to the object and causing the sensing means to sense the object using electromagnetic waves so that the motion of the object is regarded as a regular motion.
A step of detecting an abnormality of the object based on the sensing data obtained by the sensing, and
The step of identifying the three-dimensional position of the detected abnormality and
A program that runs. The step of irradiating the object with light and
The program according to any one of claims 1 to 9, wherein the processor performs a step of causing the sensing means for detecting light to sense the object. The program of claim 10, wherein in the step of irradiating the light, the light source of the light is moved relative to the object. The program according to claim 10 or 11, wherein in the step of irradiating the light, light of a plurality of wavelengths is switched and irradiated. The program according to any one of claims 1 to 12, wherein the processor executes a step of transmitting the specified three-dimensional position to the terminal device. A method performed by a computer comprising a processor and memory, wherein the processor is:
A step of acquiring the three-dimensional shape of the object based on the sensing data obtained by sensing the object using electromagnetic waves, and
A step of detecting an abnormality of the object based on the sensing data,
The step of identifying the three-dimensional position of the detected abnormality and
A step of moving a robot arm equipped with a device capable of resolving the abnormality so that the device reaches the specified three-dimensional position.
When the device reaches the three-dimensional position, the step of operating the device and
How to do it. A means for acquiring a three-dimensional shape of an object based on sensing data obtained by sensing the object using electromagnetic waves, and
A means for detecting an abnormality in the object based on the sensing data,
A means for identifying the three-dimensional position of the detected abnormality and
A robot arm equipped with a device capable of resolving the abnormality is moved so that the device reaches the specified three-dimensional position, and when the device reaches the three-dimensional position, the device is operated. Means and
A system equipped with. A method performed by a computer comprising a processor and memory, wherein the processor is:
A step of moving the sensing means with respect to the object and causing the sensing means to sense the object using electromagnetic waves so that the motion of the object is regarded as a regular motion.
A step of detecting an abnormality of the object based on the sensing data obtained by the sensing, and
The step of identifying the three-dimensional position of the detected abnormality and
How to get it done. A means for moving a sensing means with respect to the object and causing the sensing means to sense the object using electromagnetic waves so that the motion of the object is regarded as a regular motion.
A means for detecting an abnormality of the object based on the sensing data obtained by the sensing, and
A means for identifying the three-dimensional position of the detected abnormality and
A system equipped with.

Images