JP2023056623A - Remote inspection device and remote inspection method - Google Patents

Abstract

To facilitate inspection of an inspection object by remote operation.SOLUTION: A light source relative position and attitude calculation unit calculates light source relative position and attitude information on the basis of operation machine position and attitude information whose data format is aligned. A photographing apparatus relative position and attitude calculation unit calculates photographing apparatus relative position and attitude information on the basis of, view point position and attitude information and the operation machine position and attitude information whose data formats are aligned. A light source position and attitude control unit controls a position and an attitude of a light source on the basis of, the light source relative position and attitude information. A photographing apparatus position and attitude control unit controls a position and attitude of the photographing apparatus on the basis of, the photographing apparatus relative position and attitude information.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a remote inspection device and a remote inspection method.

In the virtual three-dimensional space, the position and orientation of the object in the virtual three-dimensional space are changed by changing the position and orientation of the controller held by the operator based on the position and orientation information of the head-mounted display (HMD). A thing is known (for example, patent document 1).

Further, there is known an appearance inspection apparatus capable of changing the way in which inspection light is applied to an inspection object (also referred to as an inspection object) by attaching an inspection light source to an articulated robot (for example, Patent Literature 2).

JP 2018-061667 A JP 2019-045330 A

For example, when a robot hand of an articulated robot is used to grip and move an object to be inspected, it is necessary to manufacture a robot hand and design a gripping method suitable for the object to be inspected. Therefore, it is not necessarily easy to freely change the position and orientation of the inspection object in real space.

In addition, the configuration of an inspection apparatus that performs inspection while changing the position and orientation of an inspection light source to change the way in which light is applied to an inspection object is different from the configuration of a human body. For this reason, the operator needs to understand the configuration and operation method of the relatively complicated inspection apparatus, and inspection cannot be easily performed.

In this way, it is easy to freely change the position and orientation of the inspection object in the virtual three-dimensional space, but it is not easy to freely change the position and orientation of the inspection object in the real space. There is a problem that inspection cannot be easily performed by remote control.

Accordingly, the present disclosure has been made in view of the above-described problems, and aims to provide a technology that enables easy inspection of an inspection target by remote control.

A remote inspection apparatus according to the present disclosure includes a viewpoint position/posture acquisition unit that acquires the position/posture of an operator's viewpoint as viewpoint position/posture information, and acquires the position/posture of a manipulator held by the operator as manipulator position/posture information. an operation information management unit that aligns the data format of the viewpoint position/posture information and the data format of the actuator position/posture information; a light source relative position/attitude calculation unit that calculates, as light source relative position/attitude information, a control amount of the position/attitude of the light source in a coordinate system related to the inspection object; an imaging device relative position/orientation calculation unit that calculates, based on the information, a control amount of the position and orientation of the imaging device in the coordinate system with respect to the inspection object as imaging device relative position/orientation information; and based on the light source relative position/orientation information , a light source position/attitude control unit for controlling the position/attitude of the light source; and an imaging equipment position/attitude control unit for controlling the position/attitude of the imaging equipment based on the imaging equipment relative position/attitude information.

According to the present disclosure, light source relative position/attitude information for controlling the position/attitude of the light source is calculated based on the manipulator position/attitude information, and based on the viewpoint position/attitude information and the manipulator position/attitude information, Calculation of image capture device relative position and orientation information for controlling the position and orientation. According to such a configuration, it is possible to easily inspect the object to be inspected by remote control.

1 is a block diagram showing the configuration of a remote inspection device according to Embodiment 1; FIG. 2 is a block diagram showing the functional configuration of the remote inspection device according to Embodiment 1; FIG. FIG. 9 is a block diagram showing the functional configuration of a remote inspection device according to Embodiment 2; FIG. 11 is a block diagram showing the functional configuration of a remote inspection device according to Embodiment 3; FIG. 11 is a block diagram showing the functional configuration of a remote inspection device according to Embodiment 4; FIG. 11 is a block diagram showing a hardware configuration of a remote inspection device according to another modified example; FIG. 11 is a block diagram showing a hardware configuration of a remote inspection device according to another modified example;

<Embodiment 1>
As conventional inspection devices for production sites, there have been proposed devices that capture images with a camera fixed on the production line, or devices that have cameras attached to the hands of articulated robots. The main purpose of these inspection devices is to inspect manufacturing defects such as breakage of the inspection target, and it is common to use a shadowless ring illumination to capture an image of the inspection target without a shadow. target.

On the other hand, it may be desirable to inspect the performance of the inspection object by appropriately adding shadows while changing the way the light is applied. For example, when inspecting a single item, a prototype, or a design object, it is desirable to be able to check the object to be inspected from various angles while changing the way the light is applied. Conventionally, such confirmation is performed by an inspector directly inspecting the actual object to be inspected. In addition to checking coating and shadows, inspections involving changes in the way light is applied and viewpoints are useful in various inspections, such as checking the processing quality in metal scraping, for example.

In recent years, the development of digital design technology such as CAD (Computer Aided Design) has made it possible to give work instructions to complex shaped objects, and the development of processing technology based on digital data such as 3D printers has expanded the range of processing by machines. ing. In addition, with the development of ICT (Information and Communication Technology), the scope of application of work by remote control from a remote location is also expanding. By utilizing these technologies, it is expected that the business model that outsources the manufacturing of products with a strong design aspect such as crafts and clothing and conducts inspections from a remote location of the manufacturing site will expand. In such a business, it is difficult for operators to understand and learn the configuration and operation method of inspection machines, which are relatively complicated for each subcontractor.

Therefore, according to the remote inspection apparatus according to the first embodiment, the operator can perform inspection by intuitive remote operation from a remote location as if directly grasping and visually observing the inspection target under a light source. The inspection object can be inspected while changing the way the light source hits the object and the viewpoint.

An outline of the first embodiment will be described. Based on operation input values for changing the operator's viewpoint direction and the position and orientation of the inspection object, the relative position of the light source and the imaging device with respect to the inspection object in the coordinate system related to the inspection object. A control amount including an amount of movement and an amount of rotation is calculated. Then, based on the control amount, the movement and rotation of the light source and the photographing device are appropriately controlled. This enables the operator to obtain a desired inspection image by an intuitive operation method without moving the actual object to be inspected.

A remote inspection apparatus according to the first embodiment will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the remote inspection device according to the first embodiment, and FIG. 2 is a block diagram showing the functional configuration of the remote inspection device according to the first embodiment.

First, using the block diagram of FIG. 1, the configuration of the remote inspection apparatus according to the first embodiment will be described. In FIG. 1, an articulated robot including a light source robot 55 and an imaging robot 56 and an inspection target 54 are provided at an inspection location, and an operator 51 remotely operates the articulated robot from an operation location away from the inspection location. An example of inspecting an inspection object 54 is shown. Illustrations and descriptions of devices that are not directly related to the description of the features of the first embodiment are omitted.

The equipment at the operation site will be explained. An operator 51 remotely controls the equipment at the inspection site using an HMD (Head Mounted Display) 52 and an operating device 53 , and visually inspects using the display on the HMD 52 .

The HMD 52 is worn on the head of the operator 51 and used. The HMD 52 acquires position and orientation information of the HMD 52 itself and displays an image to the operator 51 . A technique such as eye tracking that captures the movement of the eyeballs of the operator 51 is known. explain.

The manipulator 53 is held and used by the operator 51 . The manipulator 53 acquires the position and orientation information of the manipulator 53 itself, similarly to the HMD 52 . As will be described later, the HMD 52 and the manipulator 53 are calibrated in a common coordinate system so that the relative position and orientation between them can be calculated.

Describe the equipment at the inspection location. The inspection object 54 is an object to be visually inspected by the operator 51 by remote control. The inspection object 54 is placed at the inspection location by a worker (not shown) at the inspection location. The light source robot 55 is an articulated robot having a light source 55a attached to its hand, and can be controlled to freely change the position and orientation of the light source 55a. The photographing robot 56 is an articulated robot having a photographing device 56a such as a camera attached to its hand, and can be controlled to freely change the position and orientation of the photographing device 56a.

Components other than the above shown in FIG. 1 will be described. The operation control device 57 integrally manages the equipment at the operation location. For example, the operation control device 57 includes individual control devices for devices such as the HMD 52 and the manipulator 53 provided at the operation location, and a device for integrally managing them. A relative position/orientation calculation device 58 calculates information for movement of the light source robot 55 and the photographing robot 56 . The inspection facility control device 59 integrally manages the equipment at the inspection location. For example, the inspection facility control device 59 includes individual control devices for equipment provided at the inspection location such as the light source robot 55 and the photographing robot 56, and a device for integrally managing them. The image transmission device 60 is a device for transmitting an image acquired by the photographing device 56 a of the photographing robot 56 to the HMD 52 worn by the operator 51 . HMD 52 displays the video transmitted by video transmission device 60 .

Next, using the block diagram of FIG. 2, the functional configuration of the remote inspection device according to the first embodiment will be described. Illustrations and descriptions of functions that are not directly related to the description of the features of the first embodiment are omitted. In the following, an example in which each functional unit in FIG. 2 is installed in the device in FIG. 1 will be described, but the present invention is not limited to this, and may be installed in a device different from the device in FIG.

The viewpoint position/posture acquisition unit 1 acquires the viewpoint position/posture of the operator 51 as viewpoint position/posture information. The viewpoint position/posture acquisition unit 1 acquires, for example, the position/posture of the HMD 52 worn by the operator 51 as viewpoint position/posture information. The viewpoint position/posture acquisition unit 1 transmits the acquired viewpoint position/posture information to the operation information management unit 3 . The details of the position and orientation information such as the viewpoint position and orientation information will be described later together with the explanation of the operation information management section 3 .

As described above, in Embodiment 1, the position and orientation of the viewpoint of the operator 51 are handled as the position and orientation of the HMD 52 . Acquisition of the position and orientation of the HMD 52 may be performed based on difference information from a specific timing using an acceleration sensor or the like built into the HMD 52, or may be performed based on tracking by a camera or the like separately attached to the operation location. It is sufficient if at least information about the position and orientation of the HMD 52 can be acquired. This viewpoint position/orientation acquisition unit 1 is mounted on the HMD 52 in FIG.

The manipulator position/orientation acquisition unit 2 acquires the position and orientation of the manipulator 53 held by the operator 51 as manipulator position/orientation information. The operator 51 changes the position and orientation of the manipulator 53 with the intention of changing the position and orientation of the inspection target 54 . When the position and orientation of the operating device 53 are changed, as will be described later, the positions and orientations of the light source 55a and the photographing device 56a are changed without changing the position and orientation of the actual inspection object 54. The position and orientation of the inspection object 54 to be inspected is changed. The manipulator position/posture acquisition unit 2 transmits the acquired manipulator position/posture information to the operation information management unit 3 . Details of the position and orientation information such as the manipulator position and orientation information will be described later together with the explanation of the operation information management unit 3 .

The position and orientation of the manipulator 53 may be acquired based on difference information from a specific timing using an acceleration sensor or the like built into the manipulator 53, or may be obtained using a camera or the like separately attached to the operation location. It may be performed based on tracking, and at least information about the position and orientation of the manipulator 53 may be acquired. This manipulator position/posture acquisition unit 2 is mounted on the manipulator 53 of FIG.

The operation information management unit 3 calibrates the viewpoint position/posture information and the manipulator position/posture information transmitted from the viewpoint position/posture acquisition unit 1 and the manipulator position/posture acquisition unit 2 to a specific coordinate system. The data format of attitude information and the data format of manipulator position/attitude information are aligned. This facilitates handling of the viewpoint position and orientation information and the manipulator position and orientation information in the subsequent processing. By unifying the data format, for example, not only the coordinate system and expression method of the position and orientation information, but also the sampling period and time correction may be unified.

For example, if the transmitted position/orientation information is only the data of the acceleration sensor, the operation information management unit 3 converts the data into a movement amount by integrating the data, and converts the movement amount into a movement amount that has been calibrated in advance. It may be combined with work data and converted into position and orientation information in a specific coordinate system. Further, for example, when the coordinate system of the viewpoint position/posture information and the coordinate system of the controller position/posture information are different, the operation information management unit 3 converts these coordinate systems into position/posture information in a specific coordinate system. By doing so, the coordinate system may be the same and the expression method may be the same. Further, for example, the operation information management unit 3 may interpolate each piece of position and orientation information in order to match the sampling period.

The position and orientation information output from the operation information management unit 3 configured as described above is expressed in one specific coordinate system, and is output in a format with uniform sampling periods and times. Since various methods are already known for calibrating the position and orientation information of multiple devices, detailed description thereof will be omitted. The light source relative position/attitude calculation unit 4 receives the controller position/attitude information in the same data format, and the photographing equipment relative position/attitude calculation unit 5 receives the viewpoint position/attitude information and the controller position/attitude information in the same data format. sent. This operation information management unit 3 is mounted on the operation control device 57 in FIG.

The light source relative position/attitude calculation unit 4 calculates the control amount of the position/attitude of the light source 55a in the coordinate system regarding the inspection target 54 as the light source relative position/attitude information based on the manipulator position/attitude information with the same data format. In the first embodiment, the coordinate system related to the inspection target 54 is a coordinate system fixed to the inspection target 54 when the position and orientation of the inspection target 54 are temporarily controlled based on the position and orientation information. The origin of the coordinate system fixed to the inspection object 54 may be set anywhere inside or outside the inspection object 54 . For example, the origin may be set near the center of the shape of the inspection object 54, may be set based on the design information of the inspection object 54, or may be manually set by the operator. The direction of the coordinate axis of the coordinate system fixed to the inspection object 54 may also be set based on the positional relationship between the imaging device 56a and the inspection object 54 at the inspection location when the operation is started. It may be set based on the design information of 54, or may be manually set by the operator. The amount of control of the position and orientation of the light source 55a includes the amount of movement and the amount of rotation of the light source 55a.

The light source relative position/posture calculation unit 4 according to the first embodiment obtains the control amount of the position/posture of the inspection object 54 when it is tentatively controlled based on the manipulator position/posture information, and performs control in the direction opposite to the control amount. Quantities are calculated as light source relative position and orientation information. The light source relative position/attitude calculator 4 transmits the calculated light source relative position/attitude information to the light source position/attitude controller 6 . The light source relative position/orientation calculator 4 is mounted on the relative position/orientation calculator 58 of FIG.

The imaging device relative position/orientation calculation unit 5 calculates the control amount of the position and orientation of the imaging device 56a in the coordinate system regarding the inspection target 54 based on the viewpoint position/orientation information and the manipulator position/orientation information in which the data formats are aligned. Calculate as relative position and orientation information. In Embodiment 1, the coordinate system related to the inspection target 54 is a coordinate system fixed to the inspection target 54 as described above. The amount of control of the position and orientation of the photographing device 56a includes the amount of movement and the amount of rotation of the photographing device 56a.

The photographing equipment relative position/posture calculation unit 5 according to the first embodiment obtains the control amount of the position/posture of the inspection object 54 when the viewpoint is temporarily changed based on the viewpoint position/posture information and the operating machine position/posture information. Since the manipulator position/attitude information and the viewpoint position/attitude information are expressed in one specific coordinate system during the calculation, after controlling the inspection object 54 and the viewpoint, each of the inspection object 54 and the viewpoint is can be calculated in one specific coordinate system. The imaging device relative position/orientation calculation unit 5 calculates the control amount of the position and orientation of the viewpoint in the coordinate system related to the inspection object 54 based on the control amount of the viewpoint and the position/orientation of the inspection object 54 in the event of control. Calculate as relative position and orientation information. The imaging equipment relative position/attitude calculation unit 5 transmits the calculated imaging equipment relative position/attitude information to the imaging equipment position/attitude control unit 7 . The photographing equipment relative position/orientation calculator 5 is mounted on the relative position/orientation calculator 58 in FIG.

The light source position/attitude control unit 6 controls the position/attitude of the light source 55a by controlling the light source robot 55 based on the light source relative position/attitude information. The light source robot 55 and the inspection object 54 are calibrated in advance in a coordinate system fixed in advance to the inspection object 54 . Therefore, the light source position/posture control unit 6 converts the data of the light source relative position/posture information into data for operating the light source robot 55 in the coordinate system fixed to the inspection object 54 as necessary. This light source position/orientation control unit 6 is mounted on the inspection equipment control device 59 in FIG.

The imaging equipment position/attitude control section 7 controls the imaging equipment relative position/attitude information by controlling the imaging robot 56, thereby controlling the position/attitude of the imaging equipment 56a. The imaging robot 56 and the inspection target 54 are calibrated in advance in a coordinate system fixed in advance to the inspection target 54 . Therefore, the imaging device position/orientation control unit 7 converts data of the imaging device relative position/orientation information into data for operating the imaging robot 56 in a coordinate system fixed to the inspection target 54 as necessary. This photographing equipment position/orientation control unit 7 is mounted on the inspection equipment control device 59 in FIG.

The photographing device control unit 8 photographs the inspection object 54 with the photographing device 56 a attached to the hand of the photographing robot 56 and transmits the photographed image to the display unit 9 . The display unit 9 displays the transmitted image to the operator 51 . The photographing equipment control unit 8 is mounted on the video transmission device 60 shown in FIG. 1, and the display unit 9 is mounted on the HMD 52 shown in FIG.

<Summary of Embodiment 1>
According to the remote inspection apparatus according to the first embodiment, the light source relative position/posture information for controlling the position/posture of the light source 55a is calculated based on the manipulator position/posture information, and the viewpoint position/posture information and the manipulator position/posture information are calculated. Based on the orientation information, image pickup device relative position and orientation information for controlling the position and orientation of the image pickup device 56a is calculated.

Accordingly, when the manipulator 53 is operated, the positions and orientations of the light source 55a and the photographing device 56a are controlled without controlling the position and orientation of the actual inspection target 54 itself. Therefore, without controlling the position and orientation of the actual inspection target 54, the imaging device 56a can shoot an image as if the position and orientation of the inspection target 54 were controlled. Further, by associating the control of the position and orientation of the light source 55a with the control of the position and orientation of the imaging device 56a, the imaging device 56a can shoot an image as if the position and orientation of the light source 55a were fixed. can.

Further, when the position and orientation of the viewpoint of the operator 51 is changed, the position and orientation of the photographing device 56a are controlled without controlling the position and orientation of the actual inspection object 54 and the position and orientation of the light source 55a. Therefore, it is possible to capture an image of the operator 51 positioned at the operation site looking at the inspection target 54 while moving the head at the inspection site, using the imaging device 56a.

In addition, since the above control can be performed from the viewpoint of the operator 51 and intuitive operation using the operating device 53, the operator does not have to learn the configuration and operation method of the relatively complicated inspection machine. Even without it, the inspection target 54 can be easily inspected by remote control. In addition, since it is not necessary to hold the inspection object 54 and design a robot hand, etc., inspection of the inspection object 54 can be easily performed by remote control, and a reduction in equipment costs can be expected.

<Modification of Embodiment 1>
For the calibration work of each device before starting operation and the state setting work of the inspection environment when starting remote operation, for example, technology and video are used to link multiple devices such as drones and cameras in space. Remote control technology, VR (Virtual Reality) technology, or the like may be used as appropriate. Also, for various functions related to device operation, techniques known for general device operation may be used. Various functions related to equipment operation here are, for example, instruction functions such as remote control start instruction and stop instruction, emergency stop for equipment installed at the inspection site and stop function based on movement restriction, and those functions It includes a presentation function to the operator 51 at the time of execution.

Further, in Embodiment 1, the device configuration has been described using the HMD 52 of FIG. 1, the articulated robot, etc. as an example, but the configuration is not limited to this. For example, instead of the HMD 52 having the functions of the viewpoint position/orientation acquisition unit 1 and the functions of the display unit 9 in FIG. may be used. For example, the device having the function of the viewpoint position/posture acquisition unit 1 may be an eye tracking device that captures the direction of the line of sight of the operator 51 including the movement of the eyeballs. For example, the device having the function of the viewpoint position/orientation acquisition unit 1 may be a device such as a handy camera capable of acquiring the position and orientation and capable of being held by the operator 51 . In this case, for example, a device such as a handy camera may be operated with the right hand, and the operating device 53 may be operated with the left hand. For example, a display for a personal computer may be used as the device having the function of the display unit 9 .

Instead of the articulated robot, another device having a function of freely changing the positions and orientations of the light source 55a and the photographing device 56a, for example, a radio-controlled device called a drone may be used.

Various devices capable of measuring the position and orientation may be used as the manipulator 53 . For example, the operation device 53 may be a controller of a game machine, an internal sensor such as a smartphone, or a recognition device that recognizes with a plurality of external cameras attached with markers. In this case, if an object having a marker attached thereto and having the same shape as the inspection object 54 is used as the operation device 53, the operational feeling can be improved.

Note that the photographing device 56a is not limited to a monocular camera capable of photographing moving images and still images, and may be, for example, a device such as a stereo camera capable of stereoscopic viewing on the display unit 9 side. In addition, if the photographing device 56a has an image processing function such as a filter, the function may be used to improve the operational feeling during examination. For example, if the shadow of the inspection target 54 is reflected on the installation table on which the inspection target 54 is installed, the operator 51 may feel uncomfortable. Therefore, by using an image processing function installed in the imaging device 56a, an object other than the inspection object 54 at the inspection location, such as an installation table, can be deleted from the captured image, thereby improving the operational feeling during inspection.

To facilitate such image processing, the equipment at the inspection site may have a color that differs significantly from the color of the object 54 under inspection. Also, the light source 55a and the background image may be changed according to preset contents. For example, by specifying presets for an indoor environment and outdoor fine weather etc. to the remote inspection device before starting an inspection, if the image of the light from the light source 55a and the background image are changed according to the contents of the presets, the operating environment can be approximated. In this state, inspection of the inspection target 54 becomes possible. Note that the light irradiation mode of the light source 55a may actually be changed at the time of presetting.

The inspection object 54 on the installation table may be installed in various manners. For example, the inspection object 54 may be installed by being hung by a tool, or may be installed by being fixed to the tip of a rod-shaped jig. For example, in a configuration in which a transparent table is used as the installation table, it is possible to photograph the inspection target 54 from various directions without changing the installation mode of the inspection target 54 .

The inspection object 54 inspected by the remote inspection device may be a product produced on a general manufacturing line, or may be a work of art or the like.

<Embodiment 2>
In the first embodiment, it may not be possible to control the photographing device 56a at a desired position and orientation due to contact between the devices at the inspection site and restrictions on the range of motion of equipment for moving the photographing device 56a. In contrast, in the second embodiment, by changing the zoom magnification of the photographing device 56a, it is possible to photograph an image as if the distance between the photographing device 56a and the inspection object 54 was changed. It is possible to relax the above restrictions.

FIG. 3 is a block diagram showing the functional configuration of the remote inspection device according to the second embodiment. In the functional configuration of FIG. 3, a relative distance constraint solver 16 is added to the functional configuration of Embodiment 1 of FIG.

The relative distance constraint solving unit 16 determines whether or not the imaging device 56 a can be controlled based on the imaging device relative position/posture information transmitted from the imaging device relative position/posture calculation unit 5 . For example, the relative distance constraint solving unit 16 determines whether the control based on the imaging equipment relative position/orientation information is the control within the range of motion of the equipment that moves the imaging equipment 56a, and whether the equipment is in contact with each other. It is determined whether or not the photographing device 56a can be controlled, taking into account whether or not it is possible.

Then, the relative distance constraint solving unit 16 controls the zoom magnification of the photographing device 56a when it is determined that the photographing device 56a cannot be controlled, or when it is determined that the zoom magnification is better than the control. For example, the relative distance constraint solving unit 16 calculates the zoom magnification based on the relative distance between the inspection object 54 and the imaging device 56a when the imaging device 56a is temporarily controlled, and zooms the imaging device 56a at the zoom magnification. Control functions.

<Summary of Embodiment 2>
According to the remote inspection apparatus according to the second embodiment, the zoom magnification of the photographing device 56a is controlled based on the photographing device relative position/orientation information. According to such a configuration, since the change in the distance between the viewpoint and the inspection object 54 can be compensated for by the zoom magnification of the photographing device 56a, it is possible to relax the restrictions on the movement of equipment for moving the photographing device 56a. can.

<Embodiment 3>
In the first embodiment, the light source 55a and the imaging device 56a may not be controlled at desired positions and orientations depending on the installation mode of the inspection object 54 and the restrictions of the equipment installed at the imaging location. may not be able to image the inspection target 54 in a desired position and orientation. In contrast, in the third embodiment, by notifying the inspection object 54 of instruction information for changing the position and orientation of the inspection object 54, such as an instruction to turn the inspection object 54 upside down, It is possible to image the inspection target 54 in a desired position and orientation.

FIG. 4 is a block diagram showing the functional configuration of the remote inspection device according to the third embodiment. In the functional configuration of FIG. 4, an instruction acquisition unit 19 and an instruction notification unit 20 are added to the functional configuration of the first embodiment shown in FIG.

The instruction acquisition unit 19 acquires instruction information for changing the position and orientation of the inspection object 54 . The instruction information for changing the position and orientation of the inspection target 54 includes, for example, an instruction to replace the inspection target 54 and an instruction to change the installation mode of the inspection target.

It should be noted that it is preferable that the operator 51 be configured so that the instruction information can be easily input to the remote inspection device. For example, each time the operator 51 performs an input operation on a button provided on the manipulator 53, the manipulator 53 enters a state in which manipulator position/orientation information can be obtained and a state in which instruction information can be obtained. It may be configured to switch. In this configuration, the instruction acquisition unit 19 acquires the position and orientation of the operation device 53 as instruction information for changing the position and orientation of the inspection object 54 while the operation device 53 is switched to a state in which instruction information can be acquired. do. After the command acquisition unit 19 acquires the command information, the manipulator 53 is switched to a state in which the manipulator position/orientation information can be acquired by performing an input operation on the button. With the configuration as described above, the instruction acquisition unit 19 can acquire instruction information from the operator 51 .

The instruction acquisition unit 19 transmits the acquired instruction information to the operation information management unit 3 . The instruction acquisition unit 19 is mounted on the operation control device 57 of FIG. 1, for example.

As in the first embodiment, the operation information management unit 3 aligns the data format of the instruction information with the data format of the viewpoint position/orientation information and the controller position/orientation information, and transmits the data format to the instruction notification unit 20 .

The instruction notification unit 20 notifies the inspection object 54 side of the instruction information. The instruction information is notified, for example, by voice or display to workers waiting at the inspection location. According to the instruction information of the inspection target 54, the operator installs the inspection target 54 in a form that is closest to the position and orientation indicated by the instruction information. When the change of the position and orientation of the inspection target 54 is completed, information indicating that the change has been completed is transmitted to the operator 51 by, for example, the photographing equipment control unit 8 or the like. The instruction notification unit 20 is mounted in, for example, the inspection equipment control device 59 in FIG.

<Summary of Embodiment 3>
According to the remote inspection apparatus according to the third embodiment, instruction information for changing the position and orientation of the inspection object 54 is notified to the inspection object 54 side. According to such a configuration, the position and orientation of the inspection object 54 can be changed by the notification, so that work efficiency can be improved when inspecting the inspection object 54 in various positions and orientations.

<Modification of Embodiment 3>
Each time an input operation is performed on a button of the manipulator 53, the manipulator 53 does not switch between a state in which manipulator position/orientation information can be obtained and a state in which instruction information can be obtained. During the continuation, the state may be switched to a state in which the instruction information can be obtained. Further, the instruction acquisition unit 19 may acquire instruction information for changing the position and orientation of the inspection object 54 from the operator via a device other than the operation device 53 .

Further, when notifying the instruction information, the position and orientation of the three-dimensional model inspection target 54 may be superimposed on the inspection target 54 so that the operator 51 can confirm the instruction information. Also, the notified instruction information may be substantially the same as the acquired instruction information. For example, for several positions and orientations close to the position and orientation of the inspection target 54 indicated by the instruction information, the stability of the inspection target 54 on a plane is calculated using a three-dimensional model, and the positions where the stability is equal to or greater than a threshold are calculated. The posture may be notified to the inspection target 54 side.

Further, the notification of the instruction information is not limited to notification to the operator. It may be a notification to the control device. Then, the inspection object position/posture control device may control the position/posture of the inspection object 54 based on the instruction information.

Also, the configuration described in the third embodiment may be applied to the configurations of the first and second embodiments.

<Embodiment 4>
In Embodiment 1, for example, it is assumed that an object to be inspected 54, such as a top, which can only be placed in a state of being laid down, is inspected. In such a case, if the inspection is started with the imaging direction of the imaging device 56a being the same as the horizontal direction of the installation table of the inspection object 54, it is difficult to understand which part is being inspected. There is a problem that the work efficiency of is somewhat poor. On the other hand, in the fourth embodiment, the starting state setting information including the relative position and orientation information of the light source 55a and the photographing device 56a with respect to the inspection object is acquired, and at the start of the inspection, the light source 55a and the light source 55a and The position and orientation of the photographing device 56a are controlled. This makes it possible to improve the work efficiency of inspection.

FIG. 5 is a block diagram showing the functional configuration of a remote inspection device according to the fourth embodiment. In the functional configuration of FIG. 5, a starting position/orientation obtaining unit 23 is added to the functional configuration of the first embodiment of FIG.

The start position/posture acquisition unit 23 acquires start state setting information. The start state setting information includes relative position and orientation information of the light source 55a and the imaging device 56a with respect to the inspection target 54. FIG. The start position/orientation acquisition unit 23 acquires the start state setting information from the operation information so that the positions and orientations of the light source 55a and the imaging device 56a with respect to the inspection target 54 at the start of the examination are relative to the positions and orientations indicated by the start state setting information. It is transmitted to the management section 3. The operation information management section 3 transmits the start state setting information to the light source position/orientation control section 6 and the photographing equipment position/orientation control section 7 .

The light source position/attitude control unit 6 controls the position/attitude of the light source 55a based on the start state setting information at the start of examination. The imaging equipment position/orientation control unit 7 controls the position/orientation of the imaging equipment 56a based on the start state setting information at the start of the examination. After the control by the light source position/attitude control unit 6 and the photographing equipment position/attitude control unit 7 is performed, the remote inspection apparatus waits for the start of the operation for performing various controls described in the first embodiment.

Note that the starting state setting information may be created based on various information. An example in which the starting state setting information is created based on the design information of the inspection target 54 will be described below. For example, based on the CAD design information of the inspection object 54, the light source 55a and the photographing device 56a are arranged above the real space so that the front direction in the CAD is aligned with the direction from the upper side to the lower side in the real space. Starting state setting information may be created. Further, for example, based on the design information of the inspection target 54, start state setting information may be created for arranging the photographing device 56a so that the design portion of the inspection target 54 faces the front. Also, for example, by learning using AI (Artificial Intelligence) or the like on the past inspection history, the photographing is performed so that the part of the past inspection object 54 similar to the current inspection object 54 that has been intensively inspected is the front. Starting state setting information for arranging the device 56a may be created.

<Summary of Embodiment 4>
According to the remote inspection apparatus according to the fourth embodiment, at the start of inspection, the positions and orientations of the light source 55a and the imaging device 56a are controlled based on the start state setting information. According to such a configuration, for example, in the inspection of the inspection object 54 lying down on the installation table, the inspection can be started in a state in which the imaging direction of the imaging device 56a is aligned with the front direction of the inspection object 54 in the CAD. Therefore, the work efficiency of inspection can be improved.

<Modification of Embodiment 4>
The configuration described in the fourth embodiment may be applied to the configurations of the first to third embodiments. For example, in the fourth embodiment, at the start of the examination, the instruction information of the third embodiment is notified before and after the positions and orientations of the light source 55a and the photographing device 56a are controlled based on the start state setting information. good too.

<Other Modifications>
The viewpoint position/orientation acquisition unit 1, the controller position/orientation acquisition unit 2, the operation information management unit 3, the light source relative position/orientation calculation unit 4, the imaging device relative position/orientation calculation unit 5, the light source position/orientation control unit 6, and In addition, the photographing equipment position/orientation control unit 7 is hereinafter referred to as "viewpoint position/orientation acquisition unit 1, etc.". The viewpoint position/posture acquisition unit 1 and the like are implemented by the processing circuit 81 shown in FIG. That is, the processing circuit 81 includes a viewpoint position/posture acquisition unit 1 that acquires viewpoint position/posture information, a controller position/posture acquisition unit 2 that acquires manipulator position/posture information, a data format of the viewpoint position/posture information and a manipulator position An operation information management unit 3 that aligns the data format of the attitude information, and a light source relative position/attitude calculation unit 4 that calculates the light source relative position/attitude information based on the manipulator position/attitude information that has the data format aligned, and the data format is aligned. a photographing equipment relative position/attitude calculation unit 5 for calculating photographing equipment relative position/attitude information based on the obtained viewpoint position/attitude information and the controller position/attitude information; A light source position/attitude control unit 6 and an imaging equipment position/attitude control unit 7 for controlling the position/attitude of the imaging equipment 56a based on the imaging equipment relative position/attitude information are provided. Dedicated hardware may be applied to the processing circuit 81, or a processor that executes a program stored in a memory may be applied. Processors include, for example, central processing units, processing units, arithmetic units, microprocessors, microcomputers, and DSPs (Digital Signal Processors).

When the processing circuit 81 is dedicated hardware, the processing circuit 81 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. Each function of each unit such as the viewpoint position/orientation acquisition unit 1 may be realized by a circuit in which processing circuits are distributed, or the functions of each unit may be collectively realized by one processing circuit.

When the processing circuit 81 is a processor, the functions of the viewpoint position/orientation acquisition section 1 and the like are realized by combining with software and the like. Software and the like correspond to, for example, software, firmware, or software and firmware. Software or the like is written as a program and stored in memory. As shown in FIG. 7, a processor 82 applied to a processing circuit 81 reads out and executes a program stored in a memory 83 to realize functions of each section. That is, when the remote inspection apparatus is executed by the processing circuit 81, a step of acquiring viewpoint position/posture information, a step of acquiring manipulator position/posture information, a data format of the viewpoint position/posture information and a manipulator position/posture information. calculating light source relative position/posture information based on the manipulator position/attitude information with the data format aligned; viewpoint position/posture information and the manipulator position/posture information with the data format aligned; calculating the relative position and orientation information of the imaging device based on the information; controlling the position and orientation of the light source 55a based on the relative position and orientation information of the light source; and determining the position of the imaging device 56a based on the relative position and orientation information of the imaging device. A memory 83 is provided for storing the program resulting from the step of controlling the attitude. In other words, it can be said that this program causes a computer to execute the procedures and methods of the viewpoint position/posture acquisition unit 1 and the like. Here, the memory 83 is, for example, a non-volatile or Volatile semiconductor memories, HDDs (Hard Disk Drives), magnetic disks, flexible disks, optical disks, compact disks, mini disks, DVDs (Digital Versatile Disks), their drive devices, etc., or any storage media that will be used in the future. may

The configuration in which each function of the viewpoint position/orientation acquisition unit 1 and the like is realized by either hardware or software has been described above. However, the present invention is not limited to this, and a configuration may be adopted in which part of the viewpoint position/orientation acquisition unit 1 and the like is implemented by dedicated hardware, and another part is implemented by software or the like. For example, the viewpoint position/orientation acquisition unit 1 is implemented by a processing circuit 81 as dedicated hardware, an interface, a receiver, and the like, and the processing circuit 81 as a processor 82 is a program stored in a memory 83 for other functions. It is possible to realize the function by reading and executing the .

As described above, the processing circuit 81 can realize each function described above by using hardware, software, etc., or a combination thereof.

It should be noted that it is possible to freely combine each embodiment and each modification, and to modify or omit each embodiment and each modification as appropriate.

1 viewpoint position/orientation acquisition unit 2 manipulator position/orientation acquisition unit 3 operation information management unit 4 light source relative position/orientation calculation unit 5 imaging device relative position/orientation calculation unit 6 light source position/orientation control unit 7 imaging device position/orientation Control unit 16 Relative distance constraint solver 19 Instruction acquisition unit 20 Instruction notification unit 23 Start position/orientation acquisition unit 51 Operator 53 Manipulator 54 Inspection object 55a Light source 56a Imaging device.

Claims (5)

a viewpoint position/posture acquisition unit that acquires a viewpoint position/posture of an operator as viewpoint position/posture information;
a manipulator position/orientation acquisition unit that acquires the position and orientation of the manipulator held by the operator as manipulator position/orientation information;
an operation information management unit for matching a data format of the viewpoint position/attitude information and a data format of the manipulator position/attitude information;
a light source relative position/posture calculation unit that calculates, as light source relative position/posture information, a control amount of the position/posture of the light source in a coordinate system related to the inspection object, based on the manipulator position/posture information in a uniform data format;
Imaging for calculating a control amount of the position and orientation of an imaging device in the coordinate system with respect to the inspection object as imaging device relative position and orientation information based on the viewpoint position and orientation information and the manipulator position and orientation information in which data formats are aligned. a device relative position/orientation calculator;
a light source position/attitude control unit that controls the position/attitude of the light source based on the light source relative position/attitude information;
A remote inspection apparatus comprising: an imaging equipment position/orientation control unit that controls the position/orientation of the imaging equipment based on the imaging equipment relative position/orientation information. The remote inspection device of claim 1, comprising:
A remote inspection apparatus, further comprising a relative distance constraint solver that controls a zoom magnification of the imaging device based on the imaging device relative position and orientation information. 3. A remote inspection device according to claim 1 or claim 2,
an instruction acquisition unit that acquires instruction information for changing the position and orientation of the inspection target;
and an instruction notification unit that notifies the inspection target side of the instruction information. A remote inspection device according to any one of claims 1 to 3, wherein
further comprising a start position/orientation acquisition unit that acquires start state setting information including relative position/orientation information of the light source and the imaging device with respect to the inspection object;
The light source position/attitude control unit controls the position/attitude of the light source based on the start state setting information at the start of examination,
The remote inspection apparatus, wherein the imaging equipment position/orientation control unit controls the position/orientation of the imaging equipment based on the start state setting information at the start of an examination. Acquiring the position and orientation of the operator's viewpoint as viewpoint position and orientation information,
acquiring the position and orientation of the manipulator held by the operator as manipulator position and orientation information;
aligning the data format of the viewpoint position and orientation information with the data format of the manipulator position and orientation information;
calculating, as light source relative position/attitude information, a control amount of the position/attitude of the light source in a coordinate system related to the inspection object, based on the manipulator position/attitude information in which the data format is aligned;
calculating a control amount of the position and orientation of the imaging device in the coordinate system for the inspection target as imaging device relative position and orientation information based on the viewpoint position and orientation information and the manipulator position and orientation information in which data formats are aligned;
controlling the position and orientation of the light source based on the light source relative position and orientation information;
A remote inspection method, comprising controlling the position and orientation of the imaging equipment based on the imaging equipment relative position and orientation information.

Images