WO2022219780A1

WO2022219780A1 - Inspection assistance device, inspection assistance system, method for assisting inspection, and inspection assistance program

Info

Publication number: WO2022219780A1
Application number: PCT/JP2021/015597
Authority: WO
Inventors: 隆博加島; 愛梨守谷; 健央川浦; 琴由笹山; 結佳高橋; 智也上船; 淳志堀
Original assignee: 三菱電機株式会社
Priority date: 2021-04-15
Filing date: 2021-04-15
Publication date: 2022-10-20
Also published as: JPWO2022219780A1; JP7361992B2

Abstract

This inspection assistance device (100) has: a terminal position/orientation estimation unit (120) that estimates the position and orientation of a portable terminal (10) in a three-dimensional coordinate system; an inspection object detection unit (130) that detects an inspection object from a camera image photographed by a camera (110); an inspection object tracking unit (140) that tracks inspection object regions (821-825), which indicate the inspection object detected in the camera image, in time series in the order of time of photographing, and that identifies the same inspection object regions (821b-825b) in a plurality of camera images arranged in time series; an inspection object position estimation unit (150) that estimates the coordinates of the inspection object regions in the three-dimensional coordinate system on the basis of the position and orientation of the portable terminal (10) and inspection object regions in at least two camera images among the plurality of camera images; and a display control unit (170) that causes a display device (180) to display a display component for making the inspection object regions visible.

Description

Inspection support device, inspection support system, inspection support method, and inspection support program

The present disclosure relates to an inspection support device, an inspection support system, an inspection support method, and an inspection support program.

There is a proposal for a technology that supports the inspection of infrastructure facilities using mobile terminals such as smartphones and tablet computers, using augmented reality (AR) technology. See, for example, US Pat. A mobile terminal to which this technology is applied detects an inspection target equipped with a reference marker, acquires registration information of the detected inspection target from the management database, and displays the acquired information superimposed on the camera image. can be done.

JP 2020-80147 (for example, paragraphs 0012 to 0037)

However, with the above conventional technology, the user had to perform troublesome preparatory work such as attaching reference markers to the inspection target in advance and inputting the position information of the inspection target into the database in advance.

An object of the present disclosure is to provide an inspection support device, an inspection support system, an inspection support method, and an inspection support program that can support inspection work without requiring preparatory work.

An inspection support device according to the present disclosure, based on at least one of a camera image captured by a camera provided in a mobile terminal and a detection signal obtained by an inertial sensor provided in the mobile terminal, determines three-dimensional coordinates a terminal position/orientation estimation unit that estimates the position and orientation of the mobile terminal in the system; an inspection object detection unit that detects an inspection object from a camera image captured by the camera; an inspection target tracking unit that tracks the indicated inspection target area in chronological order of photographing time and identifies the same inspection target area in a plurality of camera images arranged in time series; an inspection target position estimating unit for estimating the coordinates of the inspection target region in the three-dimensional coordinate system based on the inspection target region in at least two camera images out of the camera images; and a display control unit that causes a display device to display a display component to be used.

An inspection support method of the present disclosure is a method executed by an inspection support device, and includes a camera image captured by a camera provided in a mobile terminal and a detection signal obtained by an inertial sensor provided in the mobile terminal. estimating the position and orientation of the mobile terminal in a three-dimensional coordinate system based on at least one of; detecting an inspection target from a camera image captured by the camera; a step of tracking the inspection target area indicating the inspection target in chronological order of photographing time, and identifying the same inspection target area in a plurality of camera images arranged in time series; estimating the coordinates of the inspection target area in the three-dimensional coordinate system based on the inspection target area in at least two camera images out of the camera images; and making the inspection target area visible. and a step of displaying the display component on a display device.

According to the present disclosure, inspection work can be supported without the need for preparatory work.

1 is a functional block diagram schematically showing configurations of an inspection support device and a portable terminal according to Embodiment 1; FIG. FIG. 2 is a diagram showing an example of a terminal device and an inspection target according to Embodiment 1; FIG. 4 is a diagram showing a display example of the display device of the terminal device according to Embodiment 1; FIG. 2 is a diagram illustrating an example of hardware configurations of an inspection support device and a mobile terminal according to Embodiment 1; FIG. 3 is a diagram showing another example of the hardware configuration of the inspection support device and the mobile terminal according to Embodiment 1; FIG. 7 is a flowchart showing a terminal position/orientation estimation thread by the inspection support device according to the first embodiment; 7 is a flow chart showing an inspection target acquisition thread by the inspection support device according to the first embodiment; FIG. 10 is a functional block diagram schematically showing configurations of an inspection support device and a portable terminal according to Embodiment 2; FIG. 10 is a diagram showing a display example of a camera image of the terminal device according to Embodiment 2 (at the time of reading a character string); FIG. 11 is a diagram showing a display example of a display device of a terminal device according to Embodiment 2; FIG. 11 is a functional block diagram schematically showing the configuration of an inspection support device and a portable terminal according to Embodiment 3; FIG. 11 is a diagram showing a display example of a display device of a terminal device according to Embodiment 3; FIG. 11 is a functional block diagram schematically showing the configuration of an inspection support device and a portable terminal according to Embodiment 4; FIG. 11 is a functional block diagram schematically showing the configuration of an inspection support device and a portable terminal according to Embodiment 5; FIG. 12 is a functional block diagram schematically showing the configuration of an inspection support device and a portable terminal according to Embodiment 6; FIG. 13 is a diagram showing a display example of a display device of a terminal device according to Embodiment 6; FIG. 14 is a functional block diagram schematically showing the configuration of an inspection support device and a portable terminal according to Embodiment 7; FIG. 12 is a diagram showing a display example of a display device of a terminal device according to Embodiment 7;

The inspection support device, inspection support system, inspection support method, and inspection support program according to the embodiment will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be combined as appropriate and each embodiment can be modified as appropriate.

Embodiment 1.
FIG. 1 is a functional block diagram schematically showing configurations of an inspection support device 100 and a mobile terminal 10 according to Embodiment 1. As shown in FIG. The inspection support device 100 is a device capable of executing the inspection support method according to the first embodiment. The inspection support device 100 can execute an inspection support method by executing an inspection support program.

As shown in FIG. 1, the mobile terminal 10 has an inspection support device 100, a camera 110, and a display device 180. Camera 110 is, for example, an RGB camera. Camera 110 may be a stereo camera. The display device 180 is, for example, a display device such as a liquid crystal display device. The display device 180 may be a touch panel that has a function as a display device and a function as an input unit that is a user operation unit. The mobile terminal 10 may also include an inertial sensor (described later with reference to FIG. 4).

As shown in FIG. 1 , inspection support device 100 includes terminal position/orientation estimation unit 120 , inspection object detection unit 130 , inspection object tracking unit 140 , inspection object position estimation unit 150 , and display device 180 . and a display control unit 170 for displaying. The inspection support device 100 may have an inspection target storage unit 160 that is a storage device that stores information.

The terminal position and orientation estimation unit 120 estimates the position and orientation (that is, the position and orientation) of the mobile terminal in a three-dimensional coordinate system representing the world space from the camera image captured by the camera 110 . This three-dimensional coordinate system does not need to be a coordinate system whose origin is a predetermined position (for example, predetermined longitude and latitude on the earth or a predetermined building point). The three-dimensional coordinate system does not have to be a predetermined coordinate system. It may be determined based on the position and orientation of the mobile terminal 10 . That is, the terminal position/orientation estimating unit 120 calculates the position and orientation of the mobile terminal based on the position and orientation of the mobile terminal when the mobile terminal is activated (that is, when the device is activated) or when the inspection support program is activated (that is, when the application is activated). A dimensional coordinate system may be set. Therefore, by integrating the change in the position and orientation of the mobile terminal from the position and orientation of the mobile terminal at the time of startup (that is, at the time of device startup or application startup), a three-dimensional coordinate system with the position at the time of startup as the origin , the position and orientation of the mobile terminal at each time can be estimated. As an estimation method, there is a method of calculating the position and orientation at each time based on the information of the feature amount in the camera image, and a method of calculating the position and orientation at each time based on information such as the velocity from the acceleration sensor and the rotation from the angular velocity sensor. There is An existing method can be used as a method for estimating the position and orientation.

If the mobile terminal 10 has an inertial sensor (shown in FIG. 3), the terminal position/orientation estimation unit 120 detects at least one of a camera image captured by the camera 110 and a detection signal obtained by the inertial sensor. , the position and orientation of the mobile terminal 10 in the three-dimensional coordinate system may be estimated. In addition, terminal position/orientation estimation section 120 performs 3 A position and orientation of the mobile terminal (10) in a dimensional coordinate system may be estimated.

The inspection target detection unit 130 detects inspection targets (eg, valves, gauges, etc.) from the camera image captured by the camera 110 .

The inspection target tracking unit 140 tracks the inspection target area indicating the inspection target detected in the multiple camera images arranged in time series, and identifies the same inspection target area in the multiple camera images arranged in time series. The inspection target tracking unit 140 compares the inspection target region in the current camera image detected by the inspection target detection unit 130 with the inspection target region in the previous detection, thereby determining the inspection target region on the camera image. Track position changes over time. Since the inspection target detection unit 130 performs detection processing for each camera image (that is, each frame of the camera image), the relationship between the inspection target areas detected in the camera images before and after the time is unknown. Through the processing of the inspection object tracking unit 140, the relationship between the inspection object areas detected in the camera images before and after the time is grasped. For example, the inspection target tracking unit 140 determines that the same inspection target is indicated when a part of the detected inspection target region rectangle overlaps between the camera images before and after the time. Alternatively, the inspection target tracking unit 140 uses a technique called optical flow to calculate the movement direction and amount of movement of pixels between camera images, and compares the previous and subsequent inspection target areas with the movement of pixels, thereby performing an inspection. Able to identify relationships between regions of interest.

The inspection target position estimation unit 150 calculates the coordinates of the inspection target region in the three-dimensional coordinate system based on the position and orientation of the mobile terminal 10 and the inspection target regions in at least two camera images out of the plurality of camera images. presume.

The inspection target storage unit 160 stores the coordinates of the inspection target area indicating the inspection target. The display control unit 170 causes the display device 180 to display a display component indicating the inspection target area.

The inspection target storage unit 160 stores the three-dimensional coordinates of multiple inspection targets estimated by the inspection target position estimation unit 150 . Depending on the embodiment, additional information associated with the inspection object may be stored. The inspection target storage unit 160 is a storage device that holds programs or various data (it may be located outside the inspection support device and connected via a network or the like to achieve the same purpose).

Based on the current camera image obtained from camera 110, the current position/orientation obtained from terminal position/orientation estimation section 120, and the three-dimensional coordinates of the inspection target held by inspection target storage section 160, display control section 170 , displays the AR on the screen. Since the position and orientation of the mobile terminal and the coordinates of the inspection object are in the same coordinate system of the three-dimensional space, it is sufficient to draw three-dimensional graphics representing the inspection object on the camera image. For drawing, for example, a three-dimensional graphics drawing API (Application Programming Interface) such as OpenGL (registered trademark) can be used.

FIG. 2 is a diagram showing an example of the mobile terminal 10 having the inspection support device 100 according to Embodiment 1 and an inspection object. The equipment shown in FIG. 2 has a pipe 810, a tank 830, and valves 821 to 825 provided in the pipe 810 and subject to inspection. Further, near the valves 821-825, there are provided nameplates 821a-825a on which the names and identification numbers of inspection objects are written. The equipment shown in FIG. 2 is an example of equipment to which the inspection support method according to Embodiment 1 can be applied, and the inspection support method can also be applied to other equipment. When the user 800 moves with the mobile terminal 10, the mobile terminal 10 detects the valve to be inspected, and displays the valve to be inspected on the display device 180 (for example, frame, highlighting, etc.).

FIG. 3 is a diagram showing a display example of the display device 180 of the mobile terminal 10 having the inspection support device 100 according to the first embodiment. The user 800 simply points the camera 110 of the mobile terminal 10 toward the facility, and the mobile terminal 10 detects the valves 821 to 825 to be inspected, and displays the valves 821 to 825 to be inspected on the display device 180 (for example, frames of inspection target areas 821b to 825b, etc.).

FIG. 4 is a diagram showing an example of the hardware configuration of the inspection support device 100 and the mobile terminal 10 according to Embodiment 1. As shown in FIG. The mobile terminal 10 is, for example, a smart phone, a tablet computer, or a personal computer (PC). The mobile terminal 10 has an inspection support device 100 , a storage section 103 , a communication section 104 , a camera 110 , an inertial sensor 210 , a display device 180 and an input section 190 . The inspection support device 100 has a processor 101 and a memory 102 . Inertial sensors 210 include, for example, one or more of acceleration sensors, angular velocity sensors, and geomagnetic sensors. In the example of FIG. 4, the mobile terminal 10 is the inspection support system.

As shown in FIG. 4, the inspection support device 100 has a processor 101 such as a CPU (Central Processing Unit) and a memory 102 that is a volatile storage device. The memory 102 is, for example, a volatile semiconductor memory such as a RAM (Random Access Memory). As shown in FIG. 1, the inspection support device 100 may have a nonvolatile storage device such as a hard disk drive (HDD) or solid state drive (SSD).

Each function of the inspection support device 100 is implemented by a processing circuit. The processing circuitry may be dedicated hardware or processor 101 executing a program stored in memory 102 . The processor 101 may be any of a processing device, an arithmetic device, a microprocessor, a microcomputer, and a DSP (Digital Signal Processor).

If the processing circuit is dedicated hardware, the processing circuit may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) ), or a combination of any of these.

When the processing circuit is the processor 101, the inspection support program is realized by software, firmware, or a combination of software and firmware. Software and firmware are written as programs and stored in memory 102 . The processor 101 reads out and executes the inspection support program stored in the memory 102, thereby implementing the functions of the units shown in FIG.

It should be noted that the inspection support device 100 may be partially realized by dedicated hardware and partially realized by software or firmware. As such, the processing circuitry may implement each of the functions described above in hardware, software, firmware, or any combination thereof.

FIG. 5 is a diagram showing another example of the hardware configuration of the inspection support device 100a and the mobile terminal 10a according to the first embodiment. In the example of FIG. 5, the inspection support device 100a is a device capable of communicating with the mobile terminal 10a. The mobile terminal 10a is, for example, a smart phone, a tablet computer, or a PC. Portable terminal 10 a has processor 105 , memory 106 , storage unit 107 , communication unit 108 , camera 110 , inertial sensor 210 , display device 180 , and input unit 190 . The inspection support device 100 a has a processor 101 , a memory 102 , a storage section 103 and a communication section 104 .

As shown in FIG. 5, the inspection support device 100a has a processor 101 such as a CPU and a memory 102. Memory 102 is, for example, a volatile semiconductor memory such as RAM. As shown in FIG. 1, the inspection support device 100a may have a non-volatile storage device such as an HDD or SSD. In the example of FIG. 5, the combination of the mobile terminal 10a and the inspection support device 100a is the inspection support system.

In the processing of the first embodiment, the terminal position/orientation estimation thread, the inspection target acquisition thread, and the AR display thread are processed in parallel. FIG. 6 is a flowchart showing the terminal position/orientation estimation processing S11 in the terminal position/orientation estimation thread by the inspection support apparatus 100 according to the first embodiment. The terminal position/orientation estimation unit 120 acquires the current camera image from the camera 110 and the current sensor information from the sensor, estimates the current position/orientation of the mobile terminal, and provides the inspection target position estimation unit 150 and the display control unit 170 Notify the position and orientation.

FIG. 7 is a flowchart showing an inspection target acquisition thread by the inspection support device 100 according to the first embodiment. In the inspection object acquisition thread, the inspection object detection unit 130 acquires the current camera image from the camera 110 and detects zero or more inspection object areas from the camera image (step S21).

Next, the inspection target tracking unit 140 tracks the inspection target area between the camera images in time series (step S22).

Next, the following loop processing is performed for each inspection target area (step S23).
First, the inspection target position estimation unit 150 estimates the position (ie, three-dimensional coordinates) of the inspection target region based on the current position and orientation of the mobile terminal 10 and the tracked inspection target region (step S24). Next, the coordinates estimated above are compared with the coordinates of each inspection object stored in the inspection object storage unit 160, and if the coordinates are the same or the distance between the coordinates is within a certain value ( Steps S25, S26), it is determined that the inspection object has already been added to the inspection object storage unit 160, the loop processing returns to the beginning, and in other cases, the inspection object is added to the inspection object storage unit 160. (Step S27).

In the AR display thread, the display control unit 170 acquires the current camera image from the camera 110, the current position and orientation of the mobile terminal from the terminal position/orientation estimation unit 120, and a plurality of inspection targets from the inspection target storage unit 160, Display the AR on the screen.

As described above, by using the inspection support device 100 and the mobile terminal 10 according to Embodiment 1, there is no need to perform operations such as attaching a marker to an inspection target and creating a three-dimensional model in advance. The object to be inspected can be displayed by AR without creating location information of the object to be inspected in advance. Since the AR is displayed by estimating the three-dimensional coordinates of the inspection target, even if the inspection target is no longer detected or is hidden behind something, the AR indicating the position of the inspection target can be continuously displayed. can.

Embodiment 2.
FIG. 8 is a functional block diagram schematically showing configurations of inspection support device 200 and mobile terminal 20 according to Embodiment 2. As shown in FIG. 8, the same or corresponding components as those shown in FIG. 1 are given the same reference numerals as those shown in FIG. Inspection support apparatus 200 is an inspection support apparatus in that it includes character recognition unit 220 and recognition target acquisition unit 230, and terminal position/orientation estimation unit 120 estimates the position and orientation using the output of inertial sensor 210. Different from 100.

FIG. 9 is a diagram showing a display example of a camera image of the mobile terminal 20 according to Embodiment 2 (at the time of reading a character string). The character recognition unit 220 recognizes, from the camera image acquired by the camera 110, the identification number, which is the character string of the nameplate 821a that identifies the valve 821 to be inspected. Recognition processing is executed when instructed by the user or is always executed.

Based on the position and orientation of the mobile terminal 20 estimated by the terminal position/orientation estimation unit 120, the recognition target acquisition unit 230 stores an inspection target corresponding to the inspection target with the identification number recognized by the character recognition unit 220. The inspection target stored by the unit 160 is acquired. That is, among the three-dimensional coordinates of a plurality of inspection objects stored in the inspection object storage unit 160, an inspection object having three-dimensional coordinates corresponding to the inspection object recognized by the character recognition unit 220 is searched. Specifically, for example, the recognition target acquisition unit 230 searches the inspection target storage unit 160 for an inspection target having three-dimensional coordinates closest to the current position of the mobile terminal 20, and sets it as the current recognition target. Alternatively, the recognition target acquiring unit 230 searches for the inspection target having the closest three-dimensional coordinates from a position that is an arbitrary distance (for example, 30 cm) in the viewing direction of the camera 110 from the current position of the mobile terminal 20, and performs the current recognition. set to target. In addition, the recognition target acquisition unit 230 may add the information “checked” to the inspection target stored in the inspection target storage unit 160 to display the inspection target differently from the uninspected inspection target. In this manner, the inspection target storage unit 160 holds information indicating whether or not each inspection target has been inspected in addition to the information described in the first embodiment.

FIG. 10 is a diagram showing a display example of the display device 180 of the mobile terminal 20 according to the second embodiment. The display control unit 170 changes the display method of the inspection object according to whether the inspection object recognized by the recognition object acquisition unit 230 has been inspected. The display is performed using AR, for example. For example, the color of the frame of the AR image that surrounds the inspection target that has already been inspected and the color of the frame of the AR image that surrounds the inspection target that has not yet been inspected are set to different colors. FIG. 10 shows an example in which an AR image is displayed superimposed on a camera image. By using smart glasses, an AR image can be displayed superimposed on a real landscape. Also, by using a projector, AR images may be projected onto real installations (eg, pipes, valves). Also, if the mobile terminal 20 has an application for forming an illustration figure simulating a camera image, the AR image may be superimposed on the illustration simulating the camera image.

As described above, if the inspection support device 200 and the mobile terminal 20 according to the second embodiment are used, the identification number of the inspection object is recognized by character recognition, and the inspection object storage unit 160 determines that the inspection object has been inspected. By storing the AR image, it is possible to distinguish whether or not each inspection object has been inspected. In addition, it is possible to prevent mistakes in inputting the identification number by making the inspection completed by character recognition.

Except for the above, the second embodiment is the same as the first embodiment.

Embodiment 3.
FIG. 11 is a functional block diagram schematically showing configurations of inspection support device 300 and mobile terminal 30 according to the third embodiment. 11, the same or corresponding components as those shown in FIG. 8 are given the same reference numerals as those shown in FIG. Inspection support apparatus 300 according to Embodiment 3 differs from inspection support apparatus 200 according to Embodiment 2 in terms of the function of recognition target acquisition section 230a.

The inspection target position estimating unit 150 estimates the coordinates of the inspection target region in the three-dimensional coordinate system based on the position and orientation of the mobile terminal 30 and the inspection target regions in at least two of the plurality of camera images. and stored in the inspection target storage unit 160 . The display control unit 170 causes the display device 180 to display display components (for example, AR display) and recognized characters.

The recognition target acquisition unit 230a also stores the identification number recognized by the character recognition unit 220 when rewriting the inspection target stored in the inspection target storage unit 160 to "inspected".

The inspection target storage unit 160 also stores the identification number of each inspection target. If it has not been inspected, the identification number shall be unknown. When the inspection target is displayed in AR, the display control unit 170 also displays the identification number in AR if the identification number of the inspection target is known.

FIG. 12 is a diagram showing a display example of the display device 180 of the mobile terminal 30 according to the third embodiment. FIG. 12 displays an identification number 821c around (that is, in the vicinity of) the valve 821 to be inspected.

As described above, according to the inspection support device 300 and the mobile terminal 30 according to the third embodiment, by displaying the identification number in the vicinity of the recognized inspection object by the AR image, easier to distinguish.

Except for the above, Embodiment 3 is the same as Embodiment 1 or 2.

Embodiment 4.
FIG. 13 is a functional block diagram schematically showing configurations of inspection support device 400 and portable terminal 40 according to the fourth embodiment. 13, the same or corresponding components as those shown in FIG. 11 are given the same reference numerals as those shown in FIG. Inspection support apparatus 400 according to Embodiment 4 differs from inspection support apparatus 300 according to Embodiment 3 in that it includes terminal position/posture storage section 410 and terminal position/posture recovery section 420 .

In Embodiment 4, the terminal position/orientation estimation unit 120 associates the camera image with the position/orientation of the mobile terminal 40 and stores them in the terminal position/orientation storage unit 410 . The terminal position/orientation restoring unit 420 estimates the terminal position/orientation using the position/orientation associated with the camera image closest to the current camera image among the camera images stored in the terminal position/orientation estimation unit 120 as the current position/orientation. 120.

The terminal position/orientation storage unit 410 periodically (for example, every 10 seconds) stores the relationship between the current camera image and the current position/orientation of the mobile terminal estimated by the terminal position/orientation estimation unit 120 . A plurality of such relationships are held. The camera image may not be raw data of the image, but may be processed/processed data as long as it is in a format that enables matching between images, which will be described later. The data may be in formats such as SIFT (Scale-invariant feature transform), SURF (Speed-Upped Robust Feature), ORB (Oriented FAST and Rotated BRIEF), and KAZE, which are local features.

The terminal position/orientation recovery unit 420 matches the current camera image with each camera image stored in the terminal position/orientation storage unit 410 when the inspection support device is activated, and returns the terminal position/orientation closest to the current camera image. A camera image stored in the storage unit 410 is specified. When the identification is successful, the terminal position/orientation estimating unit 120 is initialized with the position/orientation of the mobile terminal linked to the camera image as the current position/orientation of the mobile terminal.

As described above, by using the inspection support device 400 and the mobile terminal 40 according to the fourth embodiment, the camera image and the position and orientation of the mobile terminal 40 are periodically stored, and when the mobile terminal 40 is activated, the current By matching the previous camera image with the camera image stored in the past, the previous three-dimensional coordinate system can be inherited and the inspection work can be restarted. Therefore, the AR display can be resumed immediately without the need to detect and estimate the position of the object to be inspected again.

Except for the above, Embodiment 4 is the same as any of Embodiments 1 to 3.

Embodiment 5.
FIG. 14 is a functional block diagram schematically showing configurations of inspection support device 500 and mobile terminal 50 according to Embodiment 5. As shown in FIG. 14, the same or corresponding components as those shown in FIG. 13 are given the same reference numerals as those shown in FIG. Inspection support device 500 according to Embodiment 5 differs from inspection support device 400 according to Embodiment 4 in that inspection state initialization section 510 is provided.

The inspection state initialization unit 510 initializes some or all of the inspections held by the inspection target storage unit 160 when the inspection support device 500 is activated or when specified by the user via the input unit (190 in FIG. 4). Information on the inspection status of the target is set to uninspected.

As described above, a new inspection can be started using the inspection support device 500 and the mobile terminal 50 according to the fifth embodiment. In other words, since the three-dimensional coordinates used to represent the coordinates of the inspection target have already been obtained once, it may not be necessary to detect the inspection target and estimate the coordinates again. Inspection can be started in the displayed state. In addition, since the identification number is also known, it is possible to search for an uninspected object to be inspected based on the identification number.

Except for the above, Embodiment 5 is the same as any of Embodiments 1 to 4.

Embodiment 6.
FIG. 15 is a functional block diagram schematically showing configurations of inspection support device 600 and mobile terminal 60 according to Embodiment 6. As shown in FIG. 15, the same or corresponding components as those shown in FIG. 14 are given the same reference numerals as those shown in FIG. The inspection support device 600 according to the sixth embodiment differs from the inspection support device 500 according to the fifth embodiment in that it has an inspection procedure storage unit 610 and an inspection procedure management unit 620 .

The inspection procedure storage unit 610 stores information indicating an inspection procedure created in advance. The information indicating the inspection procedure is information indicating the order of inspection of a plurality of inspection targets, and is specifically a list of arranged identification numbers.

The inspection procedure management unit 620 manages the current inspection procedure status. If the identification number is read by the character recognition unit 220 and the identification number is the identification number of the current inspection procedure, the inspection procedure is advanced by one. Information on the current inspection procedure is provided to the display control unit 670 . That is, the inspection procedure management unit 620 provides information to the display control unit 670 based on the order of inspection of the inspection objects stored in the inspection procedure storage unit 610 and the recognized characters, and displays the next inspection object. is displayed on the display device 180 .

When the inspection target is displayed by AR, if the identification number of the inspection target is the identification number of the current inspection target managed by the inspection procedure management unit 620, the display control unit 670 displays the inspection target. Display the target AR so that it stands out compared to other inspection targets. For example, when it is the identification number of an inspection object currently being inspected managed by the inspection procedure management unit 620, the display control unit 670 makes the AR display for the inspection object stand out compared to other inspection objects. , change the color of the AR display, or blink the display part indicating the inspection target.

FIG. 16 is a diagram showing a display example of the display device 180 of the mobile terminal 60 according to the sixth embodiment. FIG. 16 shows that the valve 823 is to be inspected next. The display method of the next inspection target is not limited to that of FIG.

As described above, by using the inspection support device 600 and the mobile terminal 60 according to Embodiment 6, it is possible to display the position of the inspection target in AR, so that the inspection can be performed intuitively according to the procedure. . In this case, it is necessary to create inspection procedure data in advance, but since the data indicates the order of inspection, the creation effort is small and the creation cost is low.

Except for the above, Embodiment 6 is the same as any of Embodiments 1 to 5.

Embodiment 7.
FIG. 17 is a functional block diagram schematically showing configurations of inspection support device 700 and mobile terminal 70 according to Embodiment 7. As shown in FIG. 17, the same or corresponding components as those shown in FIG. 14 are given the same reference numerals as those shown in FIG. Inspection support device 700 according to Embodiment 7 differs from inspection support device 500 according to Embodiment 5 in that it includes an identification number confirmation unit 710 .

Identification number confirmation unit 710 confirms that the identification number recognized this time by character recognition unit 220 is the identification number for the same inspection object already stored in inspection object storage unit 160 (that is, the identification number recognized in the past). It further has an identification number confirming unit for displaying a warning on the screen of the display device 180 when the numbers do not match. For example, when the character recognition unit 220 and the recognition target acquisition unit 230 change the inspection status information of the inspection target stored in the inspection target storage unit 160 from "unchecked" to "checked", the inspection target storage unit 160 has already stored an identification number, and the identification number is different from the identification number recognized this time, the identification number confirmation unit 710 displays a warning on the display device. 180 screen.

FIG. 18 is a diagram showing a display example of the display device 180 of the mobile terminal 70 according to the seventh embodiment. As shown in FIG. 18 , identification number confirmation unit 710 issues a warning indicating that the identification number recognized this time does not match the identification number for the same inspection object already stored in inspection object storage unit 160 . It is displayed on the screen of the display device 180 . The warning is not limited to that shown in FIG. 18, and may be other text, video, audio, and the like.

As described above, if the inspection support device 700 and the mobile terminal 70 according to the seventh embodiment are used, the identification number recognized in the past and the identification number recognized this time are different for the same inspection object. It is possible to inform the user of an abnormal situation that there is Here, the abnormal situation is, for example, a change in the arrangement of inspection targets in the field equipment, a change in the inspection target nameplate, or the like.

Except for the above, Embodiment 7 is the same as any of Embodiments 1 to 6.

10, 20, 30, 40, 50, 60, 70 mobile terminal, 100, 200, 300, 400, 500, 600, 700 inspection support device, 10a mobile terminal, 100a inspection support device, 110 camera, 120 terminal position and orientation estimation 130 Inspection object detection unit 140 Inspection object tracking unit 150 Inspection object position estimation unit 160 Inspection

object storage unit

170, 670 Display control unit 180 Display device 210 Inertial sensor 220

Character recognition unit

230, 230a Recognition target acquisition unit 410 Terminal position/orientation storage unit 420 Terminal position/orientation return unit 510 Inspection state initialization unit 610 Inspection procedure storage unit 620 Inspection procedure management unit 710 Identification number confirmation unit 800 User 810 Piping, 821 to 825 valves (to be inspected), 821a to 825a nameplates, 821b to 825b inspection target areas, 821c identification numbers, 830 tanks.

Claims

The position and orientation of the mobile terminal in a three-dimensional coordinate system based on at least one of a camera image captured by a camera provided in the mobile terminal and a detection signal obtained by an inertial sensor provided in the mobile terminal. a terminal position/orientation estimation unit for estimating
an inspection target detection unit that detects an inspection target from a camera image captured by the camera;
an inspection target tracking unit that tracks the inspection target area indicating the inspection target detected in the camera image in chronological order of photographing time and identifies the same inspection target area in a plurality of camera images arranged in time series;
An inspection target for estimating coordinates of the inspection target area in the three-dimensional coordinate system based on the position and orientation of the mobile terminal and the inspection target area in at least two camera images out of the plurality of camera images. a position estimator;
a display control unit that causes a display device to display a display component that makes the inspection target area visible;
An inspection support device comprising:
2. The inspection support apparatus according to claim 1, wherein the inspection target position estimating unit associates the position and orientation of the mobile terminal with the coordinates of the inspection target area and stores them in an inspection target storage unit.
The inspection support device according to claim 1 or 2, wherein the display control unit causes the display device to display the inspection object that has been inspected and the inspection object that has not been inspected using different display parts.
The inspection target position estimating unit calculates, based on the position and orientation of the mobile terminal and an inspection target region in at least two camera images among the plurality of camera images, a three-dimensional coordinate system of the inspection target region. The inspection support device according to claim 1, wherein the coordinates are estimated.
a character recognition unit that recognizes characters on a nameplate provided for the inspection target;
a recognition target acquisition unit that associates the inspection target with the recognized character;
further having
The inspection support device according to claim 4, wherein the display control unit causes the display device to display the display component and the recognized characters.
further comprising a terminal position/orientation restoring unit;
The terminal position/orientation estimation unit stores the camera image and the position/orientation of the mobile terminal in a terminal position/orientation storage unit in association with each other;
The terminal position/orientation restoring unit sets the position/orientation associated with the camera image closest to the current camera image among the camera images stored in the terminal position/orientation estimation unit as the current position/orientation of the terminal. The inspection support device according to any one of claims 1 to 5, wherein the information is provided to a position/orientation estimation unit.
3. The inspection support device according to claim 2, further comprising an inspection state initialization unit that initializes the inspection state of the inspection object stored in the inspection object storage unit to uninspected.
a character recognition unit that recognizes characters on a nameplate provided for the inspection target;
an inspection procedure management unit that causes the display device to display the next inspection object based on the order of inspection of the inspection object and the recognized characters stored in the inspection procedure storage unit;
The inspection support device according to any one of claims 1 to 3, further comprising:
Identification number confirmation unit for displaying a warning on the screen of the display device when the identification number recognized this time by the character recognition unit does not match the identification number for the same inspection object already stored in the inspection object storage unit The inspection support device according to claim 4 or 5, further comprising:
The display device has a screen for displaying an image,
The inspection support device according to any one of claims 1 to 9, wherein the display control unit causes the screen to display the camera image and the display component overlapping the camera image.
The display device has a transparent or translucent screen for displaying an image,
10. The inspection support device according to any one of claims 1 to 9, wherein the display control unit causes the screen to display the display component that overlaps a real landscape viewed through the screen. .
The display device is a projector,
The inspection support device according to any one of claims 1 to 9, wherein the display control unit causes the projector to project the display component onto actual equipment.
The inspection support device according to any one of claims 1 to 12;
the camera;
the inertial sensor;
An inspection support system comprising: the display device;
An inspection support method executed by an inspection support device,
The position and orientation of the mobile terminal in a three-dimensional coordinate system based on at least one of a camera image captured by a camera provided in the mobile terminal and a detection signal obtained by an inertial sensor provided in the mobile terminal. estimating the
a step of detecting an inspection target from a camera image taken by the camera;
a step of tracking the inspection target area indicating the inspection target detected in the camera image in chronological order of photographing time, and identifying the same inspection target area in a plurality of camera images arranged in time series;
estimating coordinates of the inspection target area in the three-dimensional coordinate system based on the position and orientation of the mobile terminal and the inspection target area in at least two camera images out of the plurality of camera images; ,
a step of displaying on a display device a display component that makes the inspection target area visible;
An inspection support method characterized by having
The position and orientation of the mobile terminal in a three-dimensional coordinate system based on at least one of a camera image captured by a camera provided in the mobile terminal and a detection signal obtained by an inertial sensor provided in the mobile terminal. estimating the
a step of detecting an inspection target from a camera image taken by the camera;
a step of tracking the inspection target area indicating the inspection target detected in the camera image in chronological order of photographing time, and identifying the same inspection target area in a plurality of camera images arranged in time series;
estimating coordinates of the inspection target area in the three-dimensional coordinate system based on the position and orientation of the mobile terminal and the inspection target area in at least two camera images out of the plurality of camera images; ,
a step of displaying on a display device a display component that makes the inspection target area visible;
An inspection support program characterized by causing a computer to execute