JP2020057298A - Determination device, determination method, and determination program - Google Patents

Abstract

To allow for automatically and accurately checking whether a device or the like was operated properly.SOLUTION: A determination device 10 provided herein is configured to: identify a predetermined area within a captured image based on a marker in the captured image; identify the attitude of a camera and make ortho-correction of the image of the area based on the identified attitude; and compare the corrected image of the area with a correct image to determine if the appearance of an object in the image of the area is in a predetermined condition. For example, the determination device 10 determines if the appearance of the object matches the way the object should look like if operated correctly.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a determination device, a determination method, and a determination program.

  2. Description of the Related Art Conventionally, a technique for confirming whether or not the work has been correctly performed based on an image of a work product has been known. For example, based on an image of the facility or the like on which the worker has performed the work, the information on the facility or the like stored in the database is specified, and based on the image and the specified information, whether or not the work has been appropriately performed. Is known (for example, see Patent Document 1). In addition, there is known a method of transmitting an image of an inspection target to an external device, receiving a pass / fail result of the inspection determined by the inspector based on the image from the external device, and displaying the result (for example, see Patent Document 2). ).

JP 2001-282349 A International Publication No. WO 2017/033571

  However, the conventional technology has a problem that it is sometimes difficult to automatically and accurately check whether or not the operation of the device or the like has been performed correctly. For example, in the method described in Patent Literature 1, when an image of a facility or the like is inclined or distorted, information in a database may not be able to be specified accurately. Further, for example, in the method of Patent Document 2, the determination of pass / fail of the inspection is not automated.

  The determination device according to the embodiment includes a specifying unit, a correction unit, and a determination unit. The specifying unit specifies a predetermined region in the captured image based on a marker in the captured image. The correction unit corrects the image of the area. The determining unit determines whether the appearance of the object in the image of the area is in a predetermined state based on the image of the area corrected by the correcting unit.

FIG. 1 is a diagram for describing an AR manual according to the first embodiment. FIG. 2 is a diagram illustrating an example of a configuration of the determination device according to the first embodiment. FIG. 3 is a diagram for describing facing correction according to the first embodiment. FIG. 4 is a diagram for explaining the determination according to the first embodiment. FIG. 5 is a diagram illustrating an example of a method of displaying a determination result according to the first embodiment. FIG. 6 is a diagram illustrating an example of an AR manual creation screen according to the first embodiment. FIG. 7 is a flowchart illustrating a flow of a process of the determination device according to the first embodiment.

  Hereinafter, embodiments of a determination device, a determination method, and a determination program according to the present application will be described in detail with reference to the drawings. Note that the present invention is not limited by the embodiments described below.

(First embodiment)
In the first embodiment, the determination device executes an AR (Augmented Reality) manual. Here, the AR manual is a manual for a device or the like provided as AR content. For example, the AR manual superimposes and displays an annotation in which an operation method is described on an image of a device or the like to be operated. Further, the AR manual may use a marker AR or a markerless AR. The AR manual according to the present embodiment can determine whether or not the operation of the device or the like has been correctly completed, and can display the determination result.

  FIG. 1 is a diagram for describing an AR manual according to the first embodiment. The AR manual is provided by the determination device 10. For example, the determination device 10 is a tablet terminal equipped with a camera, a smartphone, or the like.

  As shown in FIG. 1, the user holds the determination device 10 in his hand and turns it toward the device. At this time, the determination device 10 captures an image of the device using a camera or the like. Then, the determination device 10 displays the captured image with the predetermined information superimposed thereon. Note that the determination device 10 displays the captured image in the area 12a.

  Here, the information superimposed and displayed in the AR manual is set according to the progress of the work. For example, at the start of the work, the determination device 10 displays an annotation indicating the operation method and an icon indicating the operation target in an area in the image where the operation target object is displayed. Here, the object is a device itself such as a lever, a switch, and a dial, or a device whose appearance changes according to operation of a device such as a meter or a lamp.

  For example, when the object is an analog meter, the position of the needle changes before and after the operation. When the object is a lamp, the object is turned on or off after the operation. When the object is a lever, the position of the operation unit changes before and after the operation.

  Further, as shown in FIG. 1, when the operation is completed, the determination device 10 displays, in the area 12b, information indicating whether or not the operation has been normally completed, together with an image of the object indicating the operation result. For example, the determination device 10 displays, in the area 121b of FIG. 1, a character string “NG” indicating that the operation related to the meter has not been performed normally, together with the image of the meter shown in the area 121a.

  Similarly, the determination apparatus 10 determines that the operation on each object is normal in the region 122b, the region 123b, the region 124b, and the region 125b along with the image of the object reflected in the region 122a, the region 123a, the region 124a, and the region 125a. A character string indicating whether or not the operation has been performed is displayed. Note that the character string “OK” indicates that the operation regarding the object has been performed normally.

(Configuration of the determination device of the first embodiment)
The configuration of the determination device 10 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a configuration of the determination device according to the first embodiment. As shown in FIG. 2, the determination device 10 includes an input unit 11, a display unit 12, an imaging unit 13, a storage unit 14, and a control unit 15.

  The input unit 11 receives input of data from a user. The display unit 12 displays a screen. For example, the input unit 11 and the display unit 12 are touch panel displays. The imaging unit 13 captures an image. For example, the imaging unit 13 is a camera.

  The storage unit 14 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, a random access memory (RAM), a flash memory, and a non-volatile random access memory (NVSRAM). The storage unit 14 stores an OS (Operating System) and a program executed by the determination device 10. Further, the storage unit 14 stores various information used in executing the program. Further, the storage unit 14 stores the AR information 141 and the determination information 142.

  The AR information 141 is information for executing the AR manual. The AR information 141 includes the image of the marker 120, the relative positional relationship between the marker 120 and the area of each object, and the like. For example, the AR information 141 includes spatial coordinates representing each vertex of the rectangular area of each object.

  For example, the AR information 141 includes a 2D image displayed as an icon in the AR manual, 3D CAD (Computer Aided Design) data, and the like. Further, the AR information 141 may include a message to be displayed as an annotation and content such as a still image and a moving image.

  The determination information 142 is information for determining whether the appearance of each object is in a predetermined state. For example, the determination information 142 is an image prepared in advance as a correct answer image, and is an image of an object immediately after a normal operation. The determination information 142 may be a predetermined feature amount of the correct image.

  The control unit 15 controls the determination device 10. The control unit 15 is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. For example, the control unit 15 functions as the specifying unit 151, the correction unit 152, the determination unit 153, and the display control unit 154.

  The specifying unit 151 specifies a predetermined region in the captured image based on a marker in the captured image. The specifying unit 151 specifies an area in the image captured by the imaging unit 13 that matches the image of the marker 120 acquired from the AR information 141. Further, the specifying unit 151 specifies the posture of the image capturing unit 13, that is, the attitude of the camera when capturing the captured image, based on the size and inclination of the marker 120 in the specified area.

  The correction unit 152 corrects the image of the area. At this time, the correction unit 152 corrects the image of the region directly based on the posture of the camera specified by the specifying unit 151. FIG. 3 is a diagram for describing facing correction according to the first embodiment. FIG. 3 is an example of a case where the correction unit 152 corrects the image of the area 121a directly. The facing correction is a process of converting a captured image of the target region into a facing image. The directly-facing image is an image of the target region when it is assumed that the imaging is performed with the plane defined as the front face facing the imaging surface.

  For example, it is assumed that each of the objects including the meter in the area 121a and the marker 120 are provided on a flat board perpendicular to the ground. The correction unit 152 calculates the angle of each axis between the board and the imaging surface of the image of each area from the posture of the camera specified by the specifying unit 151. Then, the correction unit 152 corrects the face of the image based on the calculated angle. For example, the correction unit 152 maps the image by performing affine transformation to convert the coordinates of each vertex of each region, and performs facing correction.

  The determination unit 153 determines whether the appearance of the object in the image of the region is in a predetermined state, based on the image of the region corrected by the correction unit 152. For example, the determination unit 153 determines the appearance in advance based on whether the degree of matching between the image of the area corrected by the correction unit 152 and the correct image prepared as the determination information 142 is equal to or greater than a predetermined value. It is determined whether or not the state has been performed. The calculation of the matching degree may be performed by a known image recognition processing technique.

  The correct answer image is an image of the object immediately after the operation has been performed normally, and is a directly-acquired image obtained by directly facing the imaging surface with the board provided with each object and the marker 120. For this reason, as shown in FIG. 4, the determination unit 153 can make a determination by comparing the directly-facing images. FIG. 4 is a diagram for explaining the determination according to the first embodiment.

  In the example of FIG. 4, since the position of the hand of the meter is different between the image of the determination target that has been directly corrected and the correct image, the determination unit 153 may determine that the state of the meter is not correct. It is considered possible. On the other hand, if the correct correction of the image is not performed, the positions of the hands of the meter coincide with each other between the image to be determined and the correct image, and the determination unit 153 makes an erroneous determination that the state of the meter is correct. It is possible to do.

  The display control unit 154 causes the display unit 12 to display the determination result on the appearance of the object by the determination unit 153 together with the image of the object. For example, as shown in FIG. 1, the display control unit 154 causes the area 12b to display the determination result together with the image of the object indicating the operation result.

  Also, as shown in FIG. 5, the display control unit 154 may display the determination result for the appearance of the object in the image of the region by the determination unit 153 on the display unit 12 so as to be superimposed on the region of the captured image. Good. FIG. 5 is a diagram illustrating an example of a method of displaying a determination result according to the first embodiment. In the example of FIG. 5, the display control unit 154 directly superimposes and displays the determination result on each area of the captured image of the area 12a.

  Here, a method of creating an AR manual will be described with reference to FIG. FIG. 6 is a diagram illustrating an example of an AR manual creation screen according to the first embodiment. As shown in FIG. 6, the AR manual is generated by the generation device 20. The generation device 20 may be the same device as the determination device 10 or may be a different device.

  As illustrated in FIG. 6, the generation device 20 displays an image of a target device of the AR manual, and receives the setting of the marker 120 and each area. At this time, the generation device 20 accepts the selection of the type of determination by the determination unit 153 for each region.

  The types of determination include meter state determination, switch state determination, color state determination, indicator lamp lighting determination, character recognition, and the like. In accordance with the type of the selected determination, the determination unit 153 performs the determination according to a predetermined determination method. For example, when the determination type is the meter state determination, the determination unit 153 performs the determination by comparing the directly corrected image with the correct image as described above. On the other hand, when the determination type is character recognition, the determination unit 153 determines whether a character string extracted from an image to be determined by a function such as OCR (Optical Character Recognition) matches a predetermined correct character string. judge.

  The AR manual is set for each step. In the example of FIG. 6, the setting of the AR manual of the step “STEP 2” is performed. Here, the display control unit 154 can prevent the AR manual of the next step from being displayed until all the determination results of the determination unit 153 for the region set in each step become normal. In addition, the display control unit 154 may automatically display the AR manual of the next step when all the determination results of the determination unit 153 for the region set in each step become normal.

  Note that the display control unit 154 may display the AR manual of the next step even if the determination results of the determination unit 153 for the area set in each step are not all normal. For example, the display control unit 154 may display the AR manual of the next step according to the operation of the user regardless of the determination result.

  In other words, the identifying unit 151 determines the area that includes the object whose appearance changes due to the operation, which is the area that is associated in advance with the step being executed among the plurality of steps related to the operation of the device, by the determination unit 153. Area. In addition, the determination device 10 displays the information related to the first step by AR, and when accepting the operation of starting the determination, corrects the facing of the determination target area and makes a determination, and determines that all the operations have been completed normally. In this case, information on the second step, which is the next step after the first step, is displayed by AR. If the determination device 10 does not determine that all operations have been completed normally in the first step, the determination device 10 does not display information on the second step. The information on each step is an annotation, an icon, and the like displayed in AR.

(Process of First Embodiment)
The processing of the determination device 10 will be described with reference to FIG. FIG. 7 is a flowchart illustrating a flow of a process of the determination device according to the first embodiment. First, the determination device 10 receives a determination start operation (step S101). The operation of starting the determination may be a tap on a specific icon on the display unit 12 displayed by the AR manual, or an image of the marker 120 using the imaging unit 13.

  Here, the determination device 10 specifies the position of the marker 120 in the captured image and the attitude of the camera (the imaging unit 13) capturing the image (Step S102). Accordingly, the determination device 10 can provide AR content based on the marker 120. Then, the determination device 10 specifies an area to be determined based on the marker 120 (Step S103).

  Next, the determination device 10 performs facing correction of the specified area based on the posture of the camera (step S104). Here, the determination device 10 performs state determination by comparing the correctly corrected image with the correct image (step S105). Then, the determination device 10 displays the determination result (Step S106).

  Here, when it is determined that the operations of all the devices have been correctly completed (Yes at Step S107), the determination device 10 checks whether there is a next step (Step S108). On the other hand, when it is not determined that the operations of all the devices have been correctly completed (No at Step S107), the determining device 10 returns to Step S101 and repeats the processing.

  Further, when there is a next step (Step S108, Yes), the determination device 10 displays the AR manual of the next step, returns to Step S101, and waits for reception of the determination start operation of the next step.

(Effect of First Embodiment)
As described above, the determination device 10 specifies a predetermined area in the captured image based on the marker in the captured image. In addition, the determination device 10 performs the facing correction of the image of the area. In addition, the determination device 10 determines whether the appearance of the object in the image of the region is in a predetermined state based on the image of the corrected region. As described above, the determination device 10 can accurately determine the appearance state of the object in the image by directly correcting the image of the determination target area. Further, the determination device 10 can automatically obtain the distortion and the inclination of the image of the determination target region using the AR, and perform the facing correction. Therefore, according to the present embodiment, it is possible to automatically and accurately check whether or not the operation of the device or the like has been performed correctly.

  The determination device 10 further specifies the attitude of the camera at the time of capturing the captured image. At this time, the determination device 10 corrects the image of the area directly based on the posture of the camera. This makes it possible to perform facing correction using the AR.

  The determination device 10 determines whether or not the appearance is in a predetermined state based on whether or not the degree of matching between the image of the corrected area and a previously prepared correct image is equal to or greater than a predetermined value. I do. As described above, accurate determination can be performed by using the directly corrected image for comparison.

  The determination device 10 displays the determination result for the appearance of the object together with the image of the object. Thereby, the user can easily grasp the determination result for each object.

  The determination device 10 displays the determination result for the appearance of the object in the image of the region in a manner superimposed on the region of the captured image. Thereby, the user can easily specify the location of the object to be determined.

  The specifying unit 151 specifies an area that includes an object whose appearance changes due to the operation, which is an area that is associated in advance with the step being executed among a plurality of steps related to the operation of the device. This makes it possible to clarify the confirmation location in each step.

  The display unit 12 displays information on the first step of the plurality of steps related to the operation of the device by AR, and the determination unit determines in advance the appearance of the object in the area corresponding to the first step in advance. If it is determined that the user is in a state in which the user is in the state, information on the second step, which is the step following the first step, is displayed by AR. Thereby, it is possible to prevent the operation of the next step from proceeding before the previous step is completed.

  The program executed by the determination device 10 of the present embodiment is provided by being incorporated in a ROM or the like in advance. The program executed by the determination device 10 of the present embodiment may be configured to be provided by being recorded in a computer-readable recording medium in an installable or executable file.

  Furthermore, the program executed by the determination device 10 of the present embodiment may be stored on a computer connected to a network such as the Internet, and provided by being downloaded via the network. Further, the program executed by the determination device 10 of the present embodiment may be provided or distributed via a network such as the Internet.

  The program executed by the determination device 10 of the present embodiment has a module configuration including the above-described units (identification unit 151, correction unit 152, determination unit 153, and display control unit 154). When the CPU reads out the program from the ROM and executes the program, the respective units are loaded on the main storage device, and the specifying unit 151, the correction unit 152, the determination unit 153, and the display control unit 154 are generated on the main storage device. It has become.

  Although embodiments of the present invention have been described, the embodiments are presented by way of example and are not intended to limit the scope of the invention. The embodiment can be implemented in other various forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. The embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

Reference Signs List 10 judgment device 11 input unit 12 display unit 12a, 12b, 121a, 122a, 123a, 124a, 125a, 121b, 122b, 123b, 124b, 125b region 13 imaging unit 14 storage unit 15 control unit 20 generation device 120 marker 141 AR information 142 Judgment information 151 Identification unit 152 Correction unit 153 Judgment unit 154 Display control unit

Claims (9)

A specifying unit that specifies a predetermined region in the captured image based on a marker in the captured image,
A correction unit that performs direct correction of the image of the region,
Based on the image of the area corrected by the correction unit, based on the appearance of the object in the image of the area, a determination unit that determines whether or not a predetermined state,
A determination device comprising: The specifying unit further specifies the posture of the camera at the time of capturing the captured image,
The apparatus according to claim 1, wherein the correction unit corrects an image of the area in the area based on the posture.   In the state where the appearance is determined in advance based on whether or not the degree of matching between the image of the area corrected by the correction unit and the previously prepared correct image is equal to or greater than a predetermined value. The determination device according to claim 1, wherein the determination device determines whether or not there is any information.   The said specification part specifies the area | region containing the object whose external appearance changes by the said operation | movement among the area | regions related to the operation currently performed among the several steps regarding operation of apparatus previously. Item 4. The determination device according to any one of Items 1 to 3.   5. The determination device according to claim 1, further comprising a display unit that displays a determination result on the appearance of the object by the determination unit together with an image of the object.   The determination device according to claim 5, wherein the display unit displays a determination result of the appearance of an object in the image of the region by the determination unit in a manner superimposed on the region of the captured image. .   The display unit displays, by AR, information on a first step of a plurality of steps relating to device operation, and the determination unit determines in advance the appearance of an object in an area corresponding to the first step in advance. 7. The determination device according to claim 5, wherein when it is determined that the state is a predetermined state, information on a second step that is a step next to the first step is displayed by AR. 8. A determination method performed by a computer,
A specifying step of specifying a predetermined region in the captured image based on a marker in the captured image,
A correction step of performing a direct correction of the image of the region,
Based on the image of the area corrected by the correction step, based on the appearance of the object in the image of the area, a determination step of determining whether or not in a predetermined state,
A determination method characterized by including: Computer
Specifying means for specifying a predetermined region in the captured image based on a marker in the captured image;
Correction means for performing direct correction of the image of the area,
Based on the image of the region corrected by the correction unit, a determination unit that determines whether the appearance of an object in the image of the region is in a predetermined state,
A determination program characterized by functioning as: