JP2023077282A - Inspection system, inspection method and program - Google Patents

Abstract

To provide an inspection system, an inspection method and a program which can more correctly set an inspection range in a photographed image.SOLUTION: An inspection system 1 comprises: an image acquisition unit 1b; a display control unit 1c; an inspection range setting unit 1e; and an inspection unit 1h. The image acquisition unit 1b acquires data of a photographed image that is the image obtained by imaging an inspection object 9 in a three-dimensional shape and includes three-dimensional position information. The display control unit 1c displays the photographed image on a screen D1. The inspection range setting unit 1e sets an inspection range indicating a range in which the inspection object 9 is captured in the photographed image by the operation of a person. The inspection unit 1h inspects the inspection object 9 by performing image recognition processing on the inspection range.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to inspection systems, inspection methods, and programs.

In Patent Document 1, at a construction site of a reinforced concrete structure, before the arranged reinforcing bars are embedded in the poured concrete, the arranged reinforcing bars are confirmed to be correctly arranged. The diameter, pitch, number, etc. are inspected.

Specifically, the bar arrangement inspection device of Patent Literature 1 includes a stereo imaging unit, an array plane specifying unit, and an inspection unit. The stereo imaging unit acquires position information of the plurality of reinforcing bars in the three-dimensional space by imaging the plurality of reinforcing bars arranged on a plane in the three-dimensional space from two different points. The arrangement plane specifying unit specifies a plane on which the plurality of reinforcing bars are arranged based on the positional information of the reinforcing bars acquired by the stereo imaging unit. The inspection unit inspects a plurality of reinforcing bars arranged on the plane specified by the arrangement plane specifying unit as objects to be inspected.

JP 2019-2737 A

If a plurality of members are arranged in a three-dimensional space, an image (captured image) captured by a stereo imaging unit (imaging unit) may include members other than the member to be inspected. At this time, the range (inspection range) in which the member to be inspected is shown in the captured image cannot be distinguished from the range in which other members are shown. As a result, it was difficult to set the inspection range accurately.

An object of the present disclosure is to provide an inspection system, an inspection method, and a program capable of setting an inspection range in a captured image more accurately.

An inspection system according to an aspect of the present disclosure includes an image acquisition unit, a display control unit, an inspection range setting unit, and an inspection unit. The image acquisition unit acquires captured image data including three-dimensional position information, which is an image of a three-dimensional inspection target. The display control unit displays the captured image on a screen. The inspection range setting unit sets an inspection range indicating a range in which the inspection target is shown in the captured image by a human operation. The inspection unit inspects the inspection object by performing image recognition processing on the inspection range.

An inspection method according to an aspect of the present disclosure includes an image acquisition step, a display step, an inspection range setting step, and an inspection step. The image acquisition step acquires data of a captured image, which is an image of a three-dimensional inspection target and includes three-dimensional position information. The display step displays the captured image on a screen. In the inspection range setting step, an inspection range indicating a range in which the inspection target is shown in the captured image is set by a human operation. The inspection step inspects the inspection target by subjecting the inspection range to image recognition processing.

A program according to an aspect of the present disclosure causes a computer system to execute the inspection method described above.

As described above, the present disclosure has the effect of being able to set the inspection range in the captured image more accurately.

FIG. 1 is a block diagram showing the configuration of the inspection system of the embodiment. FIG. 2 is a perspective view showing a usage example of the same inspection system. FIG. 3 is a plan view showing an imaging unit of a tablet terminal that constitutes the same inspection system. FIG. 4 is a diagram showing an image captured by the inspection system of the same. FIG. 5 is a diagram in which an inspection target is specified in the captured image of the inspection system. FIG. 6 is a diagram in which an inspection range is set in an image picked up by the inspection system. FIG. 7 is a perspective view showing a three-dimensional model of the same inspection system. FIG. 8 is a diagram in which a marker image is set in the captured image of the inspection system. FIG. 9 is a flow chart showing an inspection method of the inspection system same as above. FIG. 10 is a diagram in which an inspection target is specified in the captured image of the inspection system of the second modified example. FIG. 11 is a diagram in which an inspection range is set in an image picked up by the inspection system. FIG. 12 is a perspective view showing a three-dimensional model of the same inspection system. FIG. 13 is a diagram in which a marker image is set in an image captured by the inspection system. 14A and 14B are diagrams in which an inspection target is specified in the captured image of the inspection system of the third modification and the fourth modification. FIG. 15 is a block diagram showing the configuration of the inspection system of the fifth modified example. FIG. 16 is a block diagram showing the configuration of another inspection system of the fifth modified example.

The following embodiments generally relate to inspection systems, inspection methods, and programs. More specifically, the following embodiments relate to an inspection system, an inspection method, and a program for inspecting an inspection object by subjecting a captured image to image recognition processing.

Hereinafter, an inspection system, an inspection method, and a program according to embodiments will be described in detail with reference to the drawings. However, each drawing described in the following embodiments is a schematic drawing, and the ratio of the size and thickness of each component does not necessarily reflect the actual dimensional ratio.

Also, the embodiments described below are merely examples of the embodiments of the present disclosure. The present disclosure is not limited to the following embodiments, and various modifications can be made according to design and the like as long as the effects of the present disclosure can be achieved.

(1) Overview In building construction, various inspections are performed in each process. For example, the placement, shape, size, and the like of the inspection target are inspected for members such as reinforcing bars, wallpaper, wiring devices, wiring, and piping. Also, the structure, layout, shape, and the like of the inspection target are inspected using the building itself as the inspection target. In such an inspection, inspection items such as the arrangement, shape, size, and structure of the inspection object may be evaluated by performing image recognition processing on a captured image of the inspection object. Note that the building work may be any work related to building construction, and includes, for example, construction work, electrical work, water supply and drainage work, and civil engineering work.

In recent years, it has become easier to generate a captured image including three-dimensional position information instead of a two-dimensional captured image. Therefore, the inspection system of the present embodiment shown in FIG. 1 inspects the inspection object 9 using a captured image including three-dimensional position information.

The inspection system 1 includes an image acquisition unit 1b, a display control unit 1c, an inspection range setting unit 1e, and an inspection unit 1h. The image acquisition unit 1b acquires data of a captured image, which is an image of a three-dimensional inspection target 9 and includes three-dimensional position information. The display control unit 1c displays the captured image on the screen D1. The inspection range setting unit 1e sets an inspection range indicating the range in which the inspection target 9 is shown in the captured image by a human operation. The inspection unit 1h inspects the inspection object 9 by performing image recognition processing on the inspection range.

Therefore, since the inspection system 1 sets the inspection range in the captured image by human operation, it is possible to suppress the inclusion of structures other than the inspection target in the inspection range, and to set the inspection range in the captured image more accurately.

(2) Details Details of the inspection system 1 of the present embodiment will be described below.

(2.1) Objects to be inspected Objects 9 to be inspected in this embodiment are reinforcing bars such as columns, beams, walls, slabs, and foundations of a building. FIG. 2 shows reinforcement arrangements 91 and 92 . The reinforcing bars 91 and 92 exist in the three-dimensional space W1 inside the building under construction. Reinforcements 91 and 92 are beam reinforcements. The outline of the bar arrangement 91 and the outline of the bar arrangement 92 are rectangular bodies.

Each of the reinforcing bars 91 and 92 is composed of a plurality of reinforcing bars 9a arranged in a lattice, and is a member having a three-dimensional shape. In the reinforcement arrangement inspection, it is individually determined whether the arrangement and arrangement of the plurality of reinforcement bars 9a, the size of the reinforcement bars 9a, and the like satisfy predetermined specifications for each of the reinforcement arrangements 91 and 92. .

In addition, in this embodiment, it is assumed that the rectangular contour of the bar arrangement is composed of eight sides extending in the vertical direction and the horizontal direction in the three-dimensional space W1.

(2.2) Inspection system The inspection system 1 preferably comprises a computer system. The computer system implements part or all of the inspection system 1 by executing programs. A computer system has a processor that operates according to a program as a main hardware configuration. Any type of processor can be used as long as it can implement functions by executing a program. The processor is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (large scale integration). Here, they are called ICs and LSIs, but they may be called system LSIs, VLSIs (very large scale integration), or ULSIs (ultra large scale integration) depending on the degree of integration. A field programmable gate array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device capable of reconfiguring the junction relationships inside the LSI or setting up circuit partitions inside the LSI for the same purpose. can be done. A plurality of electronic circuits may be integrated on one chip or may be provided on a plurality of chips. A plurality of chips may be integrated into one device, or may be provided in a plurality of devices. The program is recorded in a non-temporary recording medium such as a ROM, optical disk, hard disk drive, etc. that can be read by the computer system. The program may be pre-stored in a non-temporary recording medium, or may be supplied to the non-temporary recording medium via a wide area network including the Internet.

The computer system is not limited to one computer device, and may be realized by a plurality of computer devices that cooperate with each other. Also, the computer system may be constructed as a cloud computing system.

As shown in FIG. 1, the inspection system 1 of this embodiment includes an imaging unit 1a, an image acquisition unit 1b, a display control unit 1c, an operation unit 1d, an inspection range setting unit 1e, a marker generation unit 1f, a marker processing unit 1g, It has an inspection unit 1h and a communication unit 1i. Moreover, it is preferable that the inspection system 1 further includes a screen D1. As shown in FIGS. 1 and 2, the inspection system 1 of the present embodiment is an inspection apparatus including a tablet terminal T1 that can be carried by a worker H1. A worker H1 carries a tablet terminal T1 and moves in the three-dimensional space W1. The inspection system 1 inspects the bar arrangement to be the inspection target 9 by the operator H1 capturing an image of the bar arrangement to be the inspection target 9 using the tablet terminal T1. Note that, in the present embodiment, the operator H1 corresponds to the person who designates the inspection range within the captured image.

(2.2.1) Imaging Unit The imaging unit 1a is a trinocular stereo camera having three lenses 101, as shown in FIG. The imaging unit 1a generates captured image data including three-dimensional position information based on image data captured by each of the three lenses 101 . The three-dimensional position information is, for example, three-dimensional coordinates (for example, world coordinates) of an object included in the imaging range of the imaging unit 1a. That is, the imaging unit 1a generates three-dimensional image data as captured image data.

The image capturing unit 1a captures an image of a three-dimensional space W1 including a three-dimensional inspection object 9, and generates captured image data. In this embodiment, the reinforcement 91 among the reinforcements 91 and 92 is the inspection object 9 . Therefore, the worker H1 operates the tablet terminal T1 so that the bar arrangement 91 appears in the captured image. However, not only the reinforcement 91 but also the reinforcement 92 may appear in the captured image depending on the arrangement of the reinforcements 91 and 92 and the position of the worker H1.

Then, the imaging unit 1a outputs the data of the captured image as real-time moving image data. That is, the image capturing unit 1a generates data of a live view image captured by the movable image capturing unit 1a as captured image data.

(2.2.2) Image Acquisition Unit The image acquisition unit 1b acquires live view image data (captured image data) from the imaging unit 1a.

In the present embodiment, the imaging unit 1a and the image acquisition unit 1b are provided in the same tablet terminal T1, but the imaging unit 1a may be provided separately from the tablet terminal T1. good.

(2.2.3) Display Control Unit The display control unit 1c displays the captured image acquired by the image acquisition unit 1b on the screen D1. The screen D1 is, for example, a liquid crystal display or an organic EL display. The screen D1 of this embodiment is a display device provided in the tablet terminal T1.

Then, the display control unit 1c receives the data of the live view image acquired by the image acquisition unit 1b, and displays the live view image on the screen D1.

The worker H1 carrying the tablet terminal T1 can confirm the captured image by looking at the screen D1. Since the captured image in this embodiment is a live view image, the worker H1 can monitor the captured image in real time.

FIG. 4 shows the captured image G1 displayed on the screen D1. In the captured image G1, not only the bar arrangement 91 that is the inspection target 9 but also the bar arrangement 92 that is not the inspection target 9 are shown. In the captured image G1, Q91 is the region where the bar arrangement 91 is shown, and Q92 is the region where the bar arrangement 92 is shown. The data of the captured image G1 includes three-dimensional positional information of the area Q91 and three-dimensional positional information of the area Q92.

(2.2.4) Operation Unit The operation unit 1d is a touch sensor placed over the screen D1, and the screen D1 and the operation unit 1d constitute a touch panel. The operator H1 can specify an arbitrary position in the captured image G1 on the screen D1 by touching the operation unit 1d, which is a touch sensor, with a finger or a touch pen. An arbitrary position in the captured image G1 is, for example, an arbitrary point, line, range, or the like in the captured image G1.

For example, the worker H1 designates at least one point in the area Q91 where the reinforcement arrangement 91 is shown in the captured image G1, or at least one point in the vicinity of the area Q91.

In this embodiment, the outline of the bar arrangement 91 is a rectangular body having six faces. Then, the worker H1 takes an image of the bar arrangement 91 from an oblique angle so that at least one of the six planes forming the rectangular contour of the bar arrangement 91 is captured in the region Q91 (in FIG. 4, three The bar arrangement 91 is imaged obliquely so that two planes are captured). Then, as shown in FIG. 5, the worker H1 places four points corresponding to the four vertices of the outline of the rectangular inspection range R1 (see FIG. 6) on the area Q91 where the reinforcement arrangement 91 is shown. Designate two points P1-P4. A point P1 corresponds to a vertex common to three mutually orthogonal surfaces of the rectangular outline. Each of points P2-P4 corresponds to each end of three sides extending from a vertex common to three mutually orthogonal faces of the rectangular body. That is, the operator H1 designates four points P1 to P4 in the captured image G1, so that, of the six rectangular planes forming the outline of the inspection range R1 (see FIG. 6), the three planes perpendicular to each other. A region corresponding to the plane can be specified in the captured image G1.

In FIG. 5, by designating four points P1-P4 for an area Q91 in which the entire bar arrangement 91 is shown, the entire bar arrangement 91 is shown in the inspection range R1 (see FIG. 6). However, the entire bar arrangement 91 may not be shown in the area Q91, and a part of the bar arrangement 91 may be shown. Even in such a case, as described above, by designating four points P1 to P4 respectively corresponding to the four vertices of the outline of the rectangular inspection range R1 (see FIG. 6) for the area Q91, , a range in which a part of the bar arrangement 91 is shown can be set as an inspection range.

Note that the operation unit 1d may be a mouse, a keyboard, or the like.

(2.2.5) Inspection Range Setting Section The inspection range setting section 1e sets an inspection range R1 shown in FIG. 6 within the captured image G1 based on the position designated by the operator H1 within the captured image G1. The inspection range R1 includes a region Q91 in which the reinforcement arrangement 91 to be inspected is shown in the captured image G1.

The inspection range setting unit 1e sets an inspection range R1 in the captured image G1 based on the four points P1 to P4 set in the captured image G1.

Specifically, the inspection range setting unit 1e recognizes the vertical direction and horizontal direction in the captured image G1 based on the detection results of a motion sensor such as an acceleration sensor, a gyro sensor, or an inclination sensor built in the tablet terminal T1. can. In addition, the inspection range setting unit 1e can recognize the position and orientation of the tablet terminal T1 using at least one (or both) of the detection result of the motion sensor and the change in the captured image G1 (difference between image frames). . Therefore, the inspection range setting unit 1e sets, as an inspection range R1 (see FIG. 6), an imaging area obtained by imaging a rectangular object in which the four points P1 to P4 correspond to the four vertices in the captured image G1. In this embodiment, it is assumed that the rectangular contour of the bar arrangement is composed of eight sides extending in the vertical direction and the horizontal direction in the three-dimensional space W1. Under this premise, an imaging area is estimated in which a rectangular body having four points P1 to P4 as its vertices is imaged.

Therefore, the inspection system 1 can distinguish the area Q91 in which the reinforcement arrangement 91 is shown from the area Q92 in which the reinforcement arrangement 92 is shown, and set the area Q91 as the inspection range R1 without including the area Q92 in the inspection range. can. That is, the inspection system 1 can more accurately set the inspection range R1 in the captured image G1.

(2.2.6) Marker Generation Unit The marker generation unit 1f generates a marker image M1 (see FIG. 8) indicating the inspection range R1 in the captured image G1.

In this embodiment, the marker generator 1f generates a rectangular three-dimensional model K1 (see FIG. 7) including three mutually orthogonal surfaces with four points P1 to P4 as vertices. From the symmetry of the rectangular body, if three of the six faces that make up the outer surface of the rectangular body are known, it is possible to estimate the remaining three faces that make up the outer surface of the rectangular body. be. Therefore, the marker generator 1f generates a rectangular three-dimensional model K1 in which the four points P1 to P4 correspond to the four vertices. That is, the marker generation unit 1f sets the shape of the three-dimensional model K1 to a rectangular shape including three mutually orthogonal planes specified by the four points P1 to P4 in the captured image G1.

Then, the marker generator 1f generates a marker image M1 shown in FIG. 8 by perspectively projecting the three-dimensional model K1 onto the captured image G1. The marker image M1 is generated in a region obtained by perspectively projecting the three-dimensional model K1 in the captured image G1, and becomes a solid figure image displaying the three-dimensional model K1 in the captured image G1. As a result, the display control unit 1c superimposes the marker image M1 on the captured image G1 and displays it on the screen D1, as shown in FIG. The marker image M1 is an image having colors, patterns, and patterns so that it can be distinguished from the captured image G1. The range of the marker image M1 in the captured image G1 is the same as the inspection range R1. That is, the range of the marker image M1 with respect to the captured image G1 includes the area Q91 where the bar arrangement 91 is shown.

Therefore, the operator H1 can confirm the inspection range R1 in the captured image G1 by viewing the marker image M1 in the captured image G1 displayed on the screen D1. As a result, the inspection system 1 can more accurately set the inspection range R1 in the captured image G1.

(2.2.7) Marker Processing Section After setting the inspection range R1 and the marker image M1 in the captured image G1 as described above, the inspection system 1 links the subsequent captured image G1 and the marker image M1. That is, the range of the marker image M1 can be regarded as the inspection range R1, and the inspection range R1 follows the change of the marker image M1 and changes in the same manner as the marker image M1.

In this embodiment, the marker processing unit 1g performs at least one process of movement, rotation, enlargement, and reduction on the marker image M1 displayed on the screen D1.

In this embodiment, the marker processing unit 1g moves, rotates, enlarges, and reduces the marker image M1 displayed on the screen D1 in accordance with the operation of the operator H1 on the operation unit 1d. Apply.

Specifically, the worker H1 operates the operation unit 1d, which is a touch sensor overlapping the screen D1, with a finger or a touch pen. The operator H1 moves or rotates the marker image M1 by performing a slide operation on the marker image M1 displayed on the screen D1. The operator H1 enlarges the marker image M1 by performing a pinch-out operation on the marker image M1 displayed on the screen D1. The worker H1 reduces the size of the marker image M1 by performing a pinch-in operation on the marker image M1 displayed on the screen D1.

The marker processing unit 1g reflects each process of moving, rotating, enlarging, and reducing the marker image M1 on the three-dimensional model K1, thereby moving the three-dimensional model K1 with respect to the projection plane, and rotating and enlarging the three-dimensional model K1. , and reduction are performed.

The inspection range setting unit 1e performs each correction of movement, rotation, enlargement, and reduction on the inspection range R1 in the same manner as the processing for the marker image M1, in accordance with each process of movement, rotation, enlargement, and reduction of the marker image M1. process. That is, the inspection range setting unit 1e corrects the inspection range R1 according to the processing performed on the marker image M1.

Therefore, the inspection system 1 can correct the inspection range R1 by correcting the display area of the marker image M1. As a result, the inspection system 1 can more accurately set the inspection range R1 in the captured image G1.

(2.2.8) Inspection Unit The inspection unit 1h inspects the bar arrangement 91, which is the inspection target 9, by performing image recognition processing on the inspection range R1 of the captured image G1.

Specifically, the inspection unit 1h extracts the feature amount of the inspection range R1 by performing binarization processing, filtering processing, edge extraction processing, and the like on the inspection range R1 of the captured image G1. The inspection unit 1h also acquires design information from the design information storage unit 2a of the server device 2 via the communication unit 1i. The design information includes, as design specifications of the bar arrangement 91, information such as the arrangement and arrangement of the plurality of reinforcing bars 9a and the size of the reinforcing bars 9a. Then, the inspection unit 1h determines whether or not the bar arrangement 91 satisfies the design specifications by comparing the feature amount of the inspection range R1 with the design information. The inspection unit 1 h uses the determination result as the inspection result of the reinforcement arrangement 91 and generates inspection notification data as image data for notifying the inspection result of the reinforcement arrangement 91 .

The display control unit 1c displays the inspection notification data of the bar arrangement 91 on the screen D1. Therefore, the worker H1 can confirm the inspection result of the bar arrangement 91 by looking at the inspection notification data displayed on the screen D1.

(2.2.9) Communication Unit The communication unit 1i connects to the network NT1 including the Internet, communicates with devices on the network NT1, and has a function of a communication interface for transmitting and receiving signals. A server device 2 is connected to the network NT1, and the communication unit 1i can communicate with the server device 2 via the network NT1.

The server device 2 is managed by, for example, a building design or construction contractor, and includes a design information storage unit 2a. The design information storage unit 2a stores various types of information regarding the specifications, design and construction of buildings. The information stored in the design information storage unit 2a also includes the design information of the bar arrangements 91 and 92. FIG.

Therefore, the inspection system 1 can acquire the design information of the bar arrangements 91 and 92 from the server device 2 via the communication unit 1i.

(2.2.10) Interlocking Captured Image and Marker Image As described above, the inspection system 1 sets the inspection range R1 and the marker image M1 in the captured image G1 of the bar arrangement 91, and conduct an inspection.

After setting the inspection range R1 and the marker image M1 in the captured image G1, the inspection system 1 links the subsequent captured image G1 with the marker image M1 and the inspection range R1. When the worker H1 moves in the three-dimensional space W1 while imaging the reinforcement arrangement 91 with the tablet terminal T1, the captured image G1 displayed on the screen D1 changes, and the area where the reinforcement arrangement 91 is shown in the captured image G1. Q91 also changes. When the tablet terminal T1 moves within the three-dimensional space W1, the positions and shapes of the marker image M1 and the inspection range R1 in the captured image G1 are also changed.

Specifically, the captured image G1 includes three-dimensional position information. Therefore, the marker generation unit 1f uses the three-dimensional positional information in the inspection range R1 (that is, the area Q91) of the captured image G1 to obtain coordinate information indicating the position of the bar arrangement 91 (inspection target 9) in the three-dimensional space W1. , to obtain information on three-dimensional coordinates (for example, world coordinates). In addition, the marker generation unit 1f generates at least one (both may be used), the moving distance and moving direction of the tablet terminal T1 can be detected. Therefore, the marker generation unit 1f corrects the three-dimensional coordinates indicating the position of the bar arrangement 91 in the three-dimensional space W1 based on the moving distance and moving direction of the tablet terminal T1. That is, the marker generation unit 1f can detect the position of the bar arrangement 91 with respect to the imaging unit 1a even if the tablet terminal T1 moves.

Then, the marker generation unit 1f adjusts the projection distance and the projection direction of the three-dimensional model K1 with respect to the captured image G1 based on the three-dimensional coordinates indicating the position of the bar arrangement 91, thereby making the marker image M1 in the captured image G1 The position and shape are made to follow changes in the inspection range R1 in the captured image G1.

Also, the inspection range setting unit 1e changes the inspection range R1 in the same manner as the marker image M1.

Therefore, the inspection system 1 can link the marker image M1 with the captured image G1 even if the captured image G1 is a live view image. Further, the inspection system 1 can link the inspection range R1 with the captured image G1 even if the captured image G1 is a live view image.

(2.2.11) Inspection Method The inspection method by the inspection system 1 described above is summarized in the flow chart of FIG. For example, the inspection method is executed by the computer system included in the inspection system 1 executing a program.

The inspection method includes an image acquisition step S1, a display step S2, an inspection range setting step S3, and an inspection step S5. Moreover, it is preferable that the inspection method further includes a marker image setting step S4.

In the image acquisition step S1, the image acquisition unit 1b acquires data of the captured image G1. The captured image G1 is an image obtained by capturing the three-dimensional inspection object 9, and includes three-dimensional position information.

In the display step S2, the display control unit 1c displays the captured image G1 on the screen D1.

In the inspection range setting step S3, the inspection range setting unit 1e sets an inspection range R1 indicating the range in which the inspection target 9 is shown in the captured image G1 by the operation of the operator H1.

In the marker image setting step S4, the marker generation unit 1f generates a marker image M1 indicating the inspection range R1 in the captured image G1, superimposes the marker image M1 on the captured image G1, and displays it on the screen D1.

In the inspection step S5, the inspection unit 1h inspects the inspection object 9 by performing image recognition processing on the inspection range R1.

The inspection method described above can more accurately set the inspection range R1 in the captured image G1.

(3) First Modification The marker generation unit 1f inputs the captured image G1 to a learning model constructed by machine learning using the captured image and the range of the marker image in the captured image as teacher data, thereby generating the marker image M1. may be acquired from the learning model.

Specifically, a learning model is constructed by learning using a large amount of teacher data. The learning model preferably uses a neural network constructed by machine learning such as deep learning. The learning model may also use other algorithms such as Support Vector Machine.

A machine learning system that constructs a learning model may be either a configuration included in the inspection system 1 or a configuration provided outside the inspection system 1 .

(4) Second Modification The inspection range setting unit 1 e may set the inspection range using information on the dimensions of the inspection object 9 .

Hereinafter, the captured image G11 in FIG. 10 will be used for description.

A captured image G11 displayed on the screen D1 includes areas Q93 and Q94. The reinforcement arrangement shown in the region Q93 is the reinforcement arrangement of the column. The bar arrangement shown in the region Q94 is the base bar arrangement. The outline of the reinforcement of the foundation is formed in the shape of a horizontally extending rectangular plate, and the outline of the reinforcement of the column is formed in the shape of a vertically elongated rectangular body. In this modified example, the worker H1 captures an image of the reinforcement arrangement of the column from the front, with the reinforcement arrangement of the column shown in the area Q93 as the inspection object 9 .

By operating the tablet terminal T1, the worker H1 obtains the bar arrangement of the column corresponding to the area Q93 (the bar arrangement as the inspection target 9) from the design information storage unit 2a of the server device 2 via the communication unit 1i. The design information is previously downloaded to the tablet terminal T1 (previously stored in the tablet terminal T1). The design information includes dimensional information of the bar arrangement as the design specification of the bar arrangement of the column corresponding to the region Q93.

Then, as shown in FIG. 10, the operator H1 designates three points P11-P13 for the area Q93, thereby making one face of the rectangular contour of the bar arrangement of the column for the area Q93. to specify. Points P11-P13 define the front face of the rectangular contour of the reinforcement. That is, the operator H1 designates three points P11 to P13 in the captured image G11, and thereby selects an area corresponding to one of the six rectangular surfaces forming the contour of the reinforcement arrangement in the captured image G11. can be specified within

The inspection range setting unit 1e can recognize the vertical direction and horizontal direction in the captured image G11 based on the detection result of the motion sensor such as an acceleration sensor, a gyro sensor, or an inclination sensor built in the tablet terminal T1. In addition, the inspection range setting unit 1e can recognize the position and orientation of the tablet terminal T1 using at least one (or both) of the detection result of the motion sensor and the change in the captured image G1 (difference between image frames). . Therefore, the inspection range setting unit 1e determines the inspection range R11 (see FIG. 11) by collating the three points P11-P13 specified in the captured image G11 with the design information of the bar arrangement shown in the region Q93. set. This inspection range R11 corresponds to an imaging area obtained by imaging a rectangular object including one plane defined by three points P11-P13.

Therefore, the inspection system 1 distinguishes the area Q93 where the reinforcement arrangement to be inspected is shown from the area Q94 where the reinforcement arrangement which is not the inspection target 9 is shown, and does not include the area Q94 in the inspection range. It can be an inspection range R11. That is, the inspection system 1 can more accurately set the inspection range R11 in the captured image G11.

Moreover, the inspection system 1 can simplify the operation of the operator H1 by using the information on the dimensions of the bar arrangement 93 .

Then, the marker generator 1f generates a marker image M11 (see FIG. 13) indicating the inspection range R11 in the captured image G11.

In the present embodiment, the marker generation unit 1f uses information about the dimensions of the reinforcement arrangement, which is the inspection target 9, to generate a rectangular three-dimensional model K11 (see FIG. 12) including one surface specified in the captured image G11. ). That is, the marker generation unit 1f generates a rectangular three-dimensional model K11 by collating the three points P11 to P13 specified in the captured image G11 with the design information of the bar arrangement that is the inspection target 9. . That is, the marker generator 1f sets the shape of the three-dimensional model K11 to a rectangular shape including one surface designated by the three points P11-P13 in the captured image G11.

Then, the marker generator 1f generates a marker image M11 shown in FIG. 13 by perspectively projecting the three-dimensional model K11 onto the captured image G11. This marker image M11 is generated in a region obtained by perspectively projecting the three-dimensional model K11 in the captured image G11, and becomes a solid figure image displaying the three-dimensional model K11 in the captured image G11. As a result, the display control unit 1c superimposes the marker image M11 on the captured image G11 and displays it on the screen D1, as shown in FIG. The marker image M11 is an image having colors, patterns, and patterns so that it can be distinguished from the captured image G11. The range of the marker image M11 in the captured image G11 is the same as the inspection range R11. That is, the range of the marker image M11 with respect to the captured image G11 includes a region Q93 in which the bar arrangement to be the inspection target 9 is shown.

Therefore, the operator H1 can confirm the inspection range R11 in the captured image G11 by looking at the marker image M11 in the captured image G11 displayed on the screen D1. As a result, the inspection system 1 can more accurately set the inspection range R11 in the captured image G11.

Note that the worker H1 may specify a plurality of surfaces of the rectangular outline of the bar arrangement for the region Q93 of the captured image G11. In this case, the inspection range setting unit 1e compares the plurality of surfaces specified in the captured image G11 with the design information of the bar arrangement (the bar arrangement shown in the area Q93) to be inspected 9, thereby determining the inspection range. Set R11 (see FIG. 11). This inspection range R11 corresponds to an imaging area in which a rectangular body including a plurality of surfaces specified in the captured image G11 is captured. In addition, the marker generation unit 1f sets the shape of the three-dimensional model K11 to a rectangular shape including a plurality of surfaces specified in the captured image G11.

(5) Third Modification As shown in FIG. 14, the worker H1 designates one point P21 in the region Q93 of the captured image G11 displayed on the screen D1, thereby forming a rectangle of bar arrangement. One side of the contour of the shape may be specified for region Q93. The point P21 in this modification defines the upper front side of the rectangular contour of the bar arrangement. That is, the worker H1 can designate one side in the captured image G11 by designating one point P21 in the captured image G11, out of the eight sides forming the contour of the bar arrangement.

In this case, the inspection range setting unit 1e collates one point P21 specified in the captured image G11 with the design information of the bar arrangement (the bar arrangement shown in the region Q93) to be inspected 9, thereby performing the inspection. A range R11 (see FIG. 11) is set. This inspection range R11 corresponds to an imaging area obtained by imaging a rectangular object including one side defined by one point P21.

Then, the marker generator 1f generates a marker image M11 (see FIG. 13) indicating the inspection range R11 in the captured image G11. Here, the marker generation unit 1f sets the shape of the three-dimensional model K11 to a rectangular shape including one side designated by one point P21 in the captured image G11.

Note that the worker H1 may specify a plurality of sides of the rectangular outline of the bar arrangement for the region Q93 of the captured image G11. In this case, the inspection range setting unit 1e compares the plurality of sides specified in the captured image G11 with the design information of the bar arrangement (the bar arrangement shown in the area Q93) to be inspected 9, thereby determining the inspection range. Set R11 (see FIG. 11). This inspection range R11 corresponds to an imaging area obtained by imaging a rectangular body including a plurality of sides specified in the captured image G11. In addition, the marker generation unit 1f sets the shape of the three-dimensional model K11 to a rectangular shape including a plurality of sides designated in the captured image G11.

(6) Fourth Modification As shown in FIG. 14, the worker H1 designates one point P21 in the region Q93 of the captured image G11 displayed on the screen D1, thereby forming a rectangle of bar arrangement. A point on the contour of the shape may be designated for region Q93. The point P31 in this modified example defines one point included in the upper front side of the rectangular contour of the reinforcement arrangement.

In this case, the inspection range setting unit 1e collates one point P21 specified in the captured image G11 with the design information of the bar arrangement (the bar arrangement shown in the region Q93) to be inspected 9, thereby performing the inspection. A range R11 (see FIG. 11) is set. This inspection range R11 corresponds to an imaging area in which a rectangular object including one point P21 is imaged.

Then, the marker generator 1f generates a marker image M11 (see FIG. 13) indicating the inspection range R11 in the captured image G11. Here, the marker generator 1f sets the shape of the three-dimensional model K11 to a rectangular shape including one point P21 in the captured image G11.

Note that the operator H1 may specify a plurality of points on the contour of the rectangular shape of the bar arrangement in the area Q93 of the captured image G11. In this case, the inspection range setting unit 1e compares the plurality of points specified in the captured image G11 with the design information of the bar arrangement that is the inspection target 9 (the bar arrangement shown in the region Q93), thereby determining the inspection range. Set R11 (see FIG. 11). This inspection range R11 corresponds to an imaging area obtained by imaging a rectangular object including a plurality of points specified in the captured image G11. In addition, the marker generation unit 1f sets the shape of the three-dimensional model K11 to a rectangular shape including the plurality of points designated in the captured image G11.

(7) Fifth Modification FIG. 15 shows an inspection system 1A as a modification of the inspection system 1. FIG.

In the inspection system 1A, an inspection section 1h is provided in the server device 2. FIG. The tablet terminal T1 and the server device 2 are configured to be able to communicate with each other via the network NT1, and the inspection unit 1h inspects the inspection object 9 by performing image recognition processing on the inspection range of the captured image. .

FIG. 16 shows an inspection system 1B as another modified example of the inspection system 1. As shown in FIG.

In the inspection system 1B, the server device 2 is also provided with an inspection range setting unit 1e, a marker generation unit 1f, and a marker processing unit 1g in addition to the inspection unit 1h. The tablet terminal T1 and the server device 2 are configured to be able to communicate with each other via the network NT1, and the inspection range setting section 1e, the marker generating section 1f, and the marker processing section 1g also have the same functions as described above.

Each unit constituting the inspection system 1 is not limited to a configuration provided collectively in one apparatus like the inspection system 1, or a configuration provided dispersedly in two apparatuses like the inspection systems 1A and 1B. It may be configured to be distributed in one or more devices.

(8) Sixth Modification The inspection system 1 may not include the screen D1. For example, the screen D1 may be provided by an external terminal such as a tablet terminal, a smart phone, or a personal computer configured to communicate with the inspection system 1 .

The object 9 to be inspected may be a member other than the bar arrangement, such as wallpaper, a wiring device, wiring, or piping. That is, the inspection object 9 is not limited to a specific member, and may be any member having a three-dimensional shape.

Further, the operator H1 may operate the tablet terminal T1 to change the live view image to a still image at a desired timing and set the inspection range for the still image. In this case, while the operator H1 monitors the live view image, when the inspection object 9 is captured in the live view image, the operator H1 switches the live view image to a still image so that the still image showing the inspection object 9 can be displayed. is displayed on the screen D1. Note that both the live view image and the still image correspond to captured image data.

The imaging unit 1a is not limited to a stereo camera having three lenses, and may be a stereo camera having two or four or more lenses. Further, the imaging unit 1a may be a monocular camera, and in this case, it is possible to generate captured image data including three-dimensional position information from a series of captured images. Further, the imaging unit 1a may be configured using a laser such as a three-dimensional laser scanner or LiDAR (Light Detection and Ranging).

The inspection object 9 is not limited to the bar arrangements 91 and 93, and may be the bar arrangements 92 and 94. Also, the inspection object 9 may be something other than the bar arrangement, and may be a member such as wallpaper, a wiring device, wiring, or piping, for example. Also, the inspection object 9 may be a building.

Further, in the above-described embodiment, on the premise that the rectangular contour of the bar arrangement is composed of eight sides extending in the vertical direction and the horizontal direction in the three-dimensional space W1, the inspection range setting unit 1e And the marker generator 1f sets the inspection range under this premise and generates the marker image. However, the premises used by the inspection system 1 may be appropriately set according to the shape, structure, arrangement, and the like of the inspection object 9, and are not limited to specific premises.

(9) Summary The inspection system (1, 1A, 1B) of the first aspect according to the above-described embodiments includes an image acquisition unit (1b), a display control unit (1c), an inspection range setting unit (1e), , and an inspection unit (1h). An image acquisition unit (1b) acquires data of captured images (G1, G11), which are images of a three-dimensional inspection target (9) and include three-dimensional position information. A display control unit (1c) displays the captured images (G1, G11) on the screen (D1). An inspection range setting unit (1e) sets an inspection range (R1, R11) indicating a range in which an inspection target (9) is shown in a captured image (G1, G11) by an operation of a person (H1). The inspection unit (1h) inspects the inspection target (9) by performing image recognition processing on the inspection range (R1, R11).

The inspection system (1, 1A, 1B) described above can more accurately set the inspection range (R1, R11) in the captured image (G1, G11).

In the inspection system (1, 1A, 1B) of the second aspect according to the above-described embodiment, in the first aspect, the inspection range setting unit (1e) sets the captured image (G1 , G11) based on the position specified by the operation of the person (H1).

The inspection system (1, 1A, 1B) described above can more accurately set the inspection range (R1, R11) in the captured image (G1, G11).

In the inspection system (1, 1A, 1B) of the third aspect according to the above-described embodiment, in the second aspect, the inspection range setting unit (1e) sets at least one specified in the captured image (G1, G11) It is preferable to set an inspection range (R1, R11) including three points (P1-P4, P11-P13, P21).

The inspection system (1, 1A, 1B) described above can more accurately set the inspection range (R1, R11) in the captured image (G1, G11).

In the inspection system (1, 1A, 1B) of the fourth aspect according to the above-described embodiment, in the second aspect, the inspection range setting unit (1e) sets at least one specified in the captured image (G1, G11) Preferably, an inspection range (R1, R11) containing two ranges is set.

The inspection system (1, 1A, 1B) described above can more accurately set the inspection range (R1, R11) in the captured image (G1, G11).

In the inspection system (1, 1A, 1B) of the fifth aspect according to the above-described embodiments, in any one of the first to fourth aspects, the inspection range setting unit (1e) includes the inspection object (9) It is preferable to set the inspection range (R11) using information on the dimensions of .

The inspection system (1, 1A, 1B) described above can simplify the operation of the person (H1) by using the information on the dimensions of the inspection object (9).

The inspection system (1, 1A, 1B) of the sixth aspect according to the above-described embodiment is the inspection range (R1, R11) in the captured image (G1, G11) in any one of the first to fifth aspects. ) is preferably further provided with a marker generator (1f) for generating marker images (M1, M11). A display control unit (1c) superimposes the marker images (M1, M11) on the captured images (G1, G11) and displays them on the screen (D1).

In the inspection system (1, 1A, 1B) described above, the person (H1) looks at the marker images (M1, M11) in the captured images (G1, G11) displayed on the screen (D1), thereby The inspection range (R1, R11) within (G1, G11) can be confirmed. As a result, the inspection system (1, 1A, 1B) can more accurately set the inspection range (R1, R11) in the captured image (G1, G11).

In the inspection system (1, 1A, 1B) of the seventh aspect according to the above-described embodiment, in the sixth aspect, the marker generator (1f) is configured to detect three-dimensional positions included in the inspection range (R1, R11). Based on the information, a three-dimensional model (K1, K11) of the examination area (R1, R11) is preferably generated. A marker generator (1f) uses the projected images of the three-dimensional models (K1, K11) as marker images (M1, M11).

In the inspection systems (1, 1A, 1B) described above, the person (H1) can easily confirm the inspection ranges (R1, R11) in the captured images (G1, G11).

In the inspection system (1, 1A, 1B) of the eighth aspect according to the above-described embodiment, in the seventh aspect, the marker generator (1f) converts the shape of the three-dimensional model (K1, K11) into the captured image It is preferable to have a rectangular body shape including three mutually orthogonal faces specified in (G1, G11).

The inspection systems (1, 1A, 1B) described above can readily generate marker images (M1, M11).

In the inspection system (1, 1A, 1B) of the ninth aspect according to the above-described embodiment, in the seventh aspect, the marker generator (1f) uses the information on the dimensions of the inspection object (9) to perform cubic The shape of the original model (K11) is preferably a three-dimensional shape including at least one plane specified in the captured image (G11).

The inspection systems (1, 1A, 1B) described above can readily generate marker images (M1, M11).

In the inspection system (1, 1A, 1B) of the tenth aspect according to the above-described embodiment, in the seventh aspect, the marker generator (1f) uses the information on the dimension of the inspection object (9) to The shape of the original model (K11) is preferably a three-dimensional shape including at least one side specified in the captured image (G11).

The inspection systems (1, 1A, 1B) described above can readily generate marker images (M1, M11).

In the inspection system (1, 1A, 1B) of the eleventh aspect according to the above-described embodiment, in the seventh aspect, the marker generator (1f) uses the dimension information of the inspection object (9) to The shape of the original model (K11) is preferably a three-dimensional shape including at least one point specified in the captured image (G11).

The inspection systems (1, 1A, 1B) described above can readily generate marker images (M1, M11).

In the inspection system (1, 1A, 1B) of the twelfth aspect according to the above-described embodiment, in the sixth aspect, the marker generator (1f) generates the captured images (G1, G11) and the captured images (G1, By inputting the captured images (G1, G11) into a learning model constructed by machine learning using the range of the marker images (M1, M11) in G11) as teacher data, the captured images (M1, M11) are superimposed. Images (G1, G11) are preferably obtained from the learning model.

The inspection systems (1, 1A, 1B) described above can readily generate marker images (M1, M11).

In the inspection system (1, 1A, 1B) of the thirteenth aspect according to the above-described embodiments, in any one of the sixth to twelfth aspects, the marker images (M1, M11) are the three-dimensional model (K1 , K11). A marker generator (1f) calculates three-dimensional coordinates of a three-dimensional model (K1, K11) based on coordinate information indicating the position of the inspection target (9) in a three-dimensional space (W1) where the inspection target (9) exists. to two-dimensional coordinates to display the marker images (M1, M11) on the screen (D1).

The inspection system (1, 1A, 1B) described above can link the marker images (M1, M11) with the captured images (G1, G11).

The inspection system (1, 1A, 1B) of the 14th aspect according to the above-described embodiment, in any one of the 6th to 13th aspects, responds to the operation of the person (H1) on the screen (D1) It is preferable to further include a marker processing unit (1g) that performs at least one process of movement, rotation, enlargement, and reduction on the marker images (M1, M11) displayed in . An inspection range setting unit (1e) corrects the inspection ranges (R1, R11) according to at least one process performed on the marker images (M1, M11).

The inspection systems (1, 1A, 1B) described above can modify the inspection ranges (R1, R11) by modifying the marker images (M1, M11).

In the inspection system (1, 1A, 1B) of the fifteenth aspect according to the above-described embodiments, in any one of the first to fourteenth aspects, the captured image (G1, G11) is a movable imaging unit It is preferably a live view image captured by (1a).

The inspection system (1, 1A, 1B) described above can monitor the captured images (G1, G11) in real time.

In the inspection system (1, 1A, 1B) according to the sixteenth aspect according to the above-described embodiments, in any one of the first to fifteenth aspects, the inspection object (9) includes bar arrangement (91-94) is preferably

The inspection system (1, 1A, 1B) described above is capable of inspecting reinforcement arrangements (91-94).

In the inspection system (1, 1A, 1B) of the seventeenth aspect according to the above-described embodiments, in any one of the first to fifteenth aspects, the inspection object (9) is preferably a building. .

The inspection system (1, 1A, 1B) described above is capable of inspecting buildings.

The inspection method of the eighteenth aspect according to the above embodiment includes an image acquisition step (S1), a display step (S2), an inspection range setting step (S3), and an inspection step (S5). The image acquisition step (S1) acquires data of captured images (G1, G11), which are images of a three-dimensional inspection object (9) and include three-dimensional position information. The display step (S2) displays the captured images (G1, G11) on the screen (D1). In the inspection range setting step (S3), an inspection range (R1, R11) indicating the range in which the inspection target (9) is shown in the captured images (G1, G11) is set by the operation of the person (H1). In the inspection step (S5), the inspection target (9) is inspected by subjecting the inspection range (R1, R11) to image recognition processing.

The inspection method described above can more accurately set the inspection ranges (R1, R11) in the captured images (G1, G11).

The program of the nineteenth aspect according to the above-described embodiments causes the computer system to execute the inspection method of the eighteenth aspect.

The above program can more accurately set the inspection ranges (R1, R11) in the captured images (G1, G11).

1, 1A, 1B inspection system 1a imaging unit 1b image acquisition unit 1c display control unit 1e inspection range setting unit 1f marker generation unit 1g marker processing unit 1h inspection unit 9 inspection object 91-94 bar arrangement D1 screen W1 three-dimensional space H1 work person (person)
G1, G11 Captured images R1, R11 Inspection range M1, M11 Marker images K1, K11 Three-dimensional model P1-P4, P11-P13, P21 Points S1 Image acquisition step S2 Display step S3 Inspection range setting step S5 Inspection step

Claims (19)

an image acquisition unit that acquires data of a captured image that is an image of a three-dimensional inspection object and that includes three-dimensional position information;
a display control unit that displays the captured image on a screen;
an inspection range setting unit that sets an inspection range indicating a range in which the inspection target is shown in the captured image by a human operation;
an inspection unit that inspects the inspection object by performing image recognition processing on the inspection range. The inspection system according to claim 1, wherein the inspection range setting unit sets the inspection range based on a position designated by the person's operation on the captured image displayed on the screen. 3. The inspection system according to claim 2, wherein the inspection range setting unit sets the inspection range including at least one point designated in the captured image. The inspection system according to claim 2, wherein the inspection range setting unit sets the inspection range including at least one range specified in the captured image. The inspection system according to any one of claims 1 to 4, wherein the inspection range setting unit sets the inspection range using information on the dimensions of the inspection object. further comprising a marker generation unit that generates a marker image indicating the inspection range in the captured image;
The inspection system according to any one of claims 1 to 5, wherein the display control unit superimposes the marker image on the captured image and displays it on the screen. The marker generator is
generating a three-dimensional model of the inspection range based on the three-dimensional position information included in the inspection range;
The inspection system according to claim 6, wherein the projected image of the three-dimensional model is used as the marker image. 8. The inspection system according to claim 7, wherein the marker generation unit makes the shape of the three-dimensional model a rectangular body shape including three mutually orthogonal surfaces specified in the captured image. 8. The inspection system according to claim 7, wherein the marker generator uses the dimension information of the inspection object to make the shape of the three-dimensional model a three-dimensional shape including at least one surface specified in the captured image. . 8. The inspection system according to claim 7, wherein the marker generator uses the dimension information of the inspection object to make the shape of the three-dimensional model into a three-dimensional shape including at least one side specified in the captured image. . 8. The inspection system according to claim 7, wherein the marker generator uses the dimension information of the inspection object to make the shape of the three-dimensional model into a three-dimensional shape including at least one point specified in the captured image. . The marker generation unit inputs the captured image to a learning model constructed by machine learning using the captured image and the range of the marker image in the captured image as teacher data, thereby obtaining the marker image superimposed on the marker image. 7. The inspection system of claim 6, wherein captured images are obtained from the learning model. the marker image is a projection image of a three-dimensional model;
The marker generation unit converts the three-dimensional coordinates of the three-dimensional model into two-dimensional coordinates based on coordinate information indicating the position of the inspection object in the three-dimensional space where the inspection object exists, so that 13. The inspection system according to any one of claims 6 to 12, wherein the marker image is displayed. further comprising a marker processing unit that performs at least one process of moving, rotating, enlarging, and reducing the marker image displayed on the screen in accordance with the operation of the person;
The inspection system according to any one of claims 6 to 13, wherein the inspection range setting section modifies the inspection range according to the at least one process performed on the marker image. The inspection system according to any one of claims 1 to 14, wherein the captured image is a live view image captured by a movable imaging unit. 16. The inspection system according to any one of claims 1 to 15, wherein the object to be inspected is bar arrangement. The inspection system according to any one of claims 1 to 15, wherein the inspection object is a building. an image acquisition step of acquiring data of a captured image that is an image of a three-dimensional inspection target and includes three-dimensional position information;
a display step of displaying the captured image on a screen;
an inspection range setting step of setting an inspection range indicating a range in which the inspection target is shown in the captured image by a human operation;
an inspection step of inspecting the inspection target by subjecting the inspection range to image recognition processing. A program that causes a computer system to execute the inspection method according to claim 18.

Images