JP2020057983A - Sensor device, sensor device control method, and program - Google Patents

Abstract

To provide a sensor device that sets a sensor in the same position and in the same direction with respect to a sensing target regardless of timing of acquiring sensor information.SOLUTION: A sensor device includes a camera unit, a camera control unit, a device drive unit, a drive control unit, and a conversion formula calculation unit. The camera unit includes a camera. The camera control unit controls the camera unit to acquire an image. The device drive unit is a mechanism for moving the device itself. The drive control unit controls the device drive unit to move the device itself. The conversion formula calculation unit calculates a conversion formula for converting the viewpoint of the camera when the captured image is acquired to the viewpoint of the camera when the reference image is acquired on the basis of at least the captured image captured by the camera unit and the previously acquired reference image. The drive control unit moves the device from the position and direction of the device with respect to the sensing target when the image is captured to the position and direction when the reference image is captured on the basis of the conversion formula.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a sensor device, a control method for the sensor device, and a program.

  In places such as factories, office buildings, public facilities, infrastructure, and plants, inspections for aging and abnormalities of equipment are performed on a daily basis. Such inspections maintain safety and functions in factories and the like, and enable continuous operation.

  In recent years, efforts have been made to utilize IoT (Internet of Things) technology, and a sensor device having a wireless communication function is installed around a facility to be inspected. The sensor device acquires the sensor information and transmits the sensor information to the server by wireless communication. The server collects sensor information from each sensor device, and the server analyzes the collected sensor information to determine whether the target equipment is normal or abnormal. In this way, the inspection work is automated.

  As described above, while the inspection work is automated, introducing the automatic inspection by the sensor device into the system has the following problems, and these problems are barriers to the introduction of the system.

  The first problem is that the number of necessary sensor devices increases according to the number of facilities to be inspected and the cost increases.

  The second problem is that the sensor device cannot be retrofitted around the equipment to be inspected from the viewpoint of safety such as the possibility of falling off or the restriction of space.

  A third problem is that a power supply for the sensor device cannot be secured.

  The fourth problem is that it is necessary to inspect the sensor device itself installed for inspecting the equipment.

  As a method for solving the above problem, there is a method of moving the sensor device and acquiring information on the equipment to be inspected (see FIG. 17).

  In this document, the method of attaching the sensor device to the equipment to be inspected is referred to as "sensor-fixed type inspection method" or simply "fixed type". On the other hand, a method of moving the sensor device itself is referred to as a "sensor moving type inspection method" or simply as a "moving type".

  The mobile sensor device used in the sensor-moving inspection method has a device drive unit such as a wheel for moving the entire device, and a sensor drive unit such as an arm robot for accurately (accurately) determining a sensor position. .

  Patent Literatures 1 to 3 disclose a technique for a camera that is photographed by a person with a hand, in which a reference image is superimposed and displayed on a live view so that the reference image and the live view are overlapped by a judgment (operation) by a person. A method for moving a camera is disclosed. In other words, Patent Literatures 1 to 3 disclose a technique for easily photographing an object using a camera from the same viewpoint and the same direction as a reference image.

JP 2001-358972 A JP 2005-026872 A JP 2013-192168 A

  The disclosures of the above prior art documents are incorporated herein by reference. The following analysis has been made by the present inventors.

  As described above, in equipment inspection using a sensor, sensor information is repeatedly collected. Specifically, sensor information when the inspection target is in a normal state is collected in advance. The collected sensor information (past acquired information) and the latest information acquired by the sensor device are compared, and by detecting outliers or change points from the past information, the state of the inspection target ( (Normal, abnormal) is determined.

  Therefore, regardless of whether it is a fixed type or a mobile type, the acquisition conditions of the acquired sensor information need to be consistent. Therefore, when the sensor moving type inspection method is adopted, the sensor device needs to always acquire the sensor information at the same position and orientation. More specifically, the sensor device needs to install the sensor at the same position and angle (direction) with respect to the part to be sensed (inspection object) and acquire information from the installed sensor. .

  For example, when an inspection of a machine tool equipped with a motor is performed based on vibration information of equipment, the characteristics (frequency, amplitude) of the vibration differ depending on the position where the sensor is installed, affected by the mechanism of the equipment to be inspected. In addition, since vibration is a physical quantity having a direction, when a microphone is used as a sensor, for example, the characteristics of sensor information obtained according to the angle of the microphone with respect to the vibration source (the attitude of the microphone) change.

  Under the circumstances described above, it is difficult to determine the abnormality of the equipment using the sensor information acquired at different positions and different postures. That is, if the position and angle at which the sensor is set with respect to the sensing target are different each time data is acquired, it becomes difficult to determine the state of the equipment (normal state, abnormal state) using the acquired data.

  In the case of the fixed type, the installation position of the sensor is fixed, so that the sensor information of the same position and orientation can be acquired every time, whereas in the case of the movable type, the sensor device itself moves, so the acquired position and orientation each time the sensor information is acquired May change. Further, such a problem relating to the relationship between the position and orientation of the sensor and data (sensor data) obtained from the sensor may occur not only in equipment inspection but also in the entire monitoring system using a mobile sensor.

  As described above, Patent Literatures 1 to 3 propose techniques for facilitating shooting from the same viewpoint and the same direction as the reference image. However, the method disclosed in these documents is a method for a camera that is held and photographed by a person, and the position of the camera is adjusted by a person's operation. Therefore, according to the methods disclosed in Patent Literatures 1 to 3, the target cannot be photographed from the same viewpoint as the reference image without human operation.

  A main object of the present invention is to provide a sensor device, a control method of a sensor device, and a program that contribute to setting a sensor at the same position and the same direction with respect to a sensing target regardless of the timing of acquiring sensor information. And

  According to a first aspect of the present invention or a disclosure, a camera unit including a camera, a camera control unit that controls the camera unit to acquire an image, a device driving unit that moves the own device, A drive control unit that controls the device drive unit to move the own device, and a viewpoint of the camera when the captured image is acquired based on at least a captured image captured by the camera unit and a previously acquired reference image. A conversion formula calculation unit for calculating a conversion formula for converting the reference image into the viewpoint of the camera when the reference image is obtained, the drive control unit, based on the conversion formula, A sensor device is provided, which moves its own device from the position and direction of the own device with respect to the sensing target when the image was taken to the position and direction when the reference image was taken.

  According to a second aspect of the present invention or a disclosure, in a sensor device including a camera unit including a camera and an apparatus driving unit for moving the own apparatus, at least a captured image captured by the camera unit A step of calculating a conversion formula for converting a viewpoint of the camera when the captured image is obtained into a viewpoint of the camera when the reference image is obtained, based on the obtained reference image; and Moving the subject device from the position and direction of the subject device relative to the sensing target when the photographed image was taken to the position and direction when the reference image was taken. A method is provided.

According to a third aspect of the present invention or a disclosure, a computer mounted on a sensor device including a camera unit including a camera, and a device driving unit for moving the own device, is photographed by at least the camera unit. Processing for calculating a conversion formula for converting a viewpoint of a camera when the captured image is obtained into a viewpoint of the camera when the reference image is obtained, based on the captured image and a previously acquired reference image. And, based on the conversion formula, a process of moving the own device from the position and direction of the own device with respect to the sensing target when the captured image is captured to the position and direction when the reference image is captured. A program to be executed is provided.
This program can be recorded on a computer-readable storage medium. The storage medium can be non-transient, such as a semiconductor memory, hard disk, magnetic recording medium, optical recording medium, and the like. The present invention can be embodied as a computer program product.

  According to the aspects of the present invention and the disclosure, a sensor device, a control method of a sensor device, and a program that contribute to setting a sensor at the same position and the same direction with respect to a sensing target regardless of the timing of acquiring sensor information Is provided.

It is a figure for explaining an outline of one embodiment. FIG. 2 is a diagram illustrating an example of an internal configuration of the mobile sensor device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the mobile sensor device according to the first embodiment. 5 is a flowchart illustrating an example of an operation of the mobile sensor device according to the first embodiment. FIG. 5 is a diagram for explaining an operation of the mobile sensor device according to the first embodiment. It is a figure showing an example of processing composition of a mobile type sensor device concerning a 1st modification. It is a figure showing an example of processing composition of a mobile type sensor device concerning a 2nd modification. It is a figure showing an example of processing composition of a mobile type sensor device concerning a 3rd modification. It is a figure for explaining operation of a mobile type sensor device concerning a 3rd modification. It is a figure showing an example of processing composition of a mobile type sensor device concerning a 4th modification. It is a flow chart which shows an example of processing of a mobile type sensor device concerning a 5th modification. It is a flow chart which shows an example of processing of a mobile type sensor device concerning a 6th modification. It is a flow chart which shows another example of processing of a mobile type sensor device concerning a 6th modification. FIG. 21 is a diagram illustrating an example of a processing configuration of a mobile sensor device according to a ninth modification. FIG. 21 is a diagram illustrating an example of a processing configuration of a mobile sensor device according to a tenth modification. It is a figure showing an example of processing composition of a mobile type sensor device concerning an 11th modification. It is a figure for explaining a mobile type sensor device.

  First, an outline of an embodiment will be described. It should be noted that the reference numerals in the drawings attached to this outline are added for convenience of each element as an example for facilitating understanding, and the description of this outline is not intended to limit the invention in any way. Further, connection lines between blocks in each drawing include both bidirectional and unidirectional. The one-way arrow schematically indicates the flow of a main signal (data), and does not exclude bidirectionality. Further, in a circuit diagram, a block diagram, an internal configuration diagram, a connection diagram, and the like shown in the disclosure of the present application, although not explicitly shown, an input port and an output port exist at an input terminal and an output terminal of each connection line. The same applies to the input / output interface.

  The sensor device 100 according to one embodiment includes a camera unit 101, a camera control unit 102, a device drive unit 103, a drive control unit 104, and a conversion formula calculation unit 105 (see FIG. 1). The camera unit 101 includes a camera. The camera control unit 102 controls the camera unit 101 to acquire an image. The device driving unit 103 is a mechanism (means) for moving the own device. The drive control unit 104 controls the device drive unit 103 to move the own device. The conversion formula calculation unit 105 determines the viewpoint of the camera when the captured image is obtained, based on at least the captured image captured by the camera unit 101 and the reference image that has been obtained in advance, of the camera when the reference image is obtained. Calculate a conversion formula for converting to a viewpoint. Based on the conversion formula, the drive control unit 104 moves the own device from the position and the direction of the own device with respect to the sensing target when the captured image is captured to the position and the direction when the reference image is captured.

  The sensor device 100 is a mobile sensor device that reproduces the position and orientation when sensor information is acquired (when a reference image is acquired). Prior to the use of the sensor device 100, the sensor device 100 determines the position and orientation (as the device) when acquiring sensor information in advance as preparation in advance. Specifically, the sensor device 100 determines the position and orientation at which sensor information is acquired from the sensor (the position and direction of the own device with respect to the sensing target). Further, an image acquired from the camera at the position and orientation is treated as a “reference image”. At the time of normal operation of the sensor device 100 (at the time of inspection of an object), the sensor device 100 collates the captured image obtained from the camera with the reference image, and determines the relationship between the current camera position and attitude and the camera position and attitude at which the reference image is captured. Is calculated (a conversion formula is calculated). The sensor device 100 performs a movement corresponding to the relationship expressed by the above conversion formula by the device driving unit 103 to match the position and orientation of the own device with the state where the reference image is captured. Thereafter, the sensor device 100 reproduces the sensor position and orientation by performing predetermined sensor movement control.

  Hereinafter, specific embodiments will be described in more detail with reference to the drawings. In each embodiment, the same components are denoted by the same reference numerals, and description thereof will be omitted.

[First Embodiment]
The first embodiment will be described in more detail with reference to the drawings.

[Device configuration]
FIG. 2 is a diagram illustrating an example of an internal configuration of the mobile sensor device 10 according to the first embodiment. Referring to FIG. 2, the mobile sensor device 10 includes a camera unit 20, a device driving unit 30, a sensor driving unit 40, a sensor unit 41, an operation unit 50, a display unit 60, a control unit 70, And various storage units. The mobile sensor device 10 includes a reference image storage unit 80, a sensor drive amount storage unit 81, and a camera parameter storage unit 82 as storage units.

  The control unit 70 is connected to various processing modules including the operation unit 50 and is a unit that controls the entire mobile sensor device 10.

  The control unit 70 includes a sub module including a device drive control unit 201, a camera control unit 202, a position and orientation conversion formula calculation unit 203, a sensor drive control unit 204, and a sensor control unit 205.

  The device drive control unit 201 is a unit (processing module) that controls the device drive unit 30 that is a moving mechanism for determining the position and orientation of the mobile sensor device 10 and moves the mobile sensor device 10.

  The device driving unit 30 is a unit (mechanism) for moving the mobile sensor device 10. The device driving unit 30 is, for example, a wheel for traveling on land, a multi-legged mechanism, a caterpillar, a propeller for hovering in the air, a screw for traveling on water, a propeller, and the like.

  The “position and orientation” of the mobile sensor device 10 indicates the position and the direction (angle) of the mobile sensor device 10 with respect to the inspection target equipment (sensing target) and the like. That is, the distance between the inspection target and the mobile sensor device 10 and the direction in which the mobile sensor device 10 faces the inspection target correspond to the position and orientation.

  The camera control unit 202 is a unit for controlling the camera unit 20 to obtain an image (captured image). The camera unit 20 includes, for example, an RGB (Red Green Blue) camera for acquiring a color image, such as a surveillance camera, a Web (Web) camera, a digital still camera, and a digital camcorder. Alternatively, the camera section 20 may include an X-ray camera, an infrared camera, a terahertz imaging camera, a millimeter-wave imaging camera, a microwave imaging camera, and the like.

  The position and orientation conversion formula calculation unit 203 is a means (processing module) for calculating a conversion formula for the captured image and the reference image based on at least the captured image captured by the camera unit 20 and the reference image acquired in advance. Note that the position and orientation conversion formula is a formula for converting two viewpoints when two images are acquired and when the two images are acquired. For example, converting the viewpoint of the camera when the image (captured image) is obtained to the viewpoint of the camera when the reference image is obtained is expressed by a position and orientation conversion formula. Alternatively, the position and orientation conversion formula converts the position and orientation (distance and direction to the target) of the mobile sensor device 10 when a certain image is acquired into the position and orientation of the mobile sensor device 10 when another image is acquired. It can also be regarded as an expression.

  The position and orientation conversion formula calculation unit 203 receives from the camera control unit 202 the reference image stored in the reference image storage unit 80 and an image captured after the own device has moved. The position and orientation conversion formula calculation unit 203 calculates a position and orientation based on a reference image, a plurality of captured images (for example, first and second comparison images described later), movement control information, and camera internal parameter information. Calculate the conversion formula. It should be noted that the movement control information is transmitted from the position and direction in which the device driving unit 30 has acquired one captured image (first comparative image) to the position and direction in which another captured image (second comparative image) has been acquired. Is movement control information relating to the movement of the own device (realizing the movement).

  The device drive control unit 201 controls the device drive unit 30 based on the calculated position and orientation conversion formula so that the position and orientation (distance and direction) of the mobile sensor device 10 match the sensor information acquisition position and orientation when the reference image is captured. Control. That is, based on the position and orientation conversion formula, the device drive control unit 201 sets the own device from the position and direction of the own device with respect to the sensing target when the captured image is captured, to the position and direction when the reference image is captured. Moving.

  The reference image storage unit 80 stores an image (reference image) captured in advance at a position where sensor information is acquired.

  The camera parameter storage unit 82 stores internal parameter information of the camera that has captured the reference image stored in the reference image storage unit 80. For example, the internal parameters of the camera include the focal length of each pixel on the horizontal axis and the vertical axis, the image center coordinates of the camera, the shear coefficient, and the like. When the reference image is captured by a camera different from the camera mounted on the mobile sensor device 10, the camera parameter storage unit 82 stores the camera that captured the reference image and the camera included in the camera unit 20. Each internal parameter information is stored.

  The sensor drive unit 40 is a unit (mechanism) that changes the position of a sensor included in the sensor unit. The sensor drive control unit 204 is a unit (processing module) that controls the operation of the sensor drive unit 40, which is a mechanism for moving the position of the sensor unit 41.

  The sensor driving unit 40 is, for example, an arm robot, a crane, a pan / tilt, a linear guide mechanism, or the like. After the mobile sensor device 10 transitions to the shooting position and orientation state at the time of the reference image under the control of the device drive control unit 201, the sensor drive control unit 204 stores the sensor drive amount information stored in the sensor drive amount storage unit 81 in advance. To control the sensor drive unit 40. The sensor drive control unit 204 moves the sensor unit 41 by the above control (determines the position and orientation of the sensor unit 41).

  The sensor drive amount storage unit 81 stores sensor drive amount information. The sensor drive amount information is information determined according to the distance between the sensor and the target. For example, the distance by which the sensor unit 41 is moved from the initial position of the sensor unit 41 to the position where the sensor unit 41 is operated (the position where sensor information is acquired) corresponds to the sensor drive amount information.

  The sensor control unit 205 is a unit (processing module) that controls the sensor unit 41 and acquires sensor information from the sensor unit 41. The sensor unit 41 includes, for example, an acceleration sensor, a vibration sensor, an angular velocity sensor, a microphone, a radio wave sensor, a voltmeter, an ammeter, a temperature sensor, a humidity sensor, an infrared sensor, a LIDAR (Light Detection and Ranging), a distance meter, a camera, and a depth. Includes sensors used for inspection of cameras and the like.

  The operation unit 50 is a unit that controls an input interface for allowing a person to manually control functions inside the mobile sensor device 10. The input interface is an interface for devices such as a keyboard, a mouse, a switch, and a touch panel.

  The display unit 60 is a unit that controls an output interface for displaying information inside the mobile sensor device 10. The output interface is an interface for a device such as a display and an LED (Light Emitting Diode).

[Hardware configuration]
Subsequently, a hardware configuration of the mobile sensor device 10 will be described.

  FIG. 3 is a diagram illustrating an example of a hardware configuration of the mobile sensor device 10. The mobile sensor device 10 has a configuration illustrated in FIG. For example, the mobile sensor device 10 includes a CPU (Central Processing Unit) 11, a memory 12, an input interface 13, an output interface 14, and the like, which are mutually connected by an internal bus. In FIG. 3, illustration of a sensor, a camera, a driving source for movement (for example, a motor) and the like is omitted.

  However, the configuration shown in FIG. 3 is not intended to limit the hardware configuration of the mobile sensor device 10. The mobile sensor device 10 may include hardware (not shown). The number of CPUs and the like included in the mobile sensor device 10 is not intended to be limited to the example illustrated in FIG. 3. For example, a plurality of CPUs 11 may be included in the mobile sensor device 10.

  The memory 12 is a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a dynamic random access memory (DRAM), and a static random access memory (SRAM).

  The input interface 13 is an interface of an input device (not shown). The input device includes, for example, an operation device. The operation device is, for example, a keyboard or a mouse.

  The output interface 14 is an interface of an output device (not shown). The output device includes, for example, a display device. The display device is, for example, a liquid crystal display or the like.

  The function of the mobile sensor device 10 is realized by the processing module described above. The processing module is realized, for example, by the CPU 11 executing a program stored in the memory 12. Also, the program can be downloaded via a network or updated using a storage medium storing the program. Further, the processing module may be realized by a semiconductor chip. That is, the function performed by the processing module may be realized by any hardware or software executed using the hardware.

[Description of operation]
Next, the operation of the mobile sensor device 10 according to the first embodiment will be described with reference to the drawings. First, shooting of a reference image and storage of camera parameters are performed as advance preparation. This operation is performed by an operator manually operating a mouse or the like.

  In response to an instruction from the user via the operation unit 50, the mobile sensor device 10 controls the device driving unit 30 and the sensor driving unit 40 to move the device itself and the sensor unit 41. Thereafter, the mobile sensor device 10 displays an image photographed by the camera unit 20 and sensor information acquired by the sensor unit 41 on an output interface (display).

  The operator determines the acquisition position and orientation of the sensor information (reference image acquisition position and orientation) while visually confirming (checking) the display. When the acquisition position and orientation of the sensor information are determined, the operator inputs that fact to the mobile sensor device 10 (input by operating the input interface).

  The mobile sensor device 10 captures an image at the acquisition position and orientation of the sensor information, and stores the image in the reference image storage unit 80 as a “reference image”. The movable sensor device 10 stores the driving amount for setting (moving) the sensor unit 41 to the acquisition position of the sensor information in the sensor driving amount storage unit 81 as “sensor driving amount”. Note that the sensor drive amount is stored in association with the reference image. For example, the sensor drive amount and the reference image are associated using the identifier of the reference image.

  The mobile sensor device 10 stores the internal parameter information of the camera (the camera included in the camera unit 20) at the time of capturing the reference image in the camera parameter storage unit 82. The internal parameter information of the camera is a parameter unique to the camera, and is a parameter represented by Expression (3.3) described in Reference Document 1. Reference documents disclosed in the present application are listed at the end of the first embodiment.

  The internal parameter information of the camera uses a value described in the specification of the camera or is calculated by camera calibration described in section 3-2 of Reference 1. As a specific implementation of the calibration, OpenCV CalibrateCamera2 described in Reference Document 2 can be used.

  FIG. 4 is a flowchart illustrating an example of the operation of the mobile sensor device 10 according to the first embodiment. FIG. 4 illustrates a process in which the mobile sensor device 10 reproduces the sensor information acquisition position and orientation (reference image acquisition position and orientation).

  In step S01, the camera control unit 202 controls the camera unit 20 to acquire an image. This image is referred to as “comparison image I1”.

  In step S02, the position / posture conversion formula calculation unit 203 calculates the position / posture conversion formula of the reference image and the comparison image I1 using the comparison image I1, the reference image, and the internal parameters of the camera.

  The calculation of the position and orientation conversion equation can be performed by the method described in Reference 1.

First, the position and orientation conversion formula calculation unit 203 obtains a 3 × 3 basic matrix F from the formula (3.7) described in Reference 1. The basic matrix F can be calculated if there are eight or more combinations of the corresponding points x ₁ and x ₂ between the two images. In addition, the corresponding points x ₁ and x ₂ can be calculated by performing feature point detection as disclosed in References 3 to 12, and then performing a round robin matching and ratio test of Reference 13. .

After that, the position and orientation conversion equation calculation unit 203 acquires an E matrix (Essential matrix; basic matrix) from equation (3 · 12) described in Reference 1. The position / posture conversion equation calculation unit 203 obtains a 3 × 1 translation vector T and a 3 × 3 rotation movement matrix R indicating a position / posture conversion equation between cameras by performing singular value decomposition of the E matrix. Note that A ₁ and A ₂ in the above equation (3 · 12) are internal parameter information of the camera.

  Here, when the position and orientation conversion equation between the cameras is obtained from the correspondence between the two images, there is a problem that the scale becomes indefinite. That is, the scale (magnification) of the obtained translation component is indefinite even if its direction (parallel translation direction) is determined.

In step S02, to obtain a translation vector T ₀₁ and the rotation movement matrix R ₀₁ for transforming a camera coordinate system were taken reference image from the camera coordinate system were taken comparative image I1 (| a _{= 1 |} T 01) .

  As described above, the position and orientation conversion formula calculation unit 203 calculates the position and orientation conversion formula for the reference image and the comparison image I1 based on the reference image, the comparison image I1 (first captured image), and the internal parameter information of the camera.

  The mobile sensor device 10 determines the scale of the translation component by the processing after step S03.

In step S03, the device drive control unit 201 controls the device drive unit 30 to perform a predetermined movement (movement of the mobile sensor device 10). Specifically, the device drive control unit 201 moves the own device from the position and the direction in which the first captured image (comparison image I1) is obtained to the position and the direction in which the next captured image (comparison image I2 described later) is obtained. The own device is moved in accordance with the movement control information for moving. For example, the apparatus drive control unit 201 performs a predetermined movement such as “horizontal movement 1 m to the right”. The movement control information includes at least the rotational movement R _C and the parallel movement T _C.

  In step S04, the camera control unit 202 controls the camera unit 20 to obtain an image. This image is referred to as “comparison image I2”.

  In step S05, the position / posture conversion formula calculation unit 203 uses the comparison images I1 and I2, the reference image, the camera internal parameter information, and the amount of movement (movement control information) in step S03 to generate the reference image and the comparison image. The position and orientation conversion formula of I2 is calculated. The calculation regarding the camera position / posture conversion equation between the two images can be performed in the same manner as in step S02.

  The position and orientation conversion formula calculation unit 203 calculates a position and orientation relational expression of three images as shown in FIG. 5 in order to determine the scale of the translation component.

As shown in FIG. 5, the camera viewpoint of the reference image C _0, the camera viewpoint of the comparison image I1 C _1, the camera viewpoint of the comparison image I2 and C _2. Further, R _{ji is} a rotational movement for converting the camera position and orientation at the camera viewpoint C _{i into} a camera position and orientation at the camera viewpoint C _j , T _{ji is} a translation movement (| T _ji | = 1), and a scale factor of the parallel movement is _Is a _ji . Here, i and j are indexes for specifying the camera viewpoint C.

Further, a conversion formula (position / posture conversion formula) for moving the mobile sensor device 10 from the camera position / posture of the camera viewpoint C _{i to} the camera position / posture of the camera viewpoint C _j is expressed as M _ji . FIG. 5 illustrates three position and orientation conversion equations M ₀₁ , M ₂₁ , and M ₀₂ .

The position and orientation conversion formula calculation unit 203 obtains scale factors a ₂₁ , a ₀₁ , and a ₀₂ by the following formulas (1) to (5).

[Equation 1]

Thus, the position and orientation conversion formula calculating unit 203 obtains a scale factor a ₂₁ from the correspondence relationship between the amount of movement to a position to get the comparison image I2 from the position when acquiring the comparison image I1.

[Equation 2]

[Equation 3]

[Equation 4]

[Equation 5]

The position and orientation conversion equation calculation unit 203 obtains scale factors a ₀₁ and a ₀₂ by solving the simultaneous equations of equations (1) to (5).

Here, when the measurement value includes an error, there is a possibility that a solution cannot be derived simply by simultaneous equations. In such a case, the position and orientation conversion formula calculation unit 203 calculates the scale factors a ₀₁ and a ₀₂ that minimize the residual sum of squares.

Specifically, the position and orientation conversion equation calculation unit 203 obtains scale factors a ₀₁ and a ₀₂ by solving the following equations (6) to (9).

[Equation 6]

Expression (6) is obtained by rewriting expression (5) from the viewpoints of a ₀₁ and a ₀₂ . Therefore, k _ji shown in Expression (6) indicates a coefficient obtained by consolidating each element of T ₀₁ , T ₀₂ , R ₀₂ , and T ₂₁ described in Expression (5).

[Equation 7]

[Equation 8]

[Equation 9]

5, the mobile sensor device 10 when executing the step S05 because the position of the camera viewpoint C _2, the mobile sensor device 10, by controlling the movement corresponding to the position and orientation transformation formula M _02, a camera The viewpoint can be matched (substantially matched) with the reference viewpoint. That is, the position and orientation conversion formula calculated by the position and orientation conversion formula calculation unit 203 includes information on the rotational movement of the own device and information on the parallel movement of the own device. The position and orientation conversion equation _M02 is an equation for converting the position and orientation (camera viewpoint) at which the comparison image I2 was acquired into the position and orientation (camera viewpoint) at which the reference image was acquired. By rotating and translating, the position and orientation at the time of shooting the reference image can be reproduced.

As described above, the scale factor is not determined by comparing the two images. However, movement sensor device 10 can determine the scale factor a ₂₁ translation of the comparative image I2 from the comparison image I1 with the movement control information used at step S03 (see equation (1)) . That is, the position / posture conversion formula calculation unit 203 uses the movement control information to set the scale factor a ₂₁ that defines the parallel movement from the position / posture at the time of acquiring the comparative image I1 to the position / posture at the time of acquiring the comparative image I2. Is calculated. Mobile sensor device 10, while using a scale factor a _21, compares the three images, the rotational movement for moving the camera position and orientation obtained by photographing the reference image from the captured camera position and orientation (current location) of the comparison image I2 R ₀₂ , and a parallel translation a ₀₂ T ₀₂ with a fixed scale are obtained.

  As described above, the position and orientation conversion formula calculation unit 203 calculates the position and orientation conversion formula for the reference image and the comparison image I2 based on the reference image, the comparison image I2, the camera internal parameter information, and the movement control information.

In step S06, the device drive control unit 201 controls the device drive unit 30 to move the mobile sensor device 10. Specifically, the device drive control unit 201 moves its own device to the position and direction at the time when the reference image was captured, based on the position and orientation conversion formula for the reference image and the comparison image I2. More specifically, the device drive control unit 201 instructs the device drive unit 30 to perform a movement corresponding to the rotational movement R ₀₂ and the parallel movement a ₀₂ T ₀₂ . With this control, the mobile sensor device 10 moves to the sensor acquisition position where the reference image was captured.

  In step S07, the sensor drive control unit 204 controls the sensor drive unit 40 based on the sensor drive amount stored in the sensor drive amount storage unit 81 in advance, and sets the sensor unit 41 to the sensor data acquisition position / posture state.

  In step S08, the sensor control unit 205 acquires sensor information from the sensor unit 41.

  As described above, the mobile sensor device 10 according to the first embodiment changes the position and orientation of the mobile sensor device 10 to the position and orientation at the time of capturing the reference image using the reference image and the plurality of captured images. As a result, the position and orientation (distance and direction with respect to the target) of the mobile sensor device 10 when the reference image is obtained (when the sensor information determined by the operator is obtained) is automatically reproduced. Further, problems in the sensor moving type inspection method (it is difficult to make the sensor acquisition position / posture conditions consistent) are solved, which contributes to improvement of the abnormality determination accuracy in monitoring of the mobile sensor device 10.

  In addition, the estimation of the camera position / posture relationship using a camera has a problem that the scale is not determined. In the first embodiment, by providing a function of moving the sensor, the first embodiment solves both the problem of automatically moving to the sensor acquisition position and the problem of determining the scale.

  In addition, by using the mobile sensor device 10 according to the first embodiment, it is possible to naturally enjoy the advantages of the inspection method of the sensor mobile type (the advantage that no change or modification of the equipment to be inspected is required).

[References]
The references cited in the above description are as follows.
[Reference 1] IEICE "Forest of Knowledge" 2nd group-2nd chapter-3rd chapter 3D geometric analysis <URL: http://www.ieice-hbkb.org/>
[Reference 2] OpenCV camera calibration and 3D configuration <URL: http://opencv.jp/opencv-2.1/cpp/camera_calibration_and_3d_reconstruction.html>
[Reference 3]
Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski, "ORB: An efficient alternative to SIFT or SURF", ICCV 2011: 2564-2571.
<URL: http://www.willowgarage.com/sites/default/files/orb_final.pdf>
[Reference 4]
E. Mair et al., "Adaptive and Generic Corner Detection Based on the Accelerated Segment Test" (2010),
<URL: http://www6.in.tum.de/Main/Publications/Mair2010c.pdf>
[Reference 5]
PF Alcantarilla et al., "Fast Explicit Diffusion for Accerelated Features in Nonlinear Scale Spaces" (2013)
<URL: http://isit.u-clermont1.fr/~ab/Publications/Alcantarilla_etal_BMVC13.pdf>
[Reference 6]
S. Leutenegger et al., "BRISK: Binary Robust Invariant Scalable Keypoints" (2011)
<URL: http://www.robots.ox.ac.uk/~vgg/rg/papers/brisk.pdf>
[Reference 7]
E. Rosten et al., "Machine Learning for High-Speed Corner Detection" (2006)
<URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.3991&rep=rep1&type=pdf>
[Reference 8]
J. Shi et al., "Good Features to Track" (1994)
<URL: https://users.cs.duke.edu/~tomasi/papers/shi/TR_93-1399_Cornell.pdf>
[Reference 9]
PF Alcantarilla et al., "KAZE Features" (2012)
<URL: https://www.doc.ic.ac.uk/~ajd/Publications/alcantarilla_etal_eccv2012.pdf>
[Reference 10]
J. Matas et al., "Robust Wide Baseline Stereo from Maximally Stable External Regions" (2002)
<URL: https://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/matas_bmvc2002.pdf>
[Reference 11]
Lowe, D. (2004) Distinctive image features from scale-invariant keypoints. Int. Journal of Computer Vision. 60 (2). Pp.91-110.
<URL: https://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/lowe_ijcv2004.pdf>
[Reference Document 12]
Bay, H., Ess, A., Tuytelaars, T. & Van Gool, L. (2008) .SURF: Speeded Up Robust Features. Computer Vision and Image Understanding (CVIU). 110 (3) .pp.346-- 359.
<URL: ftp://ftp.vision.ee.ethz.ch/publications/articles/eth_biwi_00517.pdf>
[Reference 13]
OpenCV-Python Tutorials Feature point matching <URL: http://lang.sist.chukyo-u.ac.jp/classes/OpenCV/py_tutorials/py_feature2d/py_matcher/py_matcher.html>
[Reference 14]
OpenCV
<URL: https://docs.opencv.org/3.2.0/d9/d0c/group__calib3d.html#ga13f7e34de8fa516a686a56af1196247f>

[Modification]
The configuration and operation of the mobile sensor device 10 described in the above embodiment are merely examples, and do not limit the configuration of the device.

[Modification 1]
FIG. 6 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the first modification. For example, the sensor used for inspection may be a camera. In this case, the sensor unit 41 and the sensor control unit 205 are unnecessary, and the camera control unit 202 controls the camera unit 20 to make the camera function as a sensor. The sensor driving unit 40 moves the camera unit 20 (changes the position of the camera unit 20). Alternatively, instead of the movement of the camera unit by the sensor driving unit 40, the camera control unit 202 may control the camera unit 20 to use the zoom function of the camera.

[Modification 2]
FIG. 7 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the second modification. For example, the type of sensor used for the sensor unit 41 may be a non-contact type sensor such as a camera or a distance meter. In this case, it is not necessary to control the position of the sensor by the sensor driving unit 40. The mobile sensor device 10 does not require the sensor drive control unit 204, the sensor drive amount storage unit 81, and the sensor drive unit 40. When such a mobile sensor device 10 is used, in advance preparation, sensor alignment is not necessary, and the process of step S07 shown in FIG. 4 is not executed.

[Modification 3]
FIG. 8 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the third modification. For example, the mobile sensor device 10 may include a positioning sensor such as a Global Navigation Satellite System (GNSS) or a depth camera. In this case, when calculating the scale (scale factor) by calculating the position and orientation conversion equation between the images, the position and orientation conversion equation calculation unit 203 uses the positioning sensor unit 42 instead of the control amount (movement amount) for the device driving unit 30. It is also possible to use the information obtained in.

  Here, as shown in FIG. 9, in order to determine the scale, it is necessary to determine the size of the subject (the inspection target), the depth distance to the subject, and the distance between the viewpoints. The position and orientation conversion formula calculation unit 203 calculates a scale using a distance between viewpoints and a depth distance obtained by a positioning sensor such as a GNSS or a depth camera.

[Modification 4]
FIG. 10 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the fourth modification. The mobile sensor device 10 may use a reference image or a camera parameter acquired by another device. In this case, the operation unit 50 and the display unit 60 are unnecessary in the mobile sensor device 10.

[Modification 5]
FIG. 11 is a flowchart illustrating an example of processing of the mobile sensor device 10 according to the fifth modification. In the flowcharts after FIG. 11, the same steps as those in FIG. 4 are denoted by the same steps, and detailed description thereof is omitted.

  In the operation of the mobile sensor device 10 described with reference to FIG. 4, the calculation of the position and orientation conversion formula in steps S02 and S05 shown in FIG. 4 may fail. The mobile sensor device 10 according to the fifth modification determines whether or not the calculation processing of the position and orientation conversion formula is correct. Specifically, in step S11 and step S14, when the number of corresponding feature points for which matching was possible is smaller than a certain number (threshold), the position / posture conversion formula calculation unit 203 determines The calculation is determined as “failure”. That is, the position / posture conversion formula calculation unit 203 may execute threshold processing on the calculated number of feature points, and determine whether the calculation of the position / posture conversion formula is correct or not according to the result.

  As shown in FIG. 11, when the calculation of the position and orientation conversion formula fails (calculation fails; step S11, No branch), the mobile sensor device 10 moves the shooting position (step S12) and re-comparisons the comparison image. Acquisition and recalculation of the position and orientation conversion formula are performed.

  In the process of step S05, in order to determine the scale, the position and orientation conversion formula calculation unit 203 uses the control amount (the movement amount of the device) of the device driving unit 30. However, if the calculation of the position and orientation conversion formula fails, the movement amount is accumulated each time the calculation fails. Therefore, the mobile sensor device 10 performs an accumulation process in step S13, and uses the accumulated movement amount to calculate the scale factor a21 of the parallel movement in the process in step S05.

[Modification 6]
FIG. 12 is a flowchart illustrating an example of processing of the mobile sensor device 10 according to the sixth modification. As shown in FIG. 12, after the process of step S06, the mobile sensor device 10 obtains the comparison image I3 (step S21), and performs the same process as step S05 to convert the comparison image I3 and the reference image into position and orientation. Is calculated (step S22).

  When the distance between the comparison image I3 and the reference image (parallel movement distance a03T03) is longer than a predetermined value (step S23, Yes branch), the mobile sensor device 10 moves the own device (step S24) and the comparison image I3. Repeat the acquisition. As a result, the position and orientation of the mobile sensor device 10 approaches the reference image capturing position and orientation.

  Alternatively, as shown in FIG. 13, the mobile sensor device 10 may realize the movement of the position and orientation conversion formula (Step S32) calculated from the comparison image I3 acquired in Step S31 by the sensor driving unit 40. . In the movement process in step S07a, the movement is performed by multiplying the calculation result in step S32 by the information stored in the sensor drive amount storage unit 81.

[Modification 7]
When there are a plurality of inspection targets (sensing targets), the mobile sensor device 10 uses a plurality of reference images. For example, the mobile sensor device 10 automatically selects a reference image corresponding to a place from the current captured image (comparative image I1). For example, the mobile sensor device 10 may perform the process of step S02 on all the reference pixels and select the reference image with the largest number of corresponding feature points. That is, the position and orientation conversion formula calculation unit 203 calculates a reference image corresponding to the captured image from the plurality of reference images corresponding to the plurality of sensing targets, respectively, from the captured image and the plurality of reference images. The selection may be based on the number of points.

[Modification 8]
A two-dimensional code such as a known AR (Augmented Reality) marker may be provided in the reference image to improve the calculation accuracy of the position / posture relationship between the cameras and determine the scale factor. In this case, one shot of the comparison image is sufficient.

[Modification 9]
FIG. 14 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the ninth modification. As shown in FIG. 14, the reference image storage unit 80, the sensor drive amount storage unit 81, and the camera parameter storage unit 82 may be arranged in an external server device 90. In this case, the communication unit 91 acquires the information stored in each storage unit via the network 93. The communication unit 92 of the server device 90 transmits information read from each storage unit to the mobile sensor device 10. The communication units 91 and 92 are a wireless communication device such as a wireless LAN (Local Area Network) or a mobile line, or a wired communication device such as a wired LAN. The network 93 is a closed network such as a private network or a wide area network such as the Internet.

[Modification 10]
FIG. 15 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the tenth modification. The sensor device need not be mobile. For example, the operator may carry the sensor device so that the sensor information acquisition position and orientation fall within the movable range of the sensor driving unit 40. In this case, the sensor drive unit 40 controls (moves) the sensor unit 41 and the camera unit 20, and the processing performed by the device drive unit 30 and the device drive control unit 201 is performed by the sensor drive unit 40 and the sensor drive control unit 204. It only has to be executed.

[Modification 11]
FIG. 16 is a diagram illustrating an example of a processing configuration of the mobile sensor device 10 according to the eleventh modification. Combination of each modification results in a minimum device configuration shown in FIG.

  From the above description, the industrial applicability of the present invention is clear. However, the present invention examines public infrastructure areas such as roads, bridges, tunnels, rivers, slopes, ports, airports, railways, and power plants. It can be suitably applied to monitoring. Furthermore, the present invention can be suitably applied to technical fields such as inspection and monitoring of factory equipment in the smart factory area, image recognition, signal processing, and robots.

Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited thereto.
[Appendix 1]
This is the same as the sensor device according to the first viewpoint.
[Appendix 2]
The conversion formula calculation unit,
The reference image, the first and second captured images captured by the camera unit, and the autonomous shift from the position and direction at which the first captured image was obtained to the position and direction at which the second captured image was obtained. The sensor device according to claim 1, wherein the conversion formula is calculated based on movement control information regarding movement of the device and internal parameter information of the camera.
[Appendix 3]
The camera control unit controls the camera unit to acquire the first captured image,
The conversion formula calculation unit calculates a first conversion formula for the reference image and the first captured image based on the reference image, the first captured image, and internal parameter information of the camera,
The drive control unit performs movement according to the movement control information,
The camera control unit controls the camera unit to obtain the second captured image,
The conversion equation calculation unit calculates a second conversion equation for the reference image and the second captured image based on the reference image, the second captured image, internal parameter information of the camera, and the movement control information. ,
The sensor device according to claim 2, wherein the drive control unit moves the own device to a position and a direction at the time when the reference image is captured based on the calculated second conversion formula.
[Appendix 4]
4. The sensor device according to claim 1, wherein the conversion formula calculated by the conversion formula calculation unit includes information on a rotational movement of the own device and information on a parallel movement of the own device. 4.
[Appendix 5]
The conversion formula calculation unit,
Using the movement control information, calculating a scale factor that defines a translation from the position and the direction at the time of acquiring the first captured image to the position and the direction at the time of acquiring the second captured image, The sensor device according to Appendix 4, preferably citing Appendix 3.
[Appendix 6]
A sensor unit including a sensor;
The sensor device according to any one of Supplementary Notes 1 to 5, further comprising: a sensor control unit configured to control the sensor unit to acquire sensor information.
[Appendix 7]
A sensor drive unit that changes the position of the sensor,
The sensor device according to claim 6, further comprising: a sensor drive control unit that controls the sensor drive unit.
[Appendix 8]
The conversion formula calculation unit,
From among the plurality of reference images corresponding to each of the plurality of sensing targets, a reference image corresponding to the captured image is selected based on the number of feature points calculated from each of the captured image and the plurality of reference images. Preferably, the sensor device according to any one of supplementary notes 1 to 7.
[Appendix 9]
The camera control unit acquires a third captured image,
The conversion formula calculation unit calculates the conversion formula from the third captured image,
The sensor device according to any one of Supplementary Notes 1 to 8, wherein the drive control unit executes movement of the own device according to a conversion formula calculated from the third captured image.
[Appendix 10]
The sensor device according to any one of Supplementary Notes 1 to 9, wherein the reference image includes an Augmented Reality (AR) marker.
[Appendix 11]
This is the same as the control method of the sensor device according to the second viewpoint described above.
[Supplementary Note 12]
This is as in the program according to the third aspect described above.
In addition, the form of Supplementary Note 11 and the form of Supplementary Note 12 can be expanded to the form of Supplementary Note 2 to the form of Supplementary Note 10 similarly to the form of Supplementary Note 1.

  The disclosures of the above cited patent documents and the like are incorporated herein by reference. Modifications and adjustments of the embodiments or examples are possible within the framework of the entire disclosure (including the claims) of the present invention and based on the basic technical concept thereof. In addition, various combinations or selections of various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the frame of the entire disclosure of the present invention. (Including partial deletions) are possible. That is, the present invention naturally includes various variations and modifications that can be made by those skilled in the art according to the entire disclosure including the claims and the technical idea. In particular, with respect to the numerical ranges described in this document, any numerical values or small ranges included in the ranges should be interpreted as being specifically described even if not otherwise specified.

10 Mobile sensor device 11 CPU (Central Processing Unit)
12 memory 13 input interface 14 output interface 20, 101 camera unit 30, 103 device drive unit 40 sensor drive unit 41 sensor unit 42 positioning sensor unit 50 operation unit 60 display unit 70 control unit 80 reference image storage unit 81 sensor drive amount storage unit 82 camera parameter storage unit 90 server device 91, 92 communication unit 93 network 100 sensor device 102, 202 camera control unit 104 drive control unit 105 conversion formula calculation unit 201 device drive control unit 203 position / posture conversion formula calculation unit 204 sensor drive control Unit 205 sensor control unit

Claims (10)

A camera section including a camera,
A camera control unit for controlling the camera unit to acquire an image,
A device drive unit for moving the own device,
A drive control unit that controls the device drive unit and moves the own device,
Based on at least a captured image captured by the camera unit and a previously acquired reference image, a viewpoint of the camera when the captured image is acquired is converted into a viewpoint of the camera when the reference image is acquired. A conversion formula calculation unit that calculates the conversion formula of
With
The drive control unit includes:
A sensor device, based on the conversion formula, that moves the subject device from the position and the direction of the subject device with respect to the sensing target when the photographed image is photographed to the position and the direction when the reference image is photographed. The conversion formula calculation unit,
The reference image, the first and second captured images captured by the camera unit, and the autonomous shift from the position and direction at which the first captured image was obtained to the position and direction at which the second captured image was obtained. The sensor device according to claim 1, wherein the conversion formula is calculated based on movement control information on movement of the device and information on internal parameters of the camera. The camera control unit controls the camera unit to acquire the first captured image,
The conversion formula calculation unit calculates a first conversion formula for the reference image and the first captured image based on the reference image, the first captured image, and internal parameter information of the camera,
The drive control unit performs movement according to the movement control information,
The camera control unit controls the camera unit to obtain the second captured image,
The conversion equation calculation unit calculates a second conversion equation for the reference image and the second captured image based on the reference image, the second captured image, internal parameter information of the camera, and the movement control information. ,
3. The sensor device according to claim 2, wherein the drive control unit moves the device based on the calculated second conversion formula to a position and a direction when the reference image is captured. 4.   4. The sensor device according to claim 1, wherein the conversion formula calculated by the conversion formula calculation unit includes information on a rotational movement of the own device and information on a parallel movement of the own device. 5. The conversion formula calculation unit,
Using the movement control information, calculating a scale factor that defines a translation from the position and the direction at the time of acquiring the first captured image to the position and the direction at the time of acquiring the second captured image, The sensor device according to claim 4, wherein the sensor device according to claim 3 is cited. A sensor unit including a sensor;
The sensor device according to claim 1, further comprising: a sensor control unit configured to control the sensor unit to acquire sensor information. A sensor drive unit that changes the position of the sensor,
The sensor device according to claim 6, further comprising: a sensor drive control unit that controls the sensor drive unit. The conversion formula calculation unit,
From among the plurality of reference images corresponding to each of the plurality of sensing targets, a reference image corresponding to the captured image is selected based on the number of feature points calculated from each of the captured image and the plurality of reference images. The sensor device according to any one of claims 1 to 7. A camera section including a camera,
A device drive unit for moving the own device,
In a sensor device comprising:
Based on at least a captured image captured by the camera unit and a previously acquired reference image, a viewpoint of the camera when the captured image is acquired is converted into a viewpoint of the camera when the reference image is acquired. Calculating a conversion formula of
Based on the conversion formula, from the position and direction of the own device with respect to the sensing target when the captured image was captured, moving the own device to the position and direction when the reference image was captured,
A control method for a sensor device including: A camera section including a camera,
A device drive unit for moving the own device,
A computer mounted on a sensor device having
Based on at least a captured image captured by the camera unit and a previously acquired reference image, a viewpoint of the camera when the captured image is acquired is converted into a viewpoint of the camera when the reference image is acquired. Processing for calculating the conversion formula of
Based on the conversion formula, from the position and direction of the own device with respect to the sensing target when the captured image was taken, a process of moving the own device to the position and direction when the reference image was taken,
A program that executes

Images