WO2023166868A1

WO2023166868A1 - Mobile imaging robot system and method for controlling same

Info

Publication number: WO2023166868A1
Application number: PCT/JP2023/001032
Authority: WO
Inventors: 陽久保
Original assignee: 株式会社日立製作所
Priority date: 2022-03-03
Filing date: 2023-01-16
Publication date: 2023-09-07
Also published as: JP2023128220A

Abstract

This mobile imaging robot system is configured to comprise: an imaging unit (50) for imaging the surroundings and outputting imaging data; a robot arm (40) having the imaging unit mounted on the tip thereof, the robot arm (40) adjusting the position of the tip; a movement unit (60) that moves, the robot arm being mounted on the movement unit; and a robot arm control unit (10) for controlling the robot arm, on the basis of a tracking trajectory of the position of a prescribed target in an image captured by the imaging unit, so as to prevent vibration of the tip of the robot arm caused by movement of the center of gravity of the movement unit or by an impact during contact with the ground. The mobile imaging robot system is also configured such that a stable field of view can be obtained even during movement.

Description

MOBILE IMAGING ROBOT SYSTEM AND CONTROL METHOD THEREOF

The present invention relates to a mobile imaging robot system and its control method.

There is a need for mobile robots such as multi-legged robots to perform tasks such as inventory management, visual inspection, and shape measurement in places that are difficult for people to enter or in vast premises. At that time, the robot performs the work using the imaging data acquired by the imaging device. The imaging device mentioned here is a device for obtaining two-dimensional or three-dimensional images of the surrounding environment, such as a camera or a LiDAR (light detection and ranging) sensor, or a device that fuses them.

In the actual application of mobile robots to the fields of inventory management, logistics, and maintenance, it is possible to flexibly move the viewpoint of the robot rather than moving the imaging target to check for scratches, foreign objects, the number of products, etc. End effectors such as pick and place are often used depending on the situation. At that time, there are scenes where the manipulator is moved in a narrow space such as between product display shelves or under the floor of a vehicle under maintenance to change the viewpoint from multiple angles.

For this reason, the mobile robot's imaging device is attached to the hand position of the robot arm on which it is mounted, rather than the mobile object itself. As a result, when inspecting the appearance of a large site, it is possible to accurately capture images while moving. In addition, the imaging data can be used to determine the behavior of the robot, such as collision avoidance.

For example, in Patent Document 1, in a robot arm mounted on a moving part, when the position of the workpiece or the arm is displaced after the moving part moves to a predetermined place, state monitoring using camera images A technique for generating a trajectory plan for the arm to compensate for deviations is described.

JP 2018-158391 A

When the moving part of the imaging device of the mobile robot is moving rather than in a stationary state, the tip position of the mounted robot arm may vibrate due to unevenness such as grooves and steps on the road surface on which the moving part moves. In addition, it vibrates due to disturbance due to the movement of the center of gravity of the moving part accompanying the movement and the impact at the time of grounding.

For this reason, there is a problem that the accuracy of object recognition itself based on imaging data decreases. In particular, imaging by a small MEMS-type LiDAR sensor, which is expected to be used for determining the behavior of mobile robots and robot arms mounted on mobile robots (collision avoidance, zooming in and out of cameras during inspections, etc.) , the accuracy of distance and angle measurements decreases unless fixed-point observation is used.

Also, when a person operates a robot from a remote location, the operator's fatigue increases if the image vibrates. Thus, the scope of the problem of vibration of the imaging device is not limited to the problem of measurement accuracy.

In the prior art described above, it is not considered that the mobile imaging robot performs imaging while moving.

An object of the present invention is to provide a mobile imaging robot system that can obtain a stable field of view even during movement.

In order to solve the above-described problems, the mobile imaging robot system of the present invention includes an imaging unit that photographs the surroundings and outputs imaging data; a robot arm that mounts the imaging unit on a hand and adjusts the position of the hand; Based on the moving unit that carries the arm and moves, and the tracking trajectory of the position of the predetermined target in the captured image of the imaging unit, vibration of the hand of the robot arm occurs due to the movement of the center of gravity of the moving unit or the impact at the time of contact with the ground. a robot arm control unit for controlling the robot arm so as to prevent the robot arm from

According to the present invention, it is possible to stabilize the imaging field of view while the mobile imaging robot is moving.

1 is a configuration diagram of a mobile imaging robot system of Example 1. FIG. 4 is a configuration diagram of a robot arm control unit of Example 1. FIG. It is a figure which shows the form comprised from the moving part by a robot arm and a multi-legged robot. It is a figure which shows the form comprised from the moving part by a robot arm and a multi-legged robot. FIG. 5 is a flowchart of processing of the mobile imaging robot system of Example 1; It is a figure explaining frequency analysis. It is a figure explaining preparation of a vibration compensation control command. It is a figure explaining preparation of a vibration compensation control command. FIG. 11 is a configuration diagram of a robot arm control unit of Example 2; FIG. 11 is a flowchart of processing of the mobile imaging robot system of Example 2; It is a figure which shows the analysis result of a frequency-analysis part. It is a figure which shows the analysis result of a frequency-analysis part. It is a figure which shows the positional information on a ground plane map. 4 is a control block diagram for explaining the vibration compensation operation of the robot arm 40 of this embodiment. FIG. FIG. 10 is a processing flow diagram when generating vibration data and ground plane data at the same time; It is a figure which shows the time-frequency-analysis result of a frequency-analysis part.

Hereinafter, embodiments of the mobile imaging robot system of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a configuration diagram of a mobile imaging robot system of this embodiment.
The mobile imaging robot system includes a robot arm 40 equipped with an imaging unit 50 for imaging the surroundings at the hand, a moving unit 60 on which the robot arm 40 is mounted, a computer 30 for collecting and displaying data, and controlling the robot arm 40. The robot arm control unit 10 is configured by connecting the robot arm control unit 10 , the computer 30 , and the robot arm 40 via a network 20 .

The moving part 60 is equipped with the robot arm 40 and configured to be autonomously movable by the moving part driving motor 61 .

The robot arm 40 has a rotation angle information sensor group (not shown) and is driven by a robot arm drive motor 41 so that the hand position can be adjusted. The robot arm 40 has an imaging unit 50 at its tip. Then, the imaging unit 50 performs imaging in a direction corresponding to the orientation of the hand.

The robot arm control unit 10 includes an input/output unit 11 for exchanging input/output data with the robot arm 40 and the computer 30, an arithmetic unit 12 for executing the functions of the imaging device, and an input from the user of the imaging robot. A storage unit 13 is provided in which data and data extracted by the calculation unit 12 are stored. The configuration of the robot arm control section 10 will be described in detail with reference to FIG.

FIG. 2 is a configuration diagram of the robot arm control unit 10 of the mobile imaging robot system.
The calculation unit 12 of the robot arm control unit 10 includes a target selection unit 120, a target tracking unit 121, a frequency analysis unit 122, a frequency/amplitude determination unit 123, a vibration compensation control unit 124, and a robot arm motion control unit 125. and The storage unit 13 includes vibration data 130 and vibration compensation control data 131 . Each component will be described in detail below.

The target selection unit 120 selects a predetermined stationary object as a target to be tracked from the imaging data around the imaging unit 50 when the moving unit 60 is stationary. The imaging data is input by the input/output unit 11 via the network 20 .

The target tracking unit 121 tracks the target selected by the target selection unit 120 based on the imaging data while the moving unit 60 is moving, and calculates the relative position of the target and the robot arm 40 (imaging unit 50) in the vertical direction. Get change.
That is, the target tracking unit 121 acquires changes in the hand position of the robot arm 40 accompanying movement of the moving unit 60 .

The frequency analysis unit 122 performs frequency analysis using the time-series data of the target position for each frame of the imaging data as the time-series data of the hand position of the robot arm.

The frequency/amplitude determination unit 123 uses the output result of the frequency analysis unit 122 to calculate the dominant frequency and amplitude of the vibration of the hand position of the robot arm 40 related to movement.

The vibration compensation control unit 124 causes the robot arm 40 to perform an operation to cancel the vibration of the hand position (imaging unit 50) of the robot arm 40 caused by the movement of the moving unit 60 based on the vibration calculated by the frequency/amplitude determination unit 123. Generate motion commands for

The robot arm motion control unit 125 controls rotation of the robot arm drive motor 41 that drives at least one joint of the robot arm based on the motion command generated by the vibration compensation control unit 124 .

The vibration data 130 is data relating to shaking caused by the movement of the moving part determined by the frequency/amplitude determination unit 123 and is stored in the storage unit 13 . Further, the vibration data 130 is expected to be required after the robot arm 40 is recalibrated, when walking on an unknown ground, or when the hardware part or control software of the robot arm 40 is updated. It can be referred to for creating vibration compensation controls.

The vibration compensation control data 131 is vibration compensation control data of the robot arm for canceling the vibration of the hand position of the robot arm due to the movement operation of the moving unit 60, and is stored in the storage unit 13 in association with the vibration data 130. . The vibration compensation control data 131 is newly generated together with new vibration data 130 when the moving speed of the moving unit 60 or the hardware of the moving unit 60 is changed.

Specifically, the robot arm control unit 10 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), input/output I/F (Interface), communication It is realized by an information processing device having an I/F and a media I/F.

By executing a program stored in the ROM or HDD by the CPU, the calculation unit 12 operates a target selection unit 120, a target tracking unit 121, a frequency analysis unit 122, a frequency/amplitude determination unit 123, a vibration compensation control unit 124, and a robot arm. The functions of the motion control unit 125, the vibration data 130, and the vibration compensation control data 131 are realized.

The storage unit 13 is implemented by an HDD and stores vibration data 130 and vibration compensation control data 131 .
The input/output unit 11 is implemented by an input/output I/F or a communication I/F, and is connected to the imaging unit 50, the robot arm 40, and the moving unit 60 via the network 20 to exchange data.

Here, the forms of the robot arm 40, the imaging unit 50, and the moving unit 60 in the mobile imaging robot system of this embodiment will be described with reference to FIGS. 3A and 3B.

FIG. 3A shows an example of a form composed of a robot arm 40 having an imaging unit 50 attached to its hand and a moving unit 60 made up of a multi-legged robot.
FIG. 3B shows an example of a form composed of a robot arm 40 having an end effector and an imaging unit 50 attached to the hand, and a crawler track type moving unit 60 .

Next, the processing of the mobile imaging robot system of this embodiment will be explained with reference to the flowchart of FIG.

At S101, the robot arm control unit 10 starts acquisition of surrounding imaging data by the imaging unit 50 (start of imaging).

In step S102, the target selection unit 120 selects a specific object as a target to be tracked in the range captured while the moving unit 60 is stationary.

At step S103, the target selection unit 120 determines whether the target selected at step S102 is stationary. If it is stationary (YES in S103), the process proceeds to step S104, and if not (NO in S103), the process returns to step S102. Whether or not the target is stationary is determined by, for example, whether or not the position of the target is constant in imaging data for several frames.

At step S104, the target tracking unit 121 starts tracking the target of the imaging data.

At step S105, the robot arm control unit 10 permits the movement of the moving unit 60, and the moving unit drive motor 61 starts operating, thereby starting the movement of the moving unit 60.

In step S106, the target tracking unit 121 acquires the position of the target for each frame while the moving unit 60 is moving as a tracking trajectory.

In step S107, the frequency analysis unit 122 performs frequency analysis for determining the dominant frequency regarding the vibration of the hand position of the robot arm caused by the movement of the moving unit 60 from the tracking trajectory and frame rate acquired in step S106. conduct. The frequency analyzed by the frequency analysis unit 122 is the frequency of the disturbance due to the movement of the center of gravity of the moving unit 60 and the impact when the moving unit 60 touches the ground. Details will be described separately with reference to FIG.

In particular, the dominant frequency is the value determined by the walking cycle when the moving part 60 is of the multi-legged robot type (FIG. 3A), and the rotation frequency of the belt part or the belt part when the moving part 60 is of the endless track type (FIG. 3B). is determined by the rotational frequency of the motor that drives the In other words, the frequency is determined by the kinematic behavior of the moving part 60, and the vibration is unavoidable even on flat ground without steps or unevenness.

The frequency analysis unit 122 may use Fourier transform for frequency analysis, may use a time-frequency analysis method such as wavelet transform, or may derive using peak detection or an autocorrelation function. Moreover, any direction within a plane that intersects perpendicularly with the advancing direction of movement may be selected as the direction of vibration to be analyzed.

In step S108, the frequency/amplitude determining unit 123 assumes that the cause is caused by the movement of the moving unit 60 when a frequency with a predetermined value of amplitude intensity is obtained in the frequency analysis of step S107. Also, the control signal to the moving part 60 or the periodicity information of the control signal for at least one of the moving part driving motor 61 for driving the moving part 60 is compared with the periodicity information, and if the frequency matches, the moving part 60 is moved. may be attributed to

The frequency/amplitude determination unit 123 determines the dominant frequency and amplitude of the vibration of the hand position of the robot arm 40, which is the result of the frequency analysis in step S107, when it is assumed that the vibration is caused by the movement of the moving unit 60. Vibration data 130 is assumed.

Note that the amplitude of the vibration data 130 may be obtained by directly measuring the target with a camera or LiDAR sensor that has a function of surveying position and dimensions.
Alternatively, it may be indirectly estimated from the amount of displacement of the target within the field of view of the imaging unit 50 that is offset by vibrating the robot arm 40 with a specific amplitude and in the opposite phase to the obtained vibration.

In step S109, the frequency/amplitude determination unit 123 determines whether the assumption that the result of the frequency analysis in step S107 is caused by the movement of the moving unit 60 has continued for a predetermined time. If continued (YES in S109), the vibration data 130 is confirmed as being caused by the movement of the moving part 60, and the process proceeds to step S110.If not continued (NO in S109), the process returns to step S106. .

In step S110, the vibration compensation control unit 124 predictively moves the imaging unit 50 with the opposite phase to the frequency of the vibration data 130 and the same amplitude as the amplitude of the vibration data 130, thereby canceling the vibration of the target to be imaged. Control information (vibration compensating operation command) for the robot arm 40 is created so as to be performed. Details will be described separately with reference to FIG.

In step S<b>111 , the vibration compensation control section 124 stores the vibration data 130 in the storage section 13 .
Either step S110 or step S111 may be performed first, or may be performed simultaneously.

In step S112, the robot arm motion control unit 125 notifies the robot arm 40 of the control information (vibration compensation motion command) for the robot arm 40 created in step S110, and starts vibration compensation control of the robot arm.

In step S113, the robot arm motion control unit 125 acquires the tracking trajectory by the target tracking unit 121, and determines whether or not the shaking of the target is within the allowable range by vibration compensation control of the robot arm. If it is within the allowable range (YES in S113), the process proceeds to step S114, and if it is not within the allowable range (NO in S113), the process returns to step S106.

In step S114, the robot arm motion control unit 125 stores the control information (vibration compensation motion command) notified to the robot arm 40 in step S112 in the storage unit 13 as vibration compensation control data 131 in association with the vibration data 130. . By linking with the vibration data 130 , it can be used for vibration compensation of the robot arm 40 when similar vibration is detected by the imaging unit 50 .

Next, the frequency analysis in step S107 in FIG. 4 will be described with reference to FIG.
FIG. 5 is a diagram showing an example of the tracking trajectory acquired by the target tracking unit 121 in step S106 of FIG.

The frequency analysis unit 122 calculates the time t2- from the time t1 when the target takes the maximum peak position to the time t2 when the target next takes the maximum peak position in the time-series data (dotted curve) of the target position acquired as the tracking trajectory. With t1 as the period T, the frequency F is determined as its reciprocal to extract the frequency information of the vibration of the target.
Furthermore, the difference between the maximum peak position and the minimum peak position of the target is obtained, and the amplitude A is determined.

The frequency analysis unit 122 may determine each of the frequency F and the amplitude A from one sample or the average value of a plurality of samples. Alternatively, the frequency analysis unit 122 may perform calculation through frequency transform processing such as Fourier transform.

Next, generation of the vibration compensation control command in step S110 of FIG. 4 will be described with reference to FIGS. 6 and 7. FIG.
FIG. 6 shows the relationship between the predicted vibration (solid line) of the tip of the robot arm 40 (imaging unit 50) and the motion (broken line) of the tip of the robot arm 40 (imaging unit 50) in response to notification of the vibration compensation control command. FIG. 4 is a diagram showing;

The vibration compensation control unit 124 sets the vibration data 130 to the predicted vibration (solid line) of the tip of the robot arm 40 (imaging unit 50). Then, the vibration compensation control unit 124 creates a vibration compensation control command for the robot arm 40 so as to add, to the motion of the hand of the robot arm 40 , a motion with the opposite phase and the same amplitude as the vibration data 130 .

As a result, as shown in FIG. 7, which shows the tracking trajectory of the target after the vibration compensation control is performed, the vibration of the hand (imaging unit 50) of the robot arm 40 caused by the movement of the moving unit 60 is compensated. The field of view of the imaging section 50 can be stabilized.

It is not always necessary to display the result of tracking of the target by the imaging unit 50 of FIG. .

In Example 1, the robot arm control unit 10 of the embodiment controls the robot arm 40 so that the vibration of the hand of the robot arm 40 due to the movement of the center of gravity of the moving unit 60 and the impact at the time of contact with the ground is suppressed, and the vibration is compensated. However, in the second embodiment, in addition to the above vibrations, vibrations due to irregularities or steps on the ground contact surface will be described.

The robot arm control unit 10 of the second embodiment first separates the vibration due to the movement of the center of gravity of the moving part 60 and the impact at the time of grounding from the vibration due to the unevenness of the ground surface or the difference in level. , the robot arm 40 is controlled so as to compensate for both vibrations, and the field of view of the imaging unit 50 is stabilized.

FIG. 8 is a configuration diagram of the robot arm control unit 10 of the mobile imaging robot system of the second embodiment.

The robot arm control unit 10 is configured by adding a position information acquisition unit 126 and a ground surface data generation unit 127 to the calculation unit 12 of the robot arm control unit 10 of the first embodiment described with reference to FIG. It is obtained by adding the ground plane map 132 and the ground plane data 133 . Therefore, here, the added position information acquisition unit 126, the ground surface data creation unit 127, the ground surface map 132, and the ground surface data 133 will be described, and the other parts are the same as those in FIG.

The position information acquisition unit 126 acquires position information of the moving unit 60 within the movement range of the moving unit 60 .

When the vibration compensation control unit 124 is performing vibration compensation control due to the movement of the center of gravity of the moving unit 60 and the impact at the time of contacting the ground, the ground surface data generation unit 127 causes the frequency analysis unit 122 to ) is detected, it is determined that the vibration has occurred due to unevenness or steps on the contact surface of the moving part 60 or an obstacle. Then, the ground plane data generation unit 127 associates the frequency and amplitude obtained by the frequency analysis unit 122 with the ground plane map 132 with the position information of the moving unit 60 as the ground plane data 133 and stores the data in the storage unit 13 . If the existing ground plane data 133 exists in the storage unit 13, the ground plane data 133 is updated.

The ground plane map 132 is information about the shape and layout of the range in which the moving unit 60 moves. The ground plane map 132 may be set from the input/output unit 11 as a known map in advance, or if the moving section 60 has a mapping function, the map created by the moving section 60 is stored as the ground plane map. It may be synchronized or input to unit 13 .

The ground surface data 133 is the frequency and amplitude of vibration when vibration of the tip of the robot arm 40 (imaging unit 50) caused by an unevenness or step on the ground surface of the moving unit 60 or an obstacle is detected. It is newly created or updated by the creation unit 127 . The ground plane data 133 at this time will be described with reference to FIG. 10B.

The ground plane data 133 is linked to the ground plane map 132 and stored in the storage unit 13 . The newly created or updated ground plane data 133 is stored to perform vibration compensation control of the robot arm 40 so that the imaging unit 50 does not vibrate when the moving unit 60 moves in the same place from now on.

Next, the process of creating the ground plane data 133 of the robot arm control unit 10 of the second embodiment will be described with reference to the flowchart of FIG.
The flow chart of FIG. 9 is obtained by adding steps S115 to S119 to the flow chart of FIG. Here, steps S115 to S119 will be explained, and the other steps are the same as in FIG. 4, so the explanation will be omitted.

In step S115, the vibration compensation control unit 124 continues to move the moving unit 60 while the vibration compensation control of the robot arm 40 is being executed.

In step S116, the vibration compensation control unit 124 checks whether or not the vibration (in particular, the amplitude) of the robot arm 40 tip (imaging unit 50) is outside the allowable range even though the vibration compensation control is being performed. judge. If the vibration does not deviate from the allowable range (NO in S116), the process returns to step S115, and if the vibration does not deviate from the allowable range (YES in S116), the process proceeds to step S117.

In step S<b>117 , the position information acquisition unit 126 acquires position information of the moving unit 60 within the movement range of the moving unit 60 . Then, the position information acquisition unit 126 stores this position information in the ground plane map 132 .

In step S118, the ground plane data creation unit 127 determines whether or not there is ground plane data 133 corresponding to the position information determined in step S116 that the vibration is out of the allowable range. If the ground contact surface data generating unit 127 determines that there is (YES in S118), the process proceeds to step S120, and if it determines that there is no (NO in S118), the process proceeds to step S119.

In step S119, the ground plane data generation unit 127 associates the frequency and amplitude of the vibration caused by the ground plane newly detected by the frequency analysis unit 122 with the position information obtained in step S117 in the ground plane map 132. The ground plane data 133 is stored in the storage unit 13, and the process ends.

In step S120, the ground plane data creation unit 127 converts the frequency and amplitude of the vibration caused by the ground plane newly detected by the frequency analysis unit 122 into the existing ground plane data 133 corresponding to the position information obtained in step S117. is added to and updated, and the process ends.

As described above, the robot arm control unit 10 of the second embodiment controls and moves the robot arm 40 so as to compensate for the movement of the center of gravity of the moving unit 60 and the vibration caused by the impact at the time of grounding. Vibrations due to irregularities, steps, or obstacles are detected, so that ground plane data 133 at a predetermined position can be obtained separately.

Next, an example of a method for creating the vibration compensation control data 131 and the ground plane data 133 in this embodiment will be described.

FIG. 10A is a diagram showing analysis results of the frequency analysis unit 122. FIG.
The frequency/amplitude determination unit 123 extracts the disturbance vibration due to the movement of the center of gravity of the moving part and the impact at the time of grounding as vibration data of frequency F1 and amplitude A1, and makes the vibration data 130 . Then, based on the vibration data 130, the vibration compensation control unit 124 creates vibration compensation control data 131 for the robot arm 40 so that the vibration of the tip of the robot arm 40 (imaging unit 50) is canceled. control 40; As a result, the robot arm 40 compensates for vibration that constantly occurs as long as the moving part 60 is moving, so that the imaging part 50 can continue imaging without disturbance in the field of view.

FIG. 10B is a diagram showing the analysis results of the frequency analysis unit 122 when new vibrations due to bumps or bumps on the ground surface or collision with an obstacle are detected.
The contact surface data creation unit 127 extracts the vibration due to the unevenness or step of the contact surface as the contact surface data having the frequency F2 and the amplitude A2, and converts this to the contact surface map where the vibration occurred, which is acquired by the position information acquisition unit 126. The ground plane data 133 is linked to the position information X (see FIG. 10C) on 132 .

Next, the vibration compensation operation of the robot arm 40 of this embodiment will be described with reference to the control block diagram of FIG.

The robot arm 40 adds vibration data 130 to the vibration compensation control to offset vibrations that occur permanently and temporarily at specific locations on the ground plane map 132 as long as the moving part 60 is moving. Ground contact data 133 is added to the vibration compensation control to offset terrain-induced vibrations. Thereby, the vibration compensation control can be changed so that the imaging unit 50 does not vibrate on the known map. Vibration compensation control of the robot arm 40 in this embodiment is created by inverting the phases of the vibration data 130 and the ground plane data 133 and adding them together.

In this embodiment, the vibration data 130 and the ground plane data 133 may be created at the same time. This case will be described with reference to FIGS. 12 and 13. FIG.

FIG. 12 is a processing flow for simultaneously creating vibration data 130 and ground plane data 133 in this embodiment. Steps S108-S114, steps S121, and steps S117-S120 proceed in parallel.

In step S121, it is determined whether or not the result of the frequency analysis in step S107 has detected a single pulse-like vibration. This is based on the fact that the vibration caused by bumps or bumps on the contact surface or collision with an obstacle becomes a single-pulse vibration, unlike the movement of the center of gravity of the moving part, which is a periodic vibration, or the impact at the time of contact with the ground. .

Specifically, steps S108 to S114 are the same as the processing described with reference to FIG.
In step S121, if the result of the frequency analysis unit 122 is not single-pulse vibration, the ground plane data generation unit 127 terminates the parallel processing. The processing of S117-S120 is performed.

FIG. 13 is a diagram showing the time-frequency analysis result of the frequency analysis unit 122. As shown in FIG.
The frequency analysis unit 122 obtains the spectrogram of FIG. 13 by short-time Fourier transform or wavelet transform. Vibration data 130 due to the movement of the moving body is assumed to be constant continuous vibration such as the vibration of frequency F1, and ground plane data 133 is assumed to be vibration appearing as a single pulse wave such as vibration of frequency F2. As a result, the vibration data and the ground contact data can be separately acquired at the same time.

According to this embodiment, by controlling the robot arm with the imaging device using the imaging data taken by the mobile imaging robot, it is possible to predict and cancel the shaking due to movement and stabilize the field of view. It is possible to stabilize the field of view for imaging while moving.

The present invention is not limited to the above-described embodiments, and includes various modifications. The above embodiments have been described in detail to facilitate understanding of the present invention, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.

10 robot arm control unit 11 input/output unit 12 calculation unit 120 target selection unit 121 target tracking unit 122 frequency analysis unit 123 frequency/amplitude determination unit 124 vibration compensation control unit 125 robot arm operation control unit 13 storage unit 130 vibration data 131 vibration compensation Control data 20 Network 30 Computer 40 Robot arm 41 Robot arm drive motor 50 Imaging unit 60 Moving unit 61 Moving unit drive motor

Claims

an imaging unit that captures images of the surroundings and outputs captured data;
a robot arm that mounts the imaging unit on the tip and adjusts the position of the tip;
a moving unit that carries and moves the robot arm;
A robot that controls the robot arm based on the tracking trajectory of the position of the predetermined target in the captured image of the imaging unit so that the tip of the robot arm does not vibrate due to the movement of the center of gravity of the moving unit or the impact at the time of contact with the ground. an arm control unit;
A mobile imaging robot system comprising:
In the mobile imaging robot system according to claim 1,
The robot arm control unit
a target tracking unit that acquires a change in the hand position of the robot arm accompanying movement of the moving unit from the captured image;
a frequency analysis unit for frequency-analyzing the time-series data of the hand position of the robot arm;
a frequency/amplitude determination unit that obtains a dominant frequency and its amplitude related to the vibration of the hand position of the robot arm based on the analysis result of the frequency analysis unit;
A vibration compensation control unit that creates an operation command for causing the robot arm to perform an operation that cancels out vibration of the hand position of the robot arm due to movement of the moving unit, based on the frequency and its amplitude obtained by the frequency/amplitude determination unit. and,
a robot arm motion control unit that controls rotation of a robot arm drive motor based on the motion command;
A mobile imaging robot system comprising:
In the mobile imaging robot system according to claim 2,
The mobile imaging robot system, wherein the vibration compensation control unit generates the operation command when the frequency/amplitude determination unit obtains the same dominant frequency and its amplitude for a predetermined time.
In the mobile imaging robot system according to claim 2,
The frequency/amplitude determination unit is
When the moving part is a multi-legged robot type, the walking cycle is the dominant frequency,
A mobile imaging robot system according to claim 1, wherein the dominant frequency is determined based on the rotational frequency of a belt or the rotational frequency of a motor for driving the belt, when the movable section is of the endless track type.
In the mobile imaging robot system according to claim 2,
The robot arm control unit further
a position information acquisition unit that acquires position information of the moving unit in a movement range of the moving unit;
a ground surface data creation unit that obtains the frequency and amplitude of vibration of the hand position of the robot arm due to unevenness or steps of the ground surface of the moving unit or obstacles based on the analysis result of the frequency analysis unit;
A mobile imaging robot system, wherein the robot arm is controlled so that a tip of the robot arm does not vibrate.
In the mobile imaging robot system according to claim 5,
The ground surface data creation unit is
Obtaining the frequency and amplitude of vibration of the robot arm's hand position due to irregularities or steps on the contact surface of the moving part when the robot arm is performing an operation to cancel the vibration of the robot arm's hand position caused by movement. A mobile imaging robot system characterized by:
In the mobile imaging robot system according to claim 5,
The ground surface data creation unit is
When a single pulse-like vibration of the hand position of the robot arm due to collision with unevenness or unevenness of the contact surface of the moving part is detected, the frequency and amplitude of the vibration of the hand position of the robot arm due to the movement of the moving part are obtained. In parallel with the above, a mobile imaging robot system characterized by obtaining the frequency and the amplitude of vibration of the position of the hand of the robot arm due to unevenness or steps of the contact surface of the moving part.
A mobile imaging robot system having an imaging unit that captures surroundings and outputs imaging data, a robot arm that mounts the imaging unit on a hand and adjusts the position of the hand, and a moving unit that mounts and moves the robot arm. A control method of
obtaining a tracking trajectory of a position of a predetermined target in an image captured by the imaging unit;
Obtaining a dominant frequency and an amplitude of a vibration frequency of the hand of the robot arm due to a movement of the center of gravity of the moving part or an impact at the time of contact with the ground based on the tracking trajectory;
A control method for a mobile imaging robot system, comprising: causing the robot arm to perform an action that cancels vibration of a hand of the robot arm based on the frequency and the amplitude thereof.
In the control method of the mobile imaging robot system according to claim 8,
Obtaining the frequency and amplitude of vibration of the robot arm's hand position due to irregularities or steps on the ground contact surface of the moving part when the robot arm is performing an operation to cancel the vibration of the robot arm's hand position due to movement;
A moving type imaging device, characterized in that the robot arm is caused to perform an operation that offsets impact when the center of gravity of the moving part moves or touches the ground, and vibration of the hand of the robot arm due to irregularities or steps on the ground contact surface of the moving part. How to control a robot system.