JP2023047376A - Three-dimensional image control method - Google Patents

Abstract

To provide a three-dimensional image control method capable of reducing a burden on a user in operation.SOLUTION: A three-dimensional image control method of a control device connected to a plurality of cameras capable of capturing a three-dimensional image, a display unit capable of displaying the three-dimensional image, and an end effector for executing work for an object acquires first distance information including information on a distance between the end effector and the object, and second distance information including information on a distance between the object and the camera, and adjusts the three-dimensional image displayed in the display unit on the basis of the second distance information when the first distance information is less than a first threshold and the second distance information is less than a second threshold.SELECTED DRAWING: Figure 5

Description

The present disclosure relates to a stereoscopic image control method in remote control.

Patent Literature 1 discloses a remote control device capable of operating a manipulator in an image on a display. This remote control device consists of a manipulator, multiple camera units that capture images of the work space, a head-mounted display that displays the captured images, an input device that allows the user to operate while viewing the image on the head-mounted display, and a robot control device. a computer; This allows the manipulator to always move in the direction indicated in the display.

JP-A-2000-42960

The present disclosure provides a stereoscopic image control method that reduces the user's operational burden.

A stereoscopic image control method according to the present disclosure includes a stereoscopic image of a control device connected to a plurality of cameras capable of capturing stereoscopic images, a display unit capable of displaying stereoscopic images, and an end effector working on an object. An image control method comprising obtaining first distance information including information about the distance between an end effector and an object and second distance information including information about the distance between the object and a camera; is less than the first threshold and the second distance information is less than the second threshold, the stereoscopic image displayed on the display unit is adjusted based on the second distance information.

The stereoscopic image control method according to the present disclosure can reduce the user's operational burden.

Conceptual diagram of a remote-controlled robot according to Embodiment 1 1 is a diagram showing a configuration example of an image adjustment device according to Embodiment 1; FIG. FIG. 4 is a diagram for explaining the distance relationship between objects recognized by the user in the first embodiment; Diagram explaining the position of the target object during remote operation 4 is a flowchart for explaining the operation of adjusting the distance between the right-eye camera and the left-eye camera; Diagram for explaining the change in the object position recognized by the operator when the image position is changed. Conceptual diagram of a robot with a high pressure washer attached as an end effector

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art.

It should be noted that the accompanying drawings and the following description are provided for a thorough understanding of the present disclosure by those skilled in the art and are not intended to limit the claimed subject matter.

(Embodiment 1)
Embodiment 1 will be described below with reference to FIGS. 1 to 6. FIG.

[1-1. composition]
[1-1-1. Schematic diagram of remote control robot]
FIG. 1 is a conceptual diagram of a remote control robot according to Embodiment 1. FIG. A robot arm 101 is a robot arm driven according to an input from a user. Each joint of the robot arm 101 is equipped with a sensor for acquiring the angle of the joint. A robot hand 111 as an end effector is attached to the tip of the robot arm 101 . The bottom of the robot arm is fixed to the fixed part 102 . By moving the robot arm 101 and the robot hand 111 according to the user's operation, the user can grasp objects around the remote control robot.

An HMD (head mounted display) 141 is a display device for displaying a stereoscopic image captured by the camera module 120 using parallax. By operating the controller 151 while viewing the stereoscopic image displayed on the HMD 141, the user can operate the robot hand 111 and the robot arm 101 as if they were at the work site. The camera module 120 is provided at a position where the object, the robot hand 111, and the robot arm 101 can be photographed, and can change the photographing range by following the movement of the HMD 141 fixed to the user's head.

FIG. 2 is a diagram illustrating a configuration example of an image adjustment device according to Embodiment 1. FIG. The camera module 120 is composed of a left-eye camera 121 , a right-eye camera 122 , and a camera adjustment section 123 . The camera adjuster 123 can change the width and angle between the left-eye camera 121 and the right-eye camera 122 . By adjusting the width and angle, it is possible to adjust the stereoscopic image displayed to the user.

The control device 131 is a device that controls the robot arm 101 and the robot hand 111 based on inputs from the controller 151 . Also, based on the distance information acquired by the proximity sensor 112 or the distance information obtained from the left-eye camera 121 and the right-eye camera 122, the distance between the cameras, the angle between the cameras, the lens magnification of the HMD, etc. are controlled. , is a device that adjusts the stereoscopic image displayed to the user.

The proximity sensor 112 is a sensor such as an ultrasonic sensor that measures the distance to the surroundings. By attaching a proximity sensor 112 to the tip of the robot hand 111, it is possible to measure a first distance between an object around the robot hand 111 and the robot hand.

The optical system adjustment unit 142 is a device that changes the magnification of the lens of the HMD 141 and the distance between the lenses of both eyes. By changing the magnification of the lenses and the distance between the lenses of the eyes, it is possible to adjust the stereoscopic image displayed to the user. Note that the adjustment of the stereoscopic image can be performed simultaneously with the adjustment by the camera adjustment unit 123 .

[1-1-2. Recognition of stereoscopic image when wearing HMD]
3A and 3B are diagrams for explaining the distance relationship between objects recognized by the user according to the first embodiment. FIG. The optical lens and display of the HMD 141 are installed at a close distance from the user's eyes. With this optical lens and display, the user perceives a virtual screen 220 at a distance Ls from the user. Distance Ls is the distance of virtual screen 220 from a point halfway between the user's eyes. A stereoscopic image (stereoscopic view) is realized by displaying a left-eye image 221 and a right-eye image 222 on the virtual screen 220 .

Due to the parallax between the object 232 recognized by the right eye displayed in the image 222 for the right eye and the object 231 recognized by the left eye displayed in the image 221 for the left eye, the user can see the distance from the user. An object 230 recognized by the user is recognized at the position of Lt. The distance Lt is the distance from the point midway between the user's eyes to the object 230 perceived by the user.

[1-2. Theme]
[1-2-1. Reason for the burden on the user due to focus adjustment operation during object recognition]
In FIG. 3, the images of the object 232 recognized by the right eye and the object 231 recognized by the left eye displayed on the virtual screen 220 are viewed by the user as the object 230 recognized by the user due to parallax. Recognized. An object 230 recognized by the user is located at a distance Lt.

At this time, the focus of the user's eyes is adjusted to the position of the distance Lt (focus state 1). In focus state 1, the actual image is displayed on the virtual screen at the distance Ls, and the image of the object on the virtual screen 220 appears blurred when the focus is on the distance Lt.

At this time, the focus of the user's eyes is adjusted to the position at the distance Ls where the actual image is displayed to sharpen the blurred image (focus state 2).

In focus state 2, the distance Lt of the object 230 recognized by the user is different from the adjusted focal length Ls. Therefore, in order to match the user's perception and the focal length, the user's eye focus is again adjusted to the position of the distance Lt (focus state 1). Forcing the user to perform eye movements due to the repetition of this focus adjustment is a burden on the user.

However, practically, if the difference between the distance Ls and the distance Lt is within an allowable range, the eye movement is suppressed, and this burden can be reduced. This allowable range varies depending on the individual difference of the user, the angle of view of the camera, the distance between the cameras, the size of the screen, and the like. In Embodiment 1, the HMD 141 fixed to the user's head is used, and the distance Ls is fixed, so the distance Lt should be considered. For example, in this embodiment, if the distance Lt is 150 [mm] or more, the burden on the user can be reduced. Let this 150 [mm] be the threshold (the second threshold for the distance Lt) that can reduce the burden on the user in this embodiment.

[1-2-2. Reproduction of object position on HMD]
FIG. 4 is a diagram for explaining the position of an object when performing remote control. The user operates an object present at a second distance Ltr from the camera module 120 while looking at the object 230 displayed on the HMD 141 and recognized by the user at a distance Lt from the user. At this time, if the stereoscopic image captured by the camera module 120 is reproduced on the HMD 141 as it is, the target object 230 recognized by the user is displayed at a distance Lt=Ltr from the user. Therefore, when the object is close to the camera module 120, that is, when the second distance Ltr is small, the distance Lt from the user displayed on the HMD 141 is also small, and eye movement due to repeated focus adjustment reduces the burden on the user. growing.

[1-3. motion]
The operation of the remote control robot configured as described above will be described below. The remote control robot performs each operation of the camera section based on the distance between the object and the robot hand. Each operation will be described in detail below.

[1-3-1. Object gripping motion and change in user's viewpoint]
In this embodiment, the object is grasped using a remote control system. The user performs remote control using the robot hand 111 and the robot arm 101 while viewing the stereoscopic image displayed on the HMD 141 . HMD 141 displays a stereoscopic image captured by camera module 120 .

At the start of the operation, the user looks over the surroundings in order to search for an object. At this time, the user's point of view is not fixed on a specific target, but is looking far away.

Next, the user recognizes an object at a distance Ltr (see FIG. 4) from the camera module 120 through the HMD 141, checks the surroundings, and moves the robot hand 111 toward the object while avoiding collision. is changed, the robot arm 101 is operated. When the first distance, which is the distance between the robot hand 111 and the object, is approached by the operation of the robot arm 101, the user can grasp the detailed positional relationship between the robot hand 111 and the object. A viewpoint is fixed on the hand 111 . A first distance between the robot hand 111 and the object when the user sets a viewpoint on the object and the robot hand 111 is set as a first threshold.

The timing at which the robot hand 111 starts working is when the user sets the viewpoint on the object and the robot hand 111 . That is, the first threshold is the distance at which it is determined that the robot hand 111 is working.

At this time, if the distance Ltr between the object and the camera module 120 is smaller than the second threshold, eye movement occurs due to repeated focus adjustment, causing a burden on the user. It becomes difficult to continue. Therefore, when the distance is equal to or greater than the second threshold, it is a distance at which stereoscopic vision can be continued, and when it is less than the second threshold, it is a distance at which it is difficult for the user to continue stereoscopic vision.

After grasping the object, while paying attention to the grasped object, it moves the object, works using the object, and releases the object. Immediately after releasing the object, the robot hand moves away while paying attention to the vicinity of the object. 's viewpoint moves.

In other words, when the robot hand 111 grips the target object and directly works on the target object, the user's viewpoint is on the target object, and the robot hand 111 searches for the target object. After release, if the user does not perform any direct work on the object, such as checking the surroundings, the user's viewpoint is not on the object. Then, when the user's viewpoint is on an object and the distance Ltr from the camera module 120 to the object is less than the second threshold, as described above, repeated focus adjustment causes eye movement, which is a burden on the user. will occur.

[1-3-2. Adjustment of a stereoscopic image according to the distance to the object and the user's viewpoint]
As described above, when remote control is performed, the user's viewpoint changes depending on the distance of the object itself from the camera module 120 and the details of the operation during remote control. Along with this, depending on the content of the operation during remote control, eye movement occurs due to repeated focus adjustment by the user, and the stereoscopic image is adjusted in order to reduce the burden. In this embodiment, as an example of a method of adjusting a stereoscopic image, a case of adjusting the inter-camera distance will be described.

FIG. 5 is a flowchart for explaining the operation of adjusting the distance between the right-eye camera 122 and the left-eye camera 121. FIG. The user operates the controller 151 while viewing the information (stereoscopic image) displayed on the HMD 141 to operate the robot arm 101 and the robot hand 111 .

While the user is performing remote control, the control device 131 repeatedly performs the processing shown in FIG. A first distance between the robot hand 111 and the object is acquired by the proximity sensor 112 mounted on the fingertip of the robot hand 111 (S501).

When the first distance between the robot hand and the object becomes equal to or less than the first threshold value to grab the object with the robot hand (Yes in S502), the second distance between the object and the camera module 120 Ltr is acquired (S503).

Here, for example, the second distance Ltr is calculated by measuring the position of the object according to the principle of triangulation using the right-eye camera 122 and the left-eye camera 121 and using the measured position of the object.

If the second distance is less than the second threshold (Yes in S504), the camera adjustment unit 123 adjusts the distance between the left-eye camera 121 and the right-eye camera 122 to the object 230 recognized by the user. The distance Lt of the user is reduced so as to be greater than or equal to the second threshold (S505). The principle of adjusting the distance Lt between the object 230 recognized by the user and the user will be described later.

The stereoscopic image adjusted by the camera adjustment unit 123 is presented to the user through the HMD 141 (S506).

On the other hand, when the distance between the robot hand and the object exceeds the threshold (No in S502), or when the distance between the robot hand 111 and the right-eye camera 122 exceeds the threshold (No in S504), the inter-camera The distance is not adjusted, and the unadjusted stereoscopic image is presented to the user through the HMD 141 (S506).

Here, a method of adjusting the distance Lt between the object 230 recognized by the user and the user will be described with reference to FIG. FIG. 6 is a diagram for explaining the change in the object position recognized by the user when the image position is changed.

When the inter-camera distance is not adjusted, it becomes as shown in FIG. The object 232 recognized by the right eye displayed in the image 222 moves to the right, and the object 231 recognized by the left eye displayed in the image 221 for the left eye moves left.

As a result, although the distance Ltr between the actual object and the camera module 120 remains unchanged, the object 230 recognized by the user moves toward the virtual screen 220, resulting in an increase in the distance Lt. That is, the distance Lt can be increased by reducing the inter-camera distance, and even if the actual distance Ltr is small, the distance Lt of the object 230 recognized by the user is always adjusted to be equal to or greater than the second threshold. be able to.

[1-4. effects, etc.]
[1-4-1. Effect of remote control system]
As described above, in this embodiment, the remote control system includes camera module 120 , robot hand 111 , proximity sensor 112 , robot arm 101 , HMD 141 and controller 151 . The camera module 120 captures stereoscopic images around the robot arm 101 .

Based on the second distance information between the object and the camera module 120, the camera adjustment unit 123 adjusts the distance between the left-eye camera 121 and the right-eye camera 122 so that the distance Lt of the object recognized by the user is the second distance. Reduce it so that it is equal to or greater than the threshold.

Therefore, it is possible to suppress the user's eye movement due to repeated focus adjustment, and reduce the burden on the user during operation.

(Other embodiments)
As described above, Embodiment 1 has been described as an example of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with modifications, replacements, additions, omissions, and the like. Therefore, other embodiments will be exemplified below.

In Embodiment 1, changing the inter-camera distance has been described as an example of a method for adjusting a stereoscopic image. Any method for adjusting the stereoscopic image may be used as long as the stereoscopic image recognized by the user can be adjusted. Therefore, the method of adjusting stereoscopic images is not limited to changing the inter-camera distance.

For example, changing the inter-camera angle may be used as a method of adjusting the stereoscopic image. The camera is rotated in a direction that increases the angle formed by the optical axis so that the distance Lt of the object recognized by the user is greater than or equal to the second threshold. Specifically, the left-eye camera 121 is rotated clockwise, and the right-eye camera 122 is rotated counterclockwise. As a result, as in the case where the inter-camera distance is adjusted, the object 232 recognized by the right eye displayed in the right-eye image 222 moves to the right, and the left-eye image displayed in the left-eye image 221 moves to the right. , the object 231 to be recognized moves to the left. Therefore, the distance Lt of the target object recognized by the user becomes equal to or greater than the second threshold, thereby reducing the burden on the user.

Also, changing the magnification of the camera lens may be used as a method of adjusting the stereoscopic image. The lens magnifications of the left-eye camera 121 and the right-eye camera 122 are reduced so that the distance Lt of the object recognized by the user is greater than or equal to the second threshold. As a result, the object 232 displayed in the right-eye image 222 and recognized by the right eye and the object 231 displayed in the left-eye image 221 and recognized by the left eye move closer to the center. Therefore, the distance Lt of the target object recognized by the user becomes equal to or greater than the second threshold, thereby reducing the burden on the user.

Also, changing the magnification of the lens of the HMD 141 may be used as a method of adjusting the stereoscopic image. The lens magnification of the HMD lens is increased so that the distance Ls of the virtual screen 220 is reduced. As a result, the difference between the distance Ls to the virtual screen and the distance Lt of the object recognized by the user can be reduced, the user's eye movement is suppressed, and the user's burden is reduced.

Also, the linear transformation processing of the display image may be used as a method of adjusting the stereoscopic image. The right-eye image 222 and the left-eye image 221 displayed on the virtual screen are brought closer to each other. As a result, the distance between the object 232 recognized by the right eye and the object 231 recognized by the left eye becomes smaller, and the distance Lt of the object recognized by the user becomes equal to or greater than the second threshold. Reduced burden.

Also, the image for the right eye 222 and the image for the left eye 221 displayed on the virtual screen are reduced to adjust the stereoscopic image. As a result, the distance between the object 232 recognized by the right eye and the object 231 recognized by the left eye becomes smaller, and the distance Lt of the object recognized by the user becomes equal to or greater than the second threshold. Reduced burden.

Also, the image may be adjusted by combining these techniques.

Although the distance between the object and camera module 120 is used as the second distance in Embodiment 1, the second distance may be the distance between robot hand 111 and camera module 120 . In other words, the second distance may be the distance from the camera module 120 to the target of the user's attention. In particular, when the first distance is small, it is useful because the robot hand 111 and the object are located close to each other.

Note that when the distance between the robot hand 111 and the camera module 120 is used as the second distance, the relative position to the camera module 120 can be used instead of triangulation. Specifically, it can be calculated from joint angle information of the robot arm 101 and component dimension information of the robot arm 101 , the robot hand 111 and the camera module 120 .

In Embodiment 1, the HMD has been described as a device that presents a stereoscopic image to the user. A method for presenting a stereoscopic image to the user may be any method as long as it enables the user to view stereoscopically by parallax. Therefore, devices that present stereoscopic images to users are not limited to HMDs. However, if an HMD is used as a device for presenting a stereoscopic image to a user, it is possible to enhance the feeling of immersion and bring the feeling of operation closer to that of the actual site. Specifically, a 3D display may be used as a device that presents a stereoscopic image to a user. If a 3D display is used as a device for presenting a stereoscopic image to a user, it is possible to reduce the weight of the device worn by the user.

In Embodiment 1, as an example of the first threshold, the distance between the robot hand 111 and the object at which the user's viewpoint changes has been described. The first threshold should be sufficient if it can be determined that the user's viewpoint has moved to the vicinity of the end effector. For example, when gripping an object with the robot hand, the opening width of the robot hand may be equal to or greater than the width of the object when the robot hand is moved in the opening direction. Alternatively, the time when the hand is moved in the closing direction may be set as the first threshold. Alternatively, the time when the distance between the object and the robot hand becomes zero after gripping, that is, the time when the object and the robot hand are in contact may be set as the first threshold.

Also, the first threshold may be determined as long as it can be determined that the user is paying attention to the vicinity of the end effector. For example, when a screwdriver is gripped by a robot hand and the screw is tightened, the first threshold is set to zero. In other words, the adjustment of the stereoscopic image is performed only when the device is held. If the first threshold is set to zero, it is possible to use an inexpensive contact sensor without using a ranging sensor.

In Embodiment 1, the robot hand has been described as an example of the end effector. Any end effector may be used as long as it works on an object. Moreover, the work is not limited to gripping, and the setting method of the first distance changes according to the work.

For example, another end effector may be a high pressure washer. FIG. 7 shows a conceptual diagram of the robot when a high-pressure washer is attached as an end effector. The high-pressure washer 114 is attached to the tip of the robot arm 101 and cleans the object 181 to be cleaned by jetting water. High pressure washers can only act in the direction of the water jet. Therefore, it is considered that the viewpoint moves when there is an object to be cleaned within a certain distance in the water jetting direction, and the viewpoint does not move when another object approaches other than the jetting direction.

In other words, not only the distance between the end effector (high-pressure washer) and the object, but also the direction of the tip of the end effector (in the case of a high-pressure washer, the direction in which water is sprayed) is used to determine the object (here, the object to be cleaned). 181) is determined.

When cleaning with a high-pressure cleaner is set as the work of the end effector, the first distance information can be the distance to the target existing in the water jet direction. The first threshold is the washable distance in the water jetting direction. However, the first threshold follows the water pressure setting of the high pressure washer 114 and can be changed.

Note that the above-described embodiment is for illustrating the technology in the present disclosure, and various changes, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

INDUSTRIAL APPLICABILITY The present disclosure is applicable to systems that adjust stereoscopic images according to the distance between an object and a camera. Specifically, the present disclosure can be applied to image capturing of remote-controlled robots, working robots, and the like.

101 Robot Arm 102 Fixed Part 111 Robot Hand 112 Proximity Sensor 114 High Pressure Washer 120 Camera Module 121 Left Eye Camera 122 Right Eye Camera 123 Camera Adjustment Part 131 Control Device 141 HMD (Head Mounted Display)
142 optical system adjustment unit 151 controller 181 cleaning object 220 virtual screen 221 image for left eye 222 image for right eye 230 object recognized by user 231 object recognized by left eye 232 object recognized by right eye

Claims (12)

A stereoscopic image control method for a control device connected to a plurality of cameras capable of capturing stereoscopic images, a display unit capable of displaying the stereoscopic images, and an end effector working on an object,
Acquiring first distance information including information about the distance between the end effector and the object, and second distance information including information about the distance between the object and the camera;
When the first distance information is less than a first threshold and the second distance information is less than a second threshold, the stereoscopic image displayed on the display unit is adjusted based on the second distance information. A stereoscopic image control method characterized by: 2. The stereoscopic image according to claim 1, wherein the first threshold is a distance between the end effector determined to be working on the object and the object. control method. 3. The stereoscopic image control method according to claim 2, wherein the distance between the end effector determined to be working and the object is zero. 4. The second threshold is a distance between the camera and the object that allows a user to continue stereoscopic viewing on the display unit. The stereoscopic image control method according to item 1. 4. Any one of claims 1 to 3, wherein the second threshold value is a predetermined value based on a distance between the display position of the display unit and the user and a distance between the user and the position of the object recognized by the display unit. 1. The stereoscopic image control method according to 1. The first distance information includes direction information indicating the direction of the tip of the end effector, the first distance information is less than a first threshold, the second distance information is less than a second threshold, and the 6. The stereoscopic image according to any one of claims 1 to 5, wherein the stereoscopic image displayed by the display unit is adjusted based on the second distance information when the direction information indicates a predetermined direction. control method. 7. The stereoscopic image control method according to any one of claims 1 to 6, wherein the adjustment of the stereoscopic image is controlled so as to reduce the distance between the plurality of cameras. 8. The stereoscopic image control method according to any one of claims 1 to 7, wherein the adjustment of the stereoscopic image is controlled so as to increase an angle formed by optical axes of the plurality of cameras. 9. The stereoscopic image control method according to any one of claims 1 to 8, wherein the adjustment of the stereoscopic image is controlled so as to reduce magnification of lenses of the plurality of cameras. The adjustment of the stereoscopic image is characterized in that the plurality of images used for the stereoscopic image are moved in a direction to bring them closer together, or the sizes of the plurality of images used for the stereoscopic image are reduced. Item 9. The stereoscopic image control method according to any one of Items 9. The display unit is a head-mounted display,
11. The stereoscopic image control method according to any one of claims 1 to 10, wherein the adjustment of the stereoscopic image is performed by controlling an optical system of the head mounted display. 12. The stereoscopic image control method according to claim 11, wherein the optical system is controlled to increase the magnification of a lens of the head-mounted display.

Images