JP6474905B2 - Remote operation system and operation support system - Google Patents

Description

  The present invention relates to a remote operation system for operating a moving body such as a work machine or a mobile robot from a remote location.

  There is a need to remotely control a moving body such as a work machine or a mobile robot. For example, in an environment where workers are exposed to danger, such as disaster recovery work, a driver does not board the work machine, and remote operation is performed by an operator from an operation facility with an unmanned work machine installed at a remote location. The system is activated. The same applies to mobile robots.

  When the mobile unit is operated remotely as described above, a camera is installed on the mobile unit, and the captured camera video is transmitted to the remote control facility by wired or wireless communication, and the video is displayed on the monitor in the control facility. Is projected. The remote operator operates the moving body with reference to the video on the monitor.

  In that case, in general, an image on a monitor makes it difficult to perceive the perspective as compared to when riding on a work machine, and it is relatively difficult to recognize the distance and shape to the work object. As a countermeasure, there is a method in which a camera car having a camera and communication means is arranged at the work site, the camera car photographs the work machine and the periphery of the work object from the side, and the image is transmitted to the operation equipment. According to this, depth information is given to the remotely operated operator in the form of a side image, and the distance and shape to the work object can be easily recognized. However, a method of separately preparing such a camera car and operating it is not practical because it increases costs and labor.

  As methods for providing depth information to an operator using only a work machine without using a camera car, methods described in Patent Documents 1 and 2 have been proposed.

  In Patent Document 1, a multi-view camera is mounted on a remotely operated work machine, a distance image is generated from a plurality of captured images obtained from the multi-view camera based on a predetermined parallax, and each pixel is set according to the distance. A remote control support device that synthesizes a pseudo three-dimensional image by assigning different colors and transmits the synthesized three-dimensional image to the remote control device is disclosed.

  In Patent Document 2, a three-dimensional scanner is mounted on a construction machine to acquire three-dimensional distance data of a work object, and this is displayed on a scanner image display unit of a remote control device. Discloses the display of three-dimensional distance data at different angles.

JP-A-11-213154 Japanese Patent Laying-Open No. 2015-043488

  The method of Patent Document 1 can give a pseudo three-dimensional feeling to an image with a viewpoint on a work machine, but the accuracy of depth information given to an operator is limited. For this reason, compared with the method of presenting an image from the side using a camera car, the sense of distance to the work object cannot be accurately grasped, resulting in low work accuracy and poor work efficiency. End up.

  In the method of Patent Document 2, it is possible to synthesize an image of an arbitrary angle by acquiring three-dimensional distance data (distance image) of a work object using a three-dimensional scanner and using the acquired distance image. is there. However, the distance image that can be measured from the construction machine (work machine) is naturally limited to an area that can be seen from the three-dimensional scanner. For this reason, when the work object is shielded by a part (for example, an arm part) of the work machine, it is impossible to generate an image of the work object viewed from another angle. As a result, for example, an image from another angle becomes difficult to understand, so that the sense of distance of the work object cannot be accurately grasped, work accuracy is low, and work efficiency is poor.

  An object of the present invention is to provide a remote control system that displays an image that can accurately grasp the sense of distance to a work object even when the work object is viewed from an arbitrary angle.

  The present invention is a remote control system for operating a moving body that performs a work on a work target by a remote control device. The moving body includes a three-dimensional camera that captures the front of the moving body and a three-dimensional camera. Based on the obtained 3D point cloud data, a free viewpoint image generation unit that generates a point cloud rendering image in which the work object is captured at an angle different from the viewpoint direction of the 3D camera, and movement based on the 3D model data of the moving object A CG image generation unit that generates an artificial image of the body, a superimposed image synthesis unit that superimposes the artificial image on the point cloud rendering image, and a communication unit that transmits the free viewpoint synthesized image superimposed by the superimposed image synthesis unit to the remote control device; . Further, the remote control device includes a communication unit that receives a free viewpoint composite image transmitted from a mobile body, a display unit that displays the received free viewpoint composite image, and an operation input unit that instructs the mobile body to perform an operation. It is set as the structure provided with these. Here, the superimposed image composition unit is further configured to generate an artificial image of the work object based on the three-dimensional model data of the work object and superimpose it on the point cloud rendering image.

  According to the present invention, since the operator can remotely operate the work machine while referring to a free viewpoint composite image from an arbitrary angle (for example, upward or side), an objective distance to the work object is obtained. A feeling, the size of an object, etc. can be grasped easily, and the operativity of remote work can be improved remarkably.

It is a block block diagram of a remote control system. (Example 1) It is a figure which shows the relationship between the various programs used for image processing, and various data. It is a block diagram of the working machine to which the remote operation system is applied. It is a figure which shows the working state by the working machine. 6 is a diagram showing an image displayed on the remote control device 200. FIG. It is explanatory drawing of the synthetic | combination method of a virtual upper surface image. It is explanatory drawing of the synthetic | combination method of a virtual side surface image. It is a flowchart which shows the production | generation method of a free viewpoint synthetic | combination image. It is a figure which shows the virtual upper surface image which superimposed the artificial image of the work target object. (Example 2) It is a figure which shows the virtual side surface image which superimposed the artificial image of the work target object. It is a flowchart which shows the production | generation method of a free viewpoint synthetic | combination image. It is a figure which shows an example of the working state by a working machine. Example 3 FIG. 10 is a diagram illustrating an image synthesized by the method of the second embodiment. It is a figure which shows the image displayed restrict | limiting the display area. It is a flowchart explaining the production | generation of a work object model. (Example 4) It is a figure which shows the example which changes and displays the superimposition degree of an artificial image. (Example 5) It is a figure which shows the example whose moving body is a mobile robot. It is a figure which shows the example of the virtual image in the case of a mobile robot.

  Embodiments of the present invention will be described below with reference to the drawings. In the following embodiments, a hydraulic excavator will be described as an example of a work machine that performs remote operation.

  FIG. 1A is a block diagram of a remote operation system, and FIG. 1B is a diagram showing the relationship between various programs and various data used for image processing in this embodiment. FIG. 2 is a configuration diagram of a work machine to which the remote operation system is applied. In the present embodiment, the work machine 100 is an unmanned hydraulic excavator, and is operated by an operator (operator) who rides on the remote control device 200 installed in a remote place. Of course, the work machine 100 may be various work machines other than the hydraulic excavator. As shown in FIG. 1A, the remote operation system includes a part mounted on a work machine 100 to be remotely operated and a part mounted on a remote operation device 200 operated by an operator. Hereinafter, the configuration of each unit will be described with reference to FIGS. 1A, 1B, and 2. FIG.

  First, the configuration of the work machine 100 will be described. The work machine 100 includes a main control unit 101, a camera 102, a three-dimensional camera 103, an arm unit 105, an arm angle sensor 104, a traveling unit 106, and a wireless communication unit 107.

  The arm portion 105 is a portion that performs various operations such as excavation as a hydraulic excavator, and the traveling portion 106 is a portion that causes the work machine 100 to travel. The arm angle sensor 104 acquires the angle of each joint of the arm unit 105 in real time.

  A camera 102 is installed in the operation seat (unmanned) and photographs the front of the work machine 100. Here, the front means a direction in which the operator sees when the work machine is operated by manned boarding. Further, a three-dimensional camera 103 is installed to capture a depth image in front of the work machine 100. The three-dimensional camera 103 is a camera that can acquire depth information in real time. For example, a stereo camera is used. Alternatively, a TOF (Time of Flight) camera, an optical coding camera, or a 3D laser scanner may be used. It is assumed that the 3D camera can acquire not only depth information but also color information. The wireless communication unit 107 performs wireless communication with the remote control device 200.

  The main control unit 101 controls the work machine 100. The main control unit 101 includes an information processing device 111, a storage device 120, and 3D model data 131 stored in, for example, an HDD, and can be configured by a general-purpose computer. In the information processing apparatus 111, a processor (not shown) performs various processes using programs and data stored in the storage device 120. The storage device 120 uses a 3D point cloud generation program 121 that generates 3D point cloud data 142 from 3D camera data 141 that is input data from the 3D camera 103, and a free viewpoint using the 3D point cloud data 142. Point-of-view image generation program 122 for generating point cloud rendering image 144 from CG, computer image generation program 123 for generating computer graphics (CG) image (artificial image) 145 from three-dimensional model data 131 of the moving object, point cloud rendering image The superimposition image synthesis program 124 that superimposes the artificial image 145 on the 144 to synthesize the free viewpoint composite image 125 and the arm unit / travel unit control program 129 that controls the arm unit and the travel unit are stored. The storage device 120 also includes 3D camera data 141, 3D point cloud data 142, virtual camera data 143, a point cloud rendering image 144, a population image 145, moving body posture data 146, and a free viewpoint. The composite image 125 is stored. FIG. 1B shows the relationship between the various programs handled by the main control unit 101 and various data.

  Further, the main control unit 101 transmits a front camera image and a free viewpoint composite image 125 to the remote control device 200 via the wireless communication unit 107. The main control unit 101 controls the arm unit 105 and the traveling unit 106 in accordance with an operation command received from the remote operation device 200 received via the wireless communication unit 107.

  Next, the configuration of the remote operation device 200 will be described. The remote operation device 200 includes a main control unit 201, a wireless communication unit 202, a camera image display unit 203, a free viewpoint composite image display unit 204, a free viewpoint composite image display control unit 205, an operation input unit 206, and an operation seat 209.

  The main control unit 201 is a part that controls the remote operation device 200 and can be configured by a general-purpose computer having an information processing device (not shown) including a processor and a storage device (not shown). The wireless communication unit 202 performs wireless communication with the work machine 100, receives a front camera image and a free viewpoint composite image 125 from the work machine 100, and the camera image display unit 203 and the free viewpoint composite image display unit 204 respectively. indicate. The free viewpoint composite image display control unit 205 sets the display method of the free viewpoint composite image according to the input from the operator, and transmits the information to the work machine 100 via the wireless communication unit 202.

  The operator operates the work machine 100 from the operation input unit 206 while referring to the images of the camera image display unit 203 and the free viewpoint composite image display unit 204. The operation command input to the operation input unit 206 is processed by the main control unit 201 and transmitted to the work machine 100 via the wireless communication unit 202. Note that transmission / reception between the work machine 100 and the remote control device 200 may be performed by wire. Further, the camera image display unit 203 and the free viewpoint composite image display unit 204 may be configured by one display unit, and the front camera image and the free viewpoint composite image 125 may be displayed on the single display unit.

  3A and 3B are diagrams showing image display examples by the remote operation system. FIG. 3A shows a working state by the work machine 100, and FIG. 3B shows an image displayed on the remote operation device 200. FIG.

  As shown in FIG. 3A, the work machine (hydraulic excavator) 100 is in a state where the bucket 105 a is moved and the dump 301 is being discharged. An object of work performed by the work machine 100 is called a “work object”. In this case, the dump 301 is the work object. In addition to this, a space in which work is performed is referred to as a “work environment”. In this case, the ground 302 or the like is the work environment.

  FIG. 3B is an image display example on the camera image display unit 203 and the free viewpoint composite image display unit 204 of the remote control device 200. A front camera image 311 photographed by the camera 102 is wirelessly received and displayed on the camera image display unit 203 on the lower side of the drawing. The front camera image 311 shows the work object (dump) 301, the work environment (ground) 302, the arm part 105 of the work machine, the bucket 105a, and the like.

  In the conventional remote operation, the operator remotely operates the work machine while viewing the front camera image 311. However, since the sense of depth is poor with only the front camera image 311, work efficiency may be reduced. In order to solve this, in this embodiment, a virtual virtual top image 312 and a virtual virtual side image 313 are displayed on the free viewpoint composite image display unit 204 on the upper side of the drawing.

  The virtual top surface image 312 and the virtual side surface image 313 wirelessly receive the free viewpoint composite image 125 synthesized by the main control unit 101 based on the 3D point cloud data 142 photographed by the 3D camera 103 of the work machine 100. Is displayed. The virtual top image 312 is a virtual composite of images of the work machine 100 and the work object 301 viewed from above, and the virtual side image 313 is a view of the work machine 100 and the work object 301 from the side. This is a virtual composite of images. Here, upward means a high direction parallel to the vertical direction with respect to the work machine. Further, the side means a direction perpendicular to the traveling direction and the vertical direction of the work machine. By displaying the virtual top surface image 312 and the virtual side surface image 313 in this manner, the operator can be given a sense of depth of the work object 301 and work efficiency can be improved.

Hereinafter, a method for synthesizing the virtual upper surface image 312 and the virtual side surface image 313 will be described.
FIG. 4 is an explanatory diagram of a method for synthesizing the virtual upper surface image 312 and its procedure is shown in (a) to (i).

  FIG. 3A is a top image when it is assumed that the work state shown in FIG. 3A is taken from above the work machine 100, and the generation of such an image is a target (ideal). However, in order to obtain such an ideal image, a camera must be installed at a position far away above the work machine 100, which is practically difficult. For this reason, the three-dimensional camera 103 is mounted on the work machine 100, three-dimensional point cloud data such as the work object 301 is acquired, and an image viewed virtually from above is generated from the depth information. An image generated from the three-dimensional point cloud data is called a point cloud rendering image.

  (B) shows a situation where the front is photographed from the three-dimensional camera 103 mounted on the work machine 100. In this case, a substantially triangular area 401 including the work object 301 is an area that can be photographed by the three-dimensional camera 103. (C) shows an area 401 that can be photographed from the three-dimensional camera 103. The target top surface image as shown in (a) cannot be generated only by the imageable area 401 shown in (c). This is because the region 401 includes only a part of the work machine 100 (for example, the arm unit 105). Further, only a part of the work environment 302 is included.

  Further, even within the area 401 of (c), the 3D point cloud data of the invisible area cannot be acquired from the 3D camera 103. In this work state, a part of the work object 301 is hidden by the bucket 105a of the work machine 100 (see FIG. 3A). An area 402 shown in (d) is an area of the work target 301 hidden by the bucket 105 a, and the depth cannot be measured from the three-dimensional camera 103.

  Similarly, the work object 301 hides a part of the work environment 302. For this reason, the area hidden by the work object 301 cannot be measured. An area 403 shown in (e) is an area of the work environment 302 hidden by the work object 301, and this area 403 also cannot measure the depth from the three-dimensional camera 103.

  Further, with respect to the arm unit 105 of the work machine 100, only the portion visible from the three-dimensional camera 103, that is, the left side of the arm unit 105 is measured. As a result, as shown in (f), the region 404 (dashed line portion) on the right side of the arm unit 105 cannot be measured.

  (G) shows the upper surface point cloud rendered image 144 that can be generated from the image data from the three-dimensional camera 103 in summary. In this image, the areas 402, 403, and 404 that cannot be measured are excluded from the imageable area 401. For this reason, compared with the ideal top image shown in (a), a part of the image of the work machine 100 or the work object 301 is missing, and the positional relationship between the work machine 100 and the work object 301 can be grasped. It becomes difficult.

  In order to solve this problem, in this embodiment, an artificial image (CG image) 145 of the work machine 100 that is missing from the image (g) is generated and superimposed. In order to generate the artificial image of the work machine 100, the three-dimensional model data 131 of the work machine 100, the moving body posture data 146 such as the angle of the joint part of the arm unit 105, and the CG camera for generating the artificial image are generated. There may be virtual camera data 143 that is information. (H) shows the generated artificial image 100 ′ (image viewed from above) of the work machine 100.

  (I) shows an example of an image obtained by superimposing the artificial image 100 ′ of the work machine 100 in (h) on the 3D image shown in (g). This makes it easier to grasp the positional relationship between the work machine 100 and the work object 301 than in the case of only the point cloud rendering image (g).

  FIG. 5 is an explanatory diagram of a method for synthesizing the virtual side image 313, and the procedure is shown in (a) to (e).

  (A) is a side image when it is assumed that the work state shown in FIG. 3A is taken from the side of the work machine 100, and the generation of such an image is a target (ideal). In order to obtain such an ideal image, a camera must be installed at a position far away from the side of the work machine 100, which is practically difficult. Also in this case, a side image is generated using the three-dimensional camera 103 mounted on the front surface of the work machine 100.

  (B) shows the area | region 501 which can image | photograph from the three-dimensional camera 103. FIG. In this case, the vertical sector area 501 including the work object 301 is an area that can be imaged by the three-dimensional camera 103. (C) shows a region 501 that can be photographed from the three-dimensional camera 103. Only the shootable area 501 shown in (c) does not include the entire work machine 100, and a target side image as shown in (a) cannot be generated.

  Furthermore, even within the area 501 of (c), the 3D point cloud data of the invisible area from the 3D camera 103 cannot be measured. (D) is a diagram showing a region that cannot be measured by the three-dimensional camera 103. When the three-dimensional camera 103 is mounted on the left side of the work machine 100, for example, the work object 301 and the arm part 105 of the work machine are mainly measured only on the left side, and the side images viewed from the right side are broken lines. As indicated by 502 and 503, it becomes incomplete. In addition, since the work object 301 hides a part of the work environment 302, the hidden area 504 cannot be measured. For this reason, compared with the ideal side image shown in (a), a part of the image is lost, and it is difficult to grasp the positional relationship between the work machine 100 and the work object 301.

  In order to solve this problem, in this embodiment, the missing artificial image (CG image) 145 of the work machine 100 is superimposed on the image (d). (E) shows an example of an image in which an artificial image 100 ′ (image viewed from the side) of the work machine 100 is generated and superimposed on the point cloud rendered image 144 shown in (d). This makes it easier to grasp the positional relationship between the work machine 100 and the work object 301 than in the case of only the point cloud rendering image 144 (d).

  Although the top image in FIG. 4 and the side image in FIG. 5 have been described as virtual images, the virtual image can be generated not only for this but also when viewed from an arbitrary direction (camera angle). Hereinafter, an image generated by changing the viewpoint direction is also referred to as a “free viewpoint synthesized image”. Needless to say, an image from an arbitrary viewpoint direction can also be generated for the superimposed artificial image.

FIG. 6 is a flowchart showing a method for generating a free viewpoint composite image. This processing is executed by the information processing apparatus 111.
In step S <b> 601, data is input from the three-dimensional camera 103. This input data varies depending on the type of the three-dimensional camera. For example, in the case of a stereo camera, two left and right images are input to the three-dimensional camera data 141.
In S602, a distance image is calculated from the three-dimensional camera data 141 input from the three-dimensional camera. The distance image is obtained by giving distance information d (x, y) from the camera to each pixel (x, y).

  In step S <b> 603, the arm angle sensor 104 of the work machine 100 detects the arm angle of the arm unit 105 and inputs it to the moving body posture data 146. In the hydraulic excavator shown in FIG. 2, the arm angle includes a turning angle, a boom angle, an arm angle, and a bucket angle, whereby the current posture of the hydraulic excavator can be defined.

  In S604, the virtual camera data 143 for generating the free viewpoint image is referred to and set as the virtual camera data of the free viewpoint image generation program 122 and the CG image generation program 123. The virtual camera data 143 includes information on the position, direction, and angle of view of the virtual camera for generating a virtual image. The virtual camera data 143 is set by the operator from the free viewpoint composite image display control unit 205 of the remote operation device 200. In the example of FIG. 3B, information of two virtual cameras, that is, virtual camera data from the top surface corresponding to the virtual top surface image 312 and virtual camera data from the side surface corresponding to the virtual side image 313 is stored in the virtual camera data 143. Yes. In step S <b> 604, first, virtual camera data from the upper surface corresponding to the virtual upper surface image 312 is referenced, and the virtual camera data of the free viewpoint image generation program 122 and the CG image generation program 123 is set.

  In step S605, a point cloud rendering image 144 is generated using the distance image. In this process, three-dimensional coordinates (X, Y, Z) and pixel color information C are obtained for each pixel of the distance image d (x, y) calculated in S602 and expressed as point cloud data. . The point cloud data 142 is rendered using the set virtual camera data 143 to generate a point cloud rendered image from the upper surface. FIG. 4G shows the generated upper surface point cloud rendered image. In the point cloud rendered image generated only from the point cloud data, image defect regions 402 to 404 are generated due to occlusion between objects.

  In addition, in order to generate a free viewpoint image using a distance image, information on the installation position and direction of the three-dimensional camera 103 is necessary. The point cloud data acquired from the 3D camera is defined in the local coordinate system of the 3D camera. This point cloud data needs to be converted into the coordinate system of the work machine 100. For this purpose, the position and direction of the three-dimensional camera are measured in the coordinate system of the work machine, and a coordinate transformation matrix is obtained from the position and direction to perform coordinate transformation.

  In S606, an artificial image 145 of the work machine 100 in the defect area is generated using the three-dimensional model data 131 and the moving body posture data 146 such as the arm angle acquired in S603. The three-dimensional model data 131 stores CG model data 131 of the work machine 100 in advance. The angle of the arm portion can be arbitrarily set for the CG model data 131, thereby generating an artificial image viewed from the upper surface in the same posture as the current work machine. FIG. 4H shows the generated artificial image (top CG image) 100 'of the work machine.

  In S607, the free viewpoint composite image 125 is generated by superimposing the point cloud rendering image from the point cloud data generated in S605 and the artificial image 100 'of the work machine 100 generated in S606. In this superimposition processing, the point cloud rendering image and the artificial image of the work machine 100 are superimposed in consideration of the depth. This processing can be realized by hidden surface removal processing using the Z buffer method. FIG. 4 (i) shows an image of the superimposition result. As shown in FIG. 4 (i), as a result of superimposing the artificial image of the work machine 100, the work machine 100 and the work machine 100 are compared with the point cloud rendering image display using only the point cloud data shown in FIG. It becomes easy to grasp the relationship between the work objects 301.

In step S608, the free viewpoint composite image 125 is transferred. In this example, the virtual top image (FIG. 4 (i)) is transferred to the remote control device 200.
In step S609, it is determined whether there is other virtual camera information. In the example of FIG. 3B, virtual camera information from the side surface corresponding to the virtual side image 313 exists. In this case, the process returns to S604, and the virtual camera information from the side is set in the virtual camera data of the free viewpoint image generation program 122 and the CG image generation program 123.

  Hereinafter, the side point cloud rendering image generated in S605 is as shown in FIG. 5D, and the virtual side image 313 in which the artificial image 100 ′ of the work machine 100 is superimposed in S607 is shown in FIG. 5E. It becomes like this. Also in this case, as a result of superimposing the artificial image of the work machine 100, the relationship between the work machine 100 and the work object 301 can be easily grasped.

In step S608, the free viewpoint composite image 125 is transferred. In this example, the virtual side image (FIG. 5E) is transferred to the remote operation device 200.
In step S609, it is determined whether there is other virtual camera information. In the example of FIG. 3B, the process ends because there is no other virtual camera information.

  The composite image transferred in S608 is displayed on the free viewpoint composite image display unit 204 of the remote control device 200 as shown in FIG. 3B. On the other hand, an image from the front camera 102 is displayed on the camera image display unit 203 of the remote operation device 200.

  As described above, in the first embodiment, the CG image (artificial image) of the work machine is superimposed and displayed on the point cloud rendering image obtained from the three-dimensional camera, so that the positional relationship between the work machine and the work object can be grasped. There is an effect that it becomes easy.

  In the second embodiment, not only the artificial image of the work machine in the defect area but also the artificial image of the work object is superimposed on the point cloud rendering image generated from the three-dimensional camera.

  Since the configuration of the remote operation system is the same as that of the first embodiment (FIGS. 1A, 1B, and 2), the difference in operation will be described below. The 3D model data 131 of the work machine 100 stores not only the CG model data of the work machine 100 but also the CG model data of the work object (dump) 301.

  7 and 8 are diagrams showing an example in which an artificial image of a work target is added and superimposed, FIG. 7 shows a virtual top image, and FIG. 8 shows a virtual side image. This is performed subsequent to FIGS. 4 and 5, respectively.

  In FIG. 7, (a) corresponds to FIG. 4 (g), and is a virtual upper surface point group rendering image using point group data obtained from a three-dimensional camera. (B) is an image in which the artificial images 100 ′ and 301 ′ from above the work machine 100 and the work object 301 are superimposed and displayed on the upper surface point cloud rendering image of (a). In the superimposed image of (b), the work machine 100 is more than the display of the upper surface point cloud rendering image (a) and the case where only the artificial image 100 ′ of the work machine 100 is superimposed as shown in FIG. And the positional relationship between the work object 301 can be easily understood.

  In FIG. 8, (a) corresponds to FIG. 5 (d), and is a virtual side surface point cloud rendering image using point cloud data of a three-dimensional camera. (B) is an image in which the artificial images 100 ′ and 301 ′ from the sides of the work machine 100 and the work object 301 are superimposed and displayed on the side point cloud rendering image of (a). Here, an artificial image 302 ′ for the work environment 302 is also added. In the superimposed image of (b), the work machine 100 is more than the display of the side point cloud rendering image (a) and the case where only the artificial image 100 ′ of the work machine 100 is superimposed as shown in FIG. And the positional relationship between the work object 301 can be easily understood.

FIG. 9 is a flowchart illustrating a method for generating a free viewpoint composite image according to the second embodiment. The description will focus on differences from the first embodiment (FIG. 6). This processing is executed by the information processing apparatus 111.
The processing from S901 to S903 (3D camera data input, distance image calculation, arm angle input) is the same as S601 to S603 in FIG.

  In step S904, a process for detecting the work object 301 is performed. Although the work object 301 varies depending on the work content, a dump corresponds in this example. When discharging the dump, it is determined whether the dump is reflected in the camera. In this case, identification of the work object 301 can be determined by image processing. Further, if the position and direction of the dump can be measured by GPS (Global Positioning System), it is possible to detect the work object 301 using the position and direction data of the work machine 100.

In S905, it is determined whether or not the work object 301 exists. If the work object 301 exists, the process proceeds to S906, and if not, the process proceeds to S907.
In step S906, the position and orientation of the work target 301 are calculated. For this reason, by extracting the work object 301 from the image of the camera, information on the position and orientation with respect to the work machine 100 is obtained. The obtained position and orientation are converted into the coordinate system of the work machine 100.

The virtual camera data setting in S907, the point cloud rendering image generation in S908, and the artificial image generation of the work machine in S909 are the same as S604 to S606 in FIG.
In S910, the process is branched depending on whether the work object 301 exists. If the work object 301 exists, the process proceeds to S911, and if it does not exist, the process proceeds to S912.

  In S911, an artificial image of the work target 301 is generated. That is, an artificial image 301 ′ of the work object 301 is generated from the CG model data of the work object 301 and the position and orientation of the work object 301 obtained in S <b> 906.

In S912, the artificial image 100 ′ of the work machine 100 generated in S909 and the artificial image 301 ′ of the work target 301 generated in S911 (when the work target 301 is present) are superimposed on the point cloud rendering image generated in S908. To do.
S913 and S914 (transfer of composite image, check of virtual camera data) are the same as S608 and S609 in FIG.

  As described above, in the second embodiment, the work machine and the work are displayed by superimposing and displaying both the artificial image of the work machine and the artificial image of the work object on the free viewpoint point cloud rendering image obtained from the three-dimensional camera. There is an effect that it becomes easier to grasp the positional relationship between the objects.

  In the third embodiment, the image display range is limited to a desired region with respect to the free viewpoint composite image generated from the three-dimensional camera.

FIG. 10A shows an example of a working state by the work machine 100 in the third embodiment, and FIGS. 10B and 10C show images displayed on the remote control device 200.
As shown in FIG. 10A, a state is assumed in which two obstacles 1001 and 1002 exist in the vicinity of the work target 301.

  FIG. 10B shows a virtual top image 1011 and a virtual side image 1012 synthesized by the method of the second embodiment in this state. In both cases, artificial images 100 ′ and 301 ′ of the work machine 100 and the work target 301 are superimposed and displayed. As can be seen from this figure, in the virtual side image 1012, the work machine 100 'and the work object 301' are concealed due to the obstacles 1001 and 1002, and the work state is difficult to visually recognize. In the virtual upper surface image 1011, the problem of concealment by the obstacles 1001 and 1002 does not occur. In this embodiment, in order to solve this problem, the display area of the point cloud rendering image is limited so that the point cloud data of the area including the obstacles 1001 and 1002 is not displayed.

  FIG. 10C shows an image displayed with a limited display area. In the virtual upper surface image 1021, an area 1003 is provided so as to exclude the obstacles 1001 and 1002 to the outside. The obstacles 1001 and 1002 are not rendered by generating the point cloud rendering image from the point cloud data only in the area 1003. As a result, the obstacles 1001 and 1002 are removed from the virtual side image 1022, and the concealment problem due to the obstacles 1001 and 1002 is solved.

  In order to limit the display area of the point cloud image, the step (S605, S908) of “Generate free viewpoint 3D image using distance image” in FIGS. 6 and 9 is modified. That is, when generating a free viewpoint image using a distance image, point cloud data outside the region is not displayed by a known clipping process. Thus, a virtual image as shown in FIG. 10C can be generated.

  As described above, in the third embodiment, when a free viewpoint image is generated using point cloud data obtained from a three-dimensional camera, the display area is limited. Thereby, even if there is an obstacle, the obstacle can be hidden, and the positional relationship between the work machine and the work object can be easily grasped.

  In the fourth embodiment, a CG model (work target model) of a work target used in the second embodiment is generated (updated) using input data of a three-dimensional camera.

FIG. 11 is a flowchart for explaining generation of a work target model. This generation process is executed by the information processing apparatus 111.
In S1101, 3D camera data of the work object 301 is input from the 3D camera 103, and in S1102, a distance image (distance information from the camera) is calculated.

  In S1103, the position / orientation of the three-dimensional camera 103 is calculated using the distance image and the model data of the existing work target model. The model data of the existing work target model is model data of the work target 301 that is currently owned.

In step S1104, the work target model is updated using the calculated position / posture of the three-dimensional camera 103 and the calculated distance image. That is, the data of the work target model is corrected by using new information on the position and orientation of the three-dimensional camera 103.
In step S1105, this process is repeated a predetermined number of times and the process ends. By repeating the update of the work target model in this way, a more precise work target model can be generated.

A method for updating a model using input data from the above three-dimensional camera is described in the following reference.
[References] A. Newcombe, et Al. “KinectFusion: Real-time dense surface mapping and tracking,” in Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on, 2011, pp. 127-136.
As described above, in the fourth embodiment, the work target model is configured using the point cloud data obtained from the three-dimensional camera. For this reason, it is not necessary to prepare an accurate CG model of the work target, a precise model of the work environment such as the work target and particularly the terrain can be constructed, and a more natural free viewpoint image can be generated. Thereby, there is an effect that it becomes easy to grasp the positional relationship between the work machine and the work object.

  In the fifth embodiment, when the artificial image is superimposed on the point cloud rendering image generated from the three-dimensional camera, the ratio (superimposition degree) at which the artificial image is superimposed is temporally changed.

  FIG. 12 is a diagram illustrating an example in which the degree of superimposition of the artificial image is changed and displayed. (A) is a free viewpoint composite image (superimposition degree = 0%) that does not superimpose artificial images of the work machine 100 and the work object 301, and (b) is an artificial image 100 ′ of the work machine 100 and the work object 301. , 301 ′ are superimposed on the free viewpoint composite image (superimposition degree = 100%). On the free viewpoint composite image display unit 204, for example, (a) and (b) are switched over time and displayed alternately. Alternatively, the degree of superimposition may be gradually changed between (a) and (b), and the case may be displayed including the case where the degree of superimposition = 50% as shown in (c).

  In order to realize the present embodiment, the superimposed image generation processing in S607 in FIG. 6 and S912 in FIG. 9 is modified. That is, in the process of superimposing the point cloud rendered image and the artificial image, it is possible to display a superimposed image in which the degree of superimposition is changed by temporally changing the transparency of the artificial image.

  As described above, in the fifth embodiment, the degree of superimposition of the point cloud rendering image obtained from the three-dimensional camera and the artificial image generated from the CG model is changed. This makes it easier for the operator to know whether the point cloud rendered image and the artificial image are correctly matched. If the image does not fit properly and an image shift or the like occurs, the operation is interrupted and calibration processing such as adjustment of the camera position and direction is performed.

  Note that the degree of superimposition can be changed not only in terms of time but also in accordance with, for example, an operator's instruction. The degree of superimposition can be interactively changed by an instruction using the operation input unit 206.

  In each of the above embodiments, a hydraulic excavator has been described as an example of the work machine 100 that performs remote operation. The work machine 100 is not limited to a hydraulic excavator and can be applied to various moving objects.

  FIG. 13A is a diagram illustrating an example in which the moving body is a mobile robot. The mobile robot 100 also includes the main control unit 101, the front camera 102, the three-dimensional camera 103, the arm angle sensor 104, the arm unit 105, the traveling unit 106, the wireless communication unit 107, and the like, and can be realized with the same configuration.

  FIG. 13B is an example of a virtual top image 1301 and a virtual side image 1302 in the case of a mobile robot. Also in this case, by superimposing the artificial images 100 ′ and 301 ′ of the mobile robot 100 and the work object 301 on the point cloud rendering image obtained from the three-dimensional camera 103, it is easy to grasp the positional relationship between the mobile robot and the work object. The effect of becoming is obtained.

  As described above, according to the present invention, an operator can remotely operate a work machine while referring to a free viewpoint composite image from an arbitrary angle (for example, from above or from the side). A sense of distance and the size of an object can be easily grasped, and the operability of remote work can be greatly improved.

The present invention is not limited to the embodiments described above, and includes various modifications.
In the above embodiment, it is assumed that the operator performs a remote operation on the work machine (moving body). However, the present invention is not limited to this. It is valid. That is, when it is difficult for an operator who is on the work machine to see the surrounding work object, a free viewpoint composite image in which an artificial image (CG image) is superimposed on the point cloud rendering image is displayed. Needless to say, it becomes easier to grasp the positional relationship between the work objects. Therefore, the present invention can be extended to an operation support system for operating a work machine.

  In the above embodiment, the generation of the point cloud rendering image and the synthesis of the artificial image are all executed on the mobile body side, and the remote control device receives and displays the free viewpoint synthesized image. Of these processes, part of the image processing can be performed on the remote control device side. For example, the point cloud data of the distance image and the information of the CG model (the angle information of the arm part and the type and position / direction of the work object) are transferred to the remote control device, and the point cloud rendering image and the artificial image are transmitted on the remote control device side. It is also possible to perform the above synthesis.

  The above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described. A part of the configuration of one embodiment can be replaced with the configuration of another embodiment. The configuration of another embodiment can be added to the configuration of a certain embodiment. Further, with respect to a part of the configuration of each embodiment, another configuration can be added, deleted, or replaced.

100: work machine (moving body),
101: Main control unit,
102: Camera,
103: 3D camera,
104: Arm angle sensor,
105: Arm part,
106: traveling part,
107: wireless communication unit,
111: Information processing device,
120: storage device,
121: 3D point cloud generation program,
122: Free viewpoint image generation program,
123: CG image generation program,
124: superimposed image synthesis program,
125: Free viewpoint composite image,
131: 3D model data,
141: 3D camera data,
142: 3D point cloud data,
143: virtual camera data,
144: Point cloud rendering image,
145: Artificial image,
146: moving body posture data,
200: remote control device,
201: main control unit,
202: wireless communication unit,
203: Camera image display unit
204: a free viewpoint composite image display unit,
205: a free viewpoint composite image display control unit,
206: operation input unit,
301: Work object
302: Working environment
311: Front camera image,
312: virtual top image,
313: Virtual side image,
401, 501: an imageable area,
402 to 404, 502 to 504: non-measurable area,
100 ′, 301 ′, 302 ′: CG image (artificial image).

Claims (9)

In a remote control system for operating a moving body that performs work on a work target by a remote control device,
In the moving body,
A three-dimensional camera for photographing the front of the moving body;
A free viewpoint image generation unit that generates a point group rendering image in which the work object is captured at an angle different from the viewpoint direction of the three-dimensional camera from the three-dimensional point group data obtained from the three-dimensional camera;
A CG image generation unit that generates an artificial image of the moving body based on the three-dimensional model data of the moving body;
A superimposed image synthesis unit that superimposes the artificial image on the point cloud rendering image;
A communication unit for transmitting the free viewpoint synthesized image superimposed by the superimposed image synthesizing unit to the remote control device;
With
The remote control device includes
A communication unit that receives the free viewpoint composite image transmitted from the mobile body;
A display unit for displaying the received free viewpoint composite image;
An operation input unit for instructing the mobile body to perform an operation;
A remote operation system comprising: The remote control system according to claim 1,
The CG image generation unit further generates an artificial image of the work object based on the three-dimensional model data of the work object,
The superimposed image composition unit superimposes the generated artificial image of the work object on the point cloud rendering image. The remote control system according to claim 1,
The moving body has a sensor for detecting the posture of the moving body,
The CG image generation unit generates an artificial image of the moving body based on the posture of the moving body detected by the sensor,
The superimposed image composition unit superimposes the generated artificial image of the moving body on the point cloud rendering image. The remote control system according to claim 2,
The CG image generation unit calculates the position and orientation of the work object from 3D image data captured by the 3D camera, generates an artificial image of the work object,
The superimposed image composition unit superimposes the generated artificial image of the work object on the point cloud rendering image. The remote control system according to claim 1,
The remote viewpoint system, wherein the free viewpoint image generation unit generates the point cloud rendering image using data of a desired region among the 3D image data captured by the 3D camera. The remote control system according to claim 2,
Photographing the work object using the three-dimensional camera;
A remote operation system for generating three-dimensional model data of the work object using photographed three-dimensional point cloud data and information on the position and orientation of the three-dimensional camera. The remote control system according to claim 2,
The superimposition image synthesis unit, when superimposing the artificial image on the point cloud rendering image, temporally changes a ratio of superimposing the artificial image. In an operation support system for operating a moving object that performs work on a work object,
A three-dimensional camera mounted on the moving body and photographing the front of the moving body;
A free viewpoint image generation unit that generates a point group rendering image in which the work object is captured at an angle different from the viewpoint direction of the three-dimensional camera from the three-dimensional point group data obtained from the three-dimensional camera;
A CG image generation unit that generates an artificial image of the moving body based on the three-dimensional model data of the moving body;
A superimposed image synthesis unit that superimposes the artificial image on the point cloud rendering image;
A display unit for displaying the free viewpoint composite image superimposed by the superimposed image composition unit;
An operation input unit for instructing the mobile body to perform an operation;
An operation support system comprising: The operation support system according to claim 8,
The CG image generation unit further generates an artificial image of the work object based on the three-dimensional model data of the work object,
The superimposed image composition unit superimposes the generated artificial image of the work object on the point cloud rendering image.

Images