JP2019049888A - Remote control display apparatus and remote control display method - Google Patents

Abstract

To provide a new technology capable of displaying an image for causing people to easily perceive the distance from a moving device to each position at a future time point at which, after an operation is performed on a remote control device, the moving device actually carries out a motion indicated by the operation.SOLUTION: A remote control display apparatus 20B includes an image generation device 27 that generates a three-dimensional CG image on the basis of environment data indicating an environment on the travelling direction side of a moving device 10, and a display 29 that displays the generated CG image. The image generation device 27 predicts a future position of the moving device 10 at a future time point based on a delay time considering a communication time between the moving device 10 and a remote control display apparatus 20A, and generates a CG image viewed from the predicted future position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a remote control display device and a remote control display method for displaying an environment viewed from a moving device when a person operates a remote control device to remotely control the moving device.

  In order to remotely steer a mobile device, such as an unmanned vehicle, for example, the mobile device is provided with a camera. The camera captures an area viewed from the mobile device to generate an image. This image is transmitted from the mobile device to the remote control using wireless communication. On the side of the remote control device, the image transmitted from the mobile device is received and displayed on the display. A person views the displayed display and operates the remote control device. An operation command (for example, a steering operation command in the traveling direction, an accelerator / brake operation command for acceleration / deceleration, etc.) corresponding to this operation is transmitted from the remote control device to the moving device. The control unit of the moving device receives the operation command and controls the movement of the moving device in accordance with the operation command.

Unexamined-Japanese-Patent No. 2006-279648 JP 2014-48859 A

  It takes a delay time including the communication time of the image until the above image is generated by imaging and displayed on the display. In addition, it takes a delay time including the communication time of the operation command from the time the person operates the remote control device to the actual operation of the mobile device according to the operation command indicated by the operation by looking at the image displayed on the display . Patent Literatures 1 and 2 disclose techniques for displaying an image in a state in which such delay time is taken into consideration.

  In Patent Document 1, a delay time from when a camera of a mobile device (model car) generates an image to when this image is received by the remote control device is obtained, and this delay time, speed of the mobile device, operation The image is displayed on the display by zooming or shifting the image in the vertical and horizontal directions based on a command or the like. However, there is a possibility that it is difficult for a person to see the image displayed in this manner and to grasp the sense of distance from the moving device to each position in the image.

  In Patent Document 2, a three-dimensional image is displayed as follows. The delay time from when the stereo camera of the mobile device (unmanned vehicle) generates a three-dimensional image to when this three-dimensional image is received by the remote control device is determined. Based on the delay time, the vehicle state, the operation command, etc., the position of the moving device at the time of remote control by the remote control device is predicted. The display displays the predicted position of the mobile device (the position of the hood of the vehicle) superimposed on the three-dimensional image. However, the three-dimensional image to be displayed is an image of an area viewed from the moving device at a past time when the three-dimensional image was acquired. That is, when the remote control device is operated, it is desirable to display an image viewed from the moving device at the future time point, with the time when the moving device actually performs the operation indicated by the operation as the future time point.

  Therefore, an object of the present invention is to set the distance control from the moving device to each position at the future time point as the future time point when the moving device actually performs the operation shown by the operation when the remote control device is operated. It is to provide a new technology capable of displaying an easy-to-grasp image.

The display device for remote control according to the present invention is a device for displaying an environment viewed from the mobile device when the mobile device is remotely operated.
An image generation device for generating a three-dimensional CG image based on environment data representing an environment on the traveling direction side of the mobile device;
A display for displaying the generated CG image;
The image generating device is
Predict the future position of the mobile device at a future time based on the delay time considering the communication time between the mobile device and the remote control device,
The CG image viewed from the predicted future position is generated.

A display method for remote control according to the present invention is a method for displaying an environment viewed from a mobile device at the time of remote control of the mobile device,
(A) Generate a three-dimensional CG image based on environmental data representing the environment on the traveling direction side of the mobile device,
(B) The generated CG image is displayed on a display.
In the above (A),
Predict the future position of the mobile device at a future time based on the delay time considering the communication time between the mobile device and the remote control device,
The CG image viewed from the predicted future position is generated.

  According to the present invention, the image generation device predicts the future position of the mobile device at a future time based on the delay time considering the communication delay, and generates a three-dimensional CG image seen from the predicted future position. Therefore, when a person operates the remote control device, displaying on the display a three-dimensional CG image seen from the future position of the moving device at a future time, with the time when the moving device actually operates according to the operation. Can. Therefore, it becomes unnecessary to consider that the appearance of the image displayed on the display is different between the time when the remote control device is operated and the time when the mobile device is actually operated by the operation. Is easy to grasp the sense of distance from the movement device of the future position to each position by looking at the displayed image.

  Also, the three-dimensional CG image viewed from the future position of the mobile device is similar to or similar to the way a person actually sees from the future position. Therefore, a person looks at such a three-dimensional image and easily grasps the sense of distance from the movement device at the future position to each position intuitively.

FIG. 1 is a block diagram of a remote control system including a display for remote control according to an embodiment of the present invention. An example of a movement apparatus is shown. It is explanatory drawing of the three-dimensional CG image which the display apparatus for remote control produces | generates. 5 is a flowchart of a display method for remote control according to an embodiment of the present invention.

[Overview of the embodiment]

(1) The display device for remote control according to the embodiment of the present invention may be configured as follows. The remote control display device is for displaying an environment viewed from the moving device when the moving device is remotely controlled. The remote control display device includes an image generation device that generates a three-dimensional CG image based on environment data representing an environment on the traveling direction side of the mobile device, and a display that displays the generated CG image. The image generation device predicts a future position of the mobile device at a future time based on a delay time considering a communication time between the mobile device and the remote control device, and generates the CG image viewed from the predicted future position.

(2) In the case of the above (1), the following configuration may be adopted. The environment data is point cloud data measured by a laser scanner provided in the moving device, and the point cloud data sets the three-dimensional position of each measurement point on the surface of the object in the measurement range viewed from the moving device. And is transmitted from the mobile device. The remote control display device includes a communication unit that receives the point cloud data transmitted from the mobile device.

  As described above, by using the point cloud data generated by the laser scanner provided by the moving device, it is possible to generate a three-dimensional CG image viewed from the future position of the moving device. In addition, since the point cloud data acquired by the laser scanner provided in the moving apparatus is used to generate a CG image, a special apparatus is not necessary. On the other hand, in the above-mentioned patent documents 2, a special device is needed. That is, according to Patent Document 2, a stereo camera is required on the side of the moving device, and 3D glasses and a display are required on the side of the remote control device for stereoscopically displaying an image acquired by the stereo camera to a person .

(3) In the case of the above (1), the following configuration may be adopted. The display device for remote control is transmitted as the current position of the mobile device from the storage unit that stores map data (known map data or map data generated from the point cloud data) as the environmental data, and the mobile device And a communication unit for receiving position information. The image generating device predicts the future position of the moving device at the future time point in a set coordinate system representing the map data based on the position information of the moving device and the delay time. The image generation device generates the CG image viewed from the predicted future position based on the map data.

  As described above, by using the map data stored in the storage unit and the position information transmitted as the current position of the mobile device, it is possible to generate a three-dimensional CG image viewed from the future position of the mobile device. Further, since map data is used to generate a CG image, no special device is required. On the other hand, in the above-mentioned patent documents 2, a special device is needed as mentioned above.

(4) In the case of any of the above (1) to (3), the following configuration may be adopted. When the environmental data is point cloud data or map data, the image generation device may include a position prediction unit that predicts the future position. The speed of the moving device indicated by the operation performed on the remote control device or the speed of the moving device measured at the moving device and transmitted to the display device for remote control is input to the position prediction unit. The position prediction unit predicts the future position based on the input speed and the delay time.

  Thus, the future position of the mobile device can be predicted based on the speed of the mobile device and the delay time.

(5) In the case of any of the above (1) to (4), the following configuration may be adopted. Time information indicating a time measured by a clock of the moving device is transmitted from the moving device when the time is measured. The display device for remote control comprises a communication unit for communicating with the mobile device, and the image generation device comprises a delay time acquisition unit for obtaining the delay time. The delay time acquisition unit includes a clock that measures time, and a calculation unit. The calculation unit calculates, as the communication time, the difference between the time indicated by the time information received by the communication unit and the time measured by the clock of the delay time acquisition unit when the communication unit receives the time information. And a calculation unit for obtaining a time including a time obtained by doubling the communication time as the delay time.

  Thus, the communication time required for communication from the mobile device to the remote control device is measured, and the time obtained by doubling this communication time is included in the delay time. In this way, it is possible to obtain a delay time according to the separation distance or communication condition between the mobile device and the remote control device.

(6) In the case of any of the above (1) to (4), the following configuration may be adopted. Time information indicating a time measured by a clock of the moving device is transmitted from the moving device when the time is measured. The display device for remote control comprises a communication unit for communicating with the mobile device, and the image generation device comprises a delay time acquisition unit for obtaining the delay time. The delay time acquisition unit includes a clock that measures time, and a calculation unit. The calculation unit is configured to calculate a difference between a time indicated by the time information received by the communication unit and a time measured by the clock of the delay time acquisition unit when the communication unit receives the time information. Calculated as time. Time information indicating a time measured by the clock of the delay time acquisition unit is transmitted from the communication unit when the time is measured. The difference between the time when the mobile device receives the time information and the time indicated by the time information is calculated in the mobile device as a second communication time, and the communication unit is configured to calculate the second communication time Receive from the mobile device. The calculation unit determines a time including the sum of the first communication time and the second communication time as the delay time.

  In this manner, the first communication time required for communication from the mobile device to the remote control device is measured, and the second communication time required for communication from the remote control device to the mobile device is measured. Include the total communication time in the delay time. Also in this case, it is possible to obtain the delay time according to the distance between the mobile device and the remote control device or the communication status.

[Specific Example of Embodiment]
Hereinafter, specific examples of the embodiment of the present invention will be described based on the drawings. In addition, the same code | symbol is attached | subjected to the part which is common in each figure, and the duplicate description is abbreviate | omitted.

(Configuration of remote control system 1)
FIG. 1 is a block diagram illustrating a remote control system 1 including a display 20B for remote control according to an embodiment of the present invention. The remote control system 1 includes a mobile device 10 and a remote device 20. The remote side device 20 includes a remote control device 20A and a remote control display device 20B.

  The remote control device 20A transmits an operation command corresponding to the operation to the mobile device 10 by being operated by a person. The moving device 10 moves in accordance with the operation command. The mobile device 10 acquires point cloud data representing an environment on the traveling direction side as environmental data, and transmits it to the remote control display device 20B. The remote control display device 20B displays a three-dimensional computer graphics (CG) image based on the point cloud data. A person views the displayed three-dimensional CG image and operates the remote control device 20A. The remote control display device 20B and the remote control device 20A may be integrally coupled to each other or may be separated from each other. Communication between the mobile device 10 and the remote device 20 may be performed by wireless communication.

<Configuration of Mobile Device 10>
FIG. 2 shows an example of the mobile device 10. The moving device 10 may be a vehicle having wheels 2 as shown in FIG. 2 and traveling on the ground surface 4 by being rotationally driven while the wheels 2 are in contact with the ground surface 4. Alternatively, each moving device 10 may be a traveling device traveling on the ground by a crawler.

  In the present embodiment, the mobile device 10 is configured to be remotely controlled by the remote control device 20A. Therefore, the mobile device 10 moves in accordance with the operation command received from the remote control device 20A. When the mobile device 10 does not receive an operation command from the remote control device 20A, for example, the mobile device 10 may autonomously move toward the next via point so as to avoid an obstacle as described later.

  The moving device 10 includes a laser scanner 3, a communication unit 5, a speed sensor 7, a direction sensor 8, a map generation unit 11, and a control unit 13.

  The laser scanner 3 generates point cloud data representing the three-dimensional position (coordinates) of each measurement point on the surface of the object in the measurement range viewed from the moving device 10 in a set coordinate system (a scanner coordinate system in this embodiment). That is, the laser scanner 3 scans a laser beam (for example, a pulse laser beam) with respect to the above-described measurement range and emits the laser beam in each emission direction to the measurement range, and the laser beam in each emission direction is a measurement point. Receive the reflected light reflected. Thereby, the laser scanner 3 detects the position of the measurement point for each emission direction based on the emission direction, the emission time of the laser light, and the light reception time of the reflected light of the laser light.

  The above-described scanner coordinate system representing point cloud data is a three-dimensional coordinate system whose origin is the position of the moving device 10 when the point cloud data is generated. The laser scanner 3 may be, for example, a device called Light Detection and Ranging (LiDAR) or Laser Range Finder (LRF). The laser scanner 3 inputs the generated point cloud data to the communication unit 5 and the map generation unit 11.

  The communication unit 5 transmits point cloud data input from the laser scanner 3 to the remote control display device 20B.

  The speed sensor 7 detects the magnitude (speed) of the speed of the moving device 10. The speed sensor 7 may measure, for example, the rotational speed of the wheel 2 of the moving device 10 as a vehicle, and may obtain the speed of the moving device 10 with respect to the ground surface 4 from this rotational speed. I will not. For example, the speed sensor 7 may have a configuration for estimating the speed of the mobile device 10 based on radio waves from navigation satellites in a satellite navigation system (for example, a configuration using the GNSS receiver 41 described later) Alternatively, this configuration may be combined with the configuration for measuring the rotational speed.

  The orientation sensor 8 detects the orientation (that is, the traveling direction) of the mobile device 10 with respect to a map coordinate system described later fixed to the ground surface 4. The direction sensor 8 may be configured using, for example, a gyro sensor, but is not limited thereto. For example, the direction sensor 8 may have a configuration for estimating the direction of the mobile device 10 based on radio waves from navigation satellites in a satellite navigation system (for example, a configuration using two GNSS receivers 41 described later) Alternatively, this configuration and a gyro sensor may be combined.

  The position detection unit 9 obtains the current position of the moving device 10 in the map coordinate system described later. For example, the position detection unit 9 obtains the current position of the moving device 10 based on the speed measured by the speed sensor 7 and the direction detected by the direction sensor 8. That is, assuming that the moving device 10 in the direction measured by the direction sensor 8 moves at each time point at the speed detected by the speed sensor 7, the position detection unit 9 moves the moving device based on the speed and direction at each time point Find the current position of 10. As an alternative to or in addition to this configuration, the position detection unit 9 may have a configuration for obtaining the current position of the mobile device 10 based on radio waves from positioning satellites in the satellite navigation system.

  The map generation unit 11 converts the point cloud data generated by the laser scanner 3 into data of the map coordinate system, and moves in the map coordinate system fixed to the ground surface 4 based on the point cloud data after the conversion. Map data (for example, a local map) representing the possible area and the position of the obstacle is generated. The movable area may be an area of the ground surface 4 in which the degree of unevenness is less than or equal to the threshold value, and may be an area having a spread larger than the width of the moving device 10. The obstacle may be an object whose height from the ground surface 4 exceeds the upper limit.

  The above-mentioned conversion of the point cloud data is coordinate conversion from the above-mentioned scanner coordinate system, and, for example, the direction of the moving device 10 detected by the direction sensor 8 and the current position of the moving device 10 calculated by the position detection unit 9 It may be done on the basis of The laser scanner 3 repeatedly generates point cloud data, and each time, the map generation unit 11 generates map data based on the point cloud data.

  When the control unit 13 does not receive an operation command from the remote control device 20A, based on the current position of the mobile device 10 detected by the position detection unit 9 and the latest map data generated by the map generation unit 11, A movement route toward the next via point is generated at a set period so as not to interfere with an obstacle. The next via point is a via point through which the mobile device 10 passes next among the plurality of via points through which the mobile device 10 moves to move to the target position. The target position and the plurality of via points are set in advance.

  The control unit 13 controls the drive device 15 of the moving device 10 so that the moving device 10 moves along the generated movement path. The drive device 15 is configured of, for example, a plurality of actuators for operating an accelerator, a brake, a steering, a transmission, and the like of the vehicle as the moving device 10.

<Configuration of Remote Control Device 20A>
The remote control device 20A includes an operating device 17 and a command generation unit 19. The operating device 17 is configured to be operable by a person. The operation that can be performed by the operation device 17 is one of a steering operation for specifying the traveling direction of the moving device 10 and an acceleration / deceleration operation related to the speed of the moving device 10 (for example, steering operation or acceleration / deceleration operation (accelerator pedal 23a and brake Operation of at least one of the pedals 23b) or both.

  The operating device 17 may include a steering operating unit 21 (e.g., a steering wheel) for steering operation. The steering operation unit 21 can perform an operation (for example, an operation of rotating the above-described steering wheel to the right or left) for specifying the traveling direction of the moving device 10. The traveling direction of the moving device 10 is the operation direction of the steering operation unit 21 (for example, the direction for rotating the above-mentioned steering wheel from the reference position (clockwise or counterclockwise)) and the operation amount of the steering operation unit 21 (for example Specified according to the amount of rotation from the reference position. The traveling direction of the moving device 10 is designated as the direction with respect to the front of the moving device 10 (for example, a reference line fixed to the moving device 10 and facing the front of the moving device 10). When the steering operation unit 21 is not operated (for example, when the above-described rotational position of the steering wheel is at the reference position), the traveling direction of the moving device 10 may be designated as the front of the moving device 10.

  The operating device 17 may include an acceleration / deceleration operation unit 23 (for example, an accelerator pedal 23a and a brake pedal 23b) for acceleration / deceleration operation. In the acceleration / deceleration operation unit 23, an operation (for example, operation of any of the accelerator pedal 23a and the brake pedal 23b described above) indicating which of acceleration and deceleration is to be performed and an operation (for example, the acceleration / deceleration It is possible to depress the above-mentioned accelerator pedal 23a or the brake pedal 23b by an amount corresponding to the amount.

  The command generation unit 19 generates an operation command according to the operation performed on the operation device 17. For example, the command generation unit 19 generates an operation command indicating the traveling direction of the moving device 10 designated by the operation of the steering operation unit 21. In addition, when the acceleration / deceleration operation unit 23 is operated, the command generation unit 19 generates an operation command indicating whether to execute acceleration or deceleration and an acceleration / deceleration amount. The generated operation command is transmitted from the communication unit 25 of the remote device 20 to the mobile device 10, and is input to the control unit 13 via the communication unit 5. Thereby, the control unit 13 controls the drive device 15 according to the input operation command (for example, the traveling direction specified by the operation command, or the acceleration or deceleration and the acceleration / deceleration amount indicated by the operation command). Thus, the moving device 10 performs the movement operation according to the operation command.

<Configuration of Display Device 20B for Remote Control>
The remote control display device 20B includes a communication unit 25, an image generation device 27, and a display 29. The communication unit 25 receives point cloud data transmitted from the communication unit 5 of the mobile device 10. The image generation device 27 generates a three-dimensional CG image based on the point cloud data received by the communication unit 25. The display 29 displays the three-dimensional CG image generated by the image generation device 27. The display 29 and the operating device 17 are arranged so that the user can operate the operating device 17 while looking at the CG image displayed on the display 29.

  Point cloud data repeatedly generated by the laser scanner 3 is repeatedly input to the image generation device 27 via the communication units 5 and 25. Every time point cloud data is input, the image generation device 27 generates a three-dimensional CG image based on the point cloud data. The display 29 displays the three-dimensional CG image each time a three-dimensional CG image is generated.

  The image generation device 27 predicts the future position of the mobile device 10 at a future time based on the delay time taking into consideration the communication time between the mobile device 10 and the remote control device 20A, and the three-dimensional CG image seen from the predicted future position Generate The image generation device 27 includes, for example, a delay time acquisition unit 31, a position prediction unit 33, and an image generation unit 35.

  The delay time acquisition unit 31 measures the communication time between the mobile device 10 and the remote control device 20A, and obtains the delay time based on the communication time. The delay time acquisition unit 31 has a clock 31 a and a calculation unit 31 b. In this regard, the moving device 10 is also provided with a clock 37a. Time information indicating the time measured by the clock 37 a of the mobile device 10 is transmitted from the communication unit 5 of the mobile device 10 when the time is measured. That is, the time when the time is measured and the time when the time information is transmitted coincide with each other. The calculation unit 31b is a difference between the time indicated by the time information transmitted from the communication unit 5 and received by the communication unit 25 and the time measured by the clock 31a of the delay time acquisition unit 31 when the communication unit 25 receives the time information. Is calculated as the first communication time.

Example of Acquisition of Delay Time The calculation unit 31b obtains a time including a time obtained by doubling the first communication time as a delay time. The calculation unit 31 b inputs the obtained delay time to the position prediction unit 33. Here, the delay time may be a time obtained by doubling the first communication time, or may be a time obtained by adding the data processing time to a time obtained by doubling the first communication time. The data processing time includes the time required for the laser scanner 3 to generate point cloud data, and the time required for the remote control device 20A to generate the operation command from the time when the remote control device 20A is operated. The control unit 13 may include the time from when the operation command is received to when the movement device 10 actually performs the operation according to the operation command. Data processing time may be estimated beforehand.

Another Example of Acquisition of Delay Time Alternatively, the calculation unit 31b may obtain the delay time as follows. The mobile apparatus 10 is provided with a communication time acquisition unit 37 having a clock 37a and a calculation unit 37b. Time information indicating the time measured by the clock 31a of the remote control display device 20B is transmitted from the communication unit 25 of the remote control display device 20B when the time is measured. That is, the time when the time is measured and the time when the time information is transmitted coincide with each other. The calculation unit 37b is a difference between the time indicated by the time information transmitted from the communication unit 25 and received by the communication unit 5 and the time measured by the clock 37a of the communication time acquisition unit 37 when the communication unit 5 receives the time information. Is calculated as the second communication time. The communication unit 5 of the mobile device 10 transmits the second communication time to the remote control display device 20B. The second communication time is received by the communication unit 25 and input to the calculation unit 31 b. The calculation unit 31 b obtains a time including a time obtained by adding the first communication time calculated as described above and the input second communication time as a delay time. The calculation unit 31 b inputs the obtained delay time to the position prediction unit 33. Here, the delay time may be a sum of the first and second communication times, or may be a time obtained by adding the data processing time described above to the sum.

Synchronization of the Clocks 31a and 37a The clock 31a of the display device 20B for remote control and the clock 37a of the moving device 10 may be synchronized as follows. A GNSS (Global Navigation Satellite System) receiver 39, 41 is provided for each of the remote control display device 20B and the mobile device 10. Each navigation satellite (eg, GPS satellites) transmits radio waves. The radio waves transmitted by each navigation satellite include position information indicating the position of the navigation satellite at the time of transmission, and time information indicating the time measured by the clock (for example, atomic clock) provided at the navigation satellite at the time of transmission. include.

  The GNSS receiver 39 of the remote control display device 20B receives radio waves from a plurality of navigation satellites, and based on the difference between the time indicated by the time information included in the received radio waves and the time measured by the clock 31a at the time of reception. , Find the distance between each navigation satellite and itself. Based on these distances, the GNSS receiver 39 determines the position of the GNSS receiver 39. The GNSS receiver 39 synchronizes (coincides) the time of the clock 31a with the time of the navigation satellite clock based on the determined position and position information from the navigation satellite and time information. In the same way, the GNSS receiver 41 of the mobile unit 10 also synchronizes the time of the clock 37a with the clock of the navigation satellite. The clock times are thereby synchronized with one another via the navigation satellite clock. However, the times of the clocks 31a and 37a may be synchronized with each other in other manners.

Prediction of Future Position of Mobile Device 10 The position prediction unit 33 predicts the future position of the mobile device 10. Here, the future position is the position of the mobile device 10 at the future time. The future position is a position represented in the setting coordinate system (in the present embodiment, a scanner coordinate system representing the above-described point cloud data received by the communication unit 25). In the present embodiment, the origin in the setting coordinate system is the position of the moving device 10 at the time of generation of the point group data. When a person operates the controller device 17 at the time when the image generation unit 35 generates a CG image based on the point cloud data, an operation command by the operation is transmitted from the communication unit 25 and movement according to the operation command The point in time when the action is actually performed on the mobile device 10 is the future point in time. In this embodiment, it is assumed that the point in time when the above-mentioned delay time has elapsed from the point in time of point cloud data generation is the future point in time.

  The position prediction unit 33 predicts the future position in the setting coordinate system based on the delay time obtained by the delay time acquisition unit 31 and the speed of the moving device 10 that has been input.

  Here, the speed of the moving device 10 input to the position prediction unit 33 may be the speed of the moving device 10 indicated by the operation performed on the operating device 17. In this case, the controller device 17 includes a configuration capable of specifying the speed of the moving device 10 (for example, an input unit for inputting a velocity value), and the speed designated by the operation of the controller device 17 corresponds to the communication unit 25 as an operation command. , And are input to the position prediction unit 33. The designated speed is received by the communication unit 5 of the mobile device 10 and is input to the control unit 13, and the controller 13 controls the driving device 15 so that the speed of the mobile device 10 is maintained at the designated speed. Control. The position prediction unit 33 may use the speed input at the time of prediction of the future position for prediction of the future position.

  Alternatively, the speed of the moving device 10 input to the position prediction unit 33 may be the speed of the moving device 10 measured by the speed sensor 7 of the moving device 10. In this case, the speed of the moving device 10 measured by the speed sensor 7 is input to the position prediction unit 33 via the communication units 5 and 25. Note that the speed measured repeatedly by the speed sensor 7 is repeatedly input to the position prediction unit 33 via the communication units 5 and 25, and the position prediction unit 33 uses the latest speed input to predict the future position. You may

  The position prediction unit 33 obtains a distance obtained by multiplying the above delay time by the speed of the input moving device 10, and obtains a position shifted forward from the origin of the scanner coordinate system by this distance as a future position. Here, “forward” may be the direction of a reference line extending from the origin of the scanner coordinate system. The reference line may be fixed in the scanner coordinate system (i.e. the mobile device 10) and may coincide with the front of the mobile device 10. Alternatively, the “forward” may be the traveling direction of the moving device 10 designated by the operation of the steering operation unit 21. That is, in the scanner coordinate system, “forward” may be a traveling direction designated with respect to the front by the operation of the steering operation unit 21 with the above-described reference line as the front of the moving device 10. In this case, the traveling direction designated by the operation of the steering operation unit 21 is transmitted from the communication unit 25 to the moving device 10 as an operation command indicating this, and is input to the position prediction unit 33. The designated traveling direction is input to the control unit 13 of the mobile device 10 through the communication units 25 and 5, and the control unit 13 controls the drive device 15 so that the mobile device 10 moves in the designated traveling direction. Do.

  The image generation unit 35 generates a three-dimensional CG image viewed from the future position obtained by the position prediction unit 33 as described above based on the point cloud data. The nearest position in the generated three-dimensional CG image may be the future position. The position prediction unit 33 may generate a three-dimensional CG image according to perspective.

  The image generation unit 35 may display a mark (for example, a mark M2 described later) indicating a part (for example, the front end) of the moving device 10 at the future position in the generated three-dimensional CG image.

  FIG. 3A is a three-dimensional CG image viewed from the position of the moving device 10 at the time of generation of point cloud data (that is, the origin of the scanner coordinate system representing the point cloud data). In FIG. 3 (A), the code M1 is a mark indicating the front end of the moving device 10 at the time of generation of point cloud data. In FIG. 3A, reference symbol M2 denotes a mark (thin line mark) indicating the front end of the moving apparatus 10 at a future point in time. In FIG. 3A and FIG. 3B described later, a broken line X indicates a bonnet of the vehicle 10. In FIG. 3 (A) and FIG. 3 (B) described later, the frame of the thick line may be the outer edge of the three-dimensional CG image, and in this case, the frame may be the outer edge of the screen of the display 29.

  FIG. 3B is a three-dimensional CG image viewed from the position of the moving apparatus 10 at the future time (that is, the above-described future position). In FIG. 3 (B), the code M2 is a mark indicating the front end of the moving apparatus 10 at a future point in time. Thus, a three-dimensional CG image seen from the future position of the moving apparatus 10 at the future time point is displayed, for example, as shown in FIG. 3 (B). Therefore, since it becomes unnecessary for a person to operate the operating device 17 at the timing which considered delay time, operation becomes easy. For example, when it is desired to turn the mobile device 10 left at the intersection P (a portion surrounded by an alternate long and short dash line in FIGS. 3A and 3B), FIG. In the three-dimensional CG image displayed on the display 29, it is not necessary to perform the left operation at the timing when the mark M1 of) reaches, and in the position where the mobile device 10 actually starts to turn left in FIG. When the mark M2 arrives, it suffices to perform a left turn operation on the steering operation unit 21.

(Display method for remote control)
FIG. 4 is a flow chart illustrating a display method for remote control according to an embodiment of the present invention. The display method for remote control is performed using the display device 20B for remote control, the remote control device 20A, and the moving device 10 described above. The remote control display method includes steps S1 to S13.

<Process for generating image>
In step S <b> 1, the laser scanner 3 generates the above-mentioned point cloud data by scanning the laser light in the measurement range viewed from the moving device 10. Step S1 is repeatedly performed while the mobile device 10 is moving. Every time step S1 is performed, steps S2 to S5 are performed on the point cloud data generated in step S1.

  In step S2, the communication unit 5 transmits the point cloud data generated in step S1 to the remote control display device 20B.

  In step S3, the communication unit 25 of the remote control display device 20B receives the point cloud data transmitted in step S2, and the received point cloud data is input to the image generation device 27.

  In step S4, based on the point cloud data input in step S3, the image generation device 27 sees the third direction seen from the future position of the movement device 10 (for example, in front of the movement device 10). Generate an original CG image. The future position is the position of the mobile apparatus 10 at the future time point when the delay time has elapsed since the point cloud data was generated at step S1.

  In step S5, the display 29 displays the three-dimensional CG image generated in step S4.

<Process for finding delay time>
The following steps S6 to S13 are repeatedly performed in parallel with the repetition of the steps S1 to S5 described above.

In step S6, the communication unit 5 of the mobile device 10 transmits time information indicating the time measured by the clock 37a of the mobile device 10 to the remote control display device 20B.
In step S7, in the remote control display device 20B, the communication unit 25 receives the time information transmitted in step S6, and the clock 31a measures the time when the communication unit 25 receives the time information.
In step S8, the calculation unit 31b of the delay time acquisition unit 31 obtains a difference between the time measured by the clock 31a in step S7 and the time indicated by the time information received in step S7 as the first communication time.

In step S9, the communication unit 25 of the remote control display device 20B transmits time information indicating the time measured by the clock 31a of the remote control display device 20B to the mobile device 10.
In step S10, in the mobile device 10, the communication unit 5 receives the time information transmitted in step S9, and the clock 37a measures the time when the communication unit 5 receives the time information.
In step S11, the calculation unit 37b of the communication time acquisition unit 37 obtains a difference between the time measured by the clock 37a in step S10 and the time indicated by the time information received in step S10 as a second communication time.
In step S12, the communication unit 5 transmits the second communication time obtained in step S11 to the remote control display device 20B, and the communication unit 25 receives this second communication time.

  In step S13, the calculation unit 31b sums the first and second communication times based on the first communication time obtained in step S8 and the second communication time received by the communication unit 25 in step S12. The included time is determined as the above-mentioned delay time. In step S13, the calculation unit 31b may obtain a time including a time obtained by doubling the first communication time as the above-mentioned delay time. In this case, the above-described steps S9 to 12 are omitted.

  The delay time may be repeatedly determined by repeating steps S6 to S13. In this case, in step S4 described above, a three-dimensional CG image may be determined based on the determined latest delay time. That is, in step S4, the position prediction unit 33 predicts the future position in the setting coordinate system based on the latest delay time obtained in step S13 and the speed of the moving apparatus 10 input as described above. A three-dimensional CG image seen from the predicted future position is generated based on the point cloud data received in step S3.

(Other configuration example)
In the above description, the image generation unit 35 generates a three-dimensional CG image based on point cloud data, but may generate a three-dimensional CG image based on map data as environmental data. Although the example of a structure in this case is demonstrated below, the content of the said example of a structure which is not demonstrated below may be the same as the content of the above-mentioned embodiment described with reference to drawings.

The remote control display device 20B includes a storage unit 43 (see FIG. 1) that stores map data.
The map data may be known map data, or may be map data generated by the map generation unit 11 from the point cloud data described above. In any case, the map data may be data representing the position (coordinates) of each obstacle (building or curb on a road) in the setting coordinate system. When the setting coordinate system is a two-dimensional coordinate along the horizontal plane, the map data includes data representing the two-dimensional position of each obstacle in the setting coordinate system and data of the height of each obstacle. The map data may be data representing a three-dimensional position of each obstacle in a three-dimensional setting coordinate system. When the map data is map data generated from point cloud data, the set coordinate system may be the above-mentioned map coordinate system, and the map data is stored from the map generation unit 11 through the communication unit 5 or 25. It may be input to the unit 43. Further, in this case, the map data may be input from the map generation unit 11 to the storage unit 43 via the communication units 5 and 25 each time the map data generation unit 11 generates the map data.

  The communication unit 25 receives position information transmitted as the current position of the mobile device 10 from the communication unit 5 of the mobile device 10. Here, the current position of the mobile device 10 may be the position of the mobile device 10 determined by the position detection unit 9 at the time of the transmission. Alternatively, the current position of the mobile device 10 is the latest current position among the current positions of the mobile device 10 repeatedly obtained by the position detection unit 9 and repeatedly input to the position prediction unit 33 via the communication units 5 and 25. It may be.

  The position prediction unit 33 obtains a distance obtained by multiplying the delay time described above by the speed of the input moving device 10, and the position indicated by the position information received by the communication unit 25 in the setting coordinate system by this distance. The position shifted forward from the position (that is, the position transmitted from the communication unit 5 as the current position of the mobile device 10) is determined as the future position. Here, “forward” may be the traveling direction of the moving device 10 designated by the operation of the steering operation unit 21.

  The present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made within the scope of the technical idea of the present invention.

Reference Signs List 1 remote control system, 2 wheels, 3 laser scanners, 4 ground surface, 5 communication units, 7 speed sensors, 8 direction sensors, 9 position detecting units, 10 moving devices, 11 map generating units, 13 control units, 15 driving devices, Reference Signs List 17 operation device, 19 command generation unit, 21 steering operation unit, 23 acceleration / deceleration operation unit, 23a accelerator pedal, 23b brake pedal,
Reference Signs List 25 communication unit, 27 image generation apparatus, 29 display, 31 delay time acquisition unit, 31a clock, 31b calculation unit, 33 position prediction unit, 35 image generation unit, 37 communication time acquisition unit, 37a clock, 37b calculation unit, 39 GNSS Receiver, 41 GNSS receiver, 43 storage unit, 20 remote unit, 20A remote control unit, 20B display unit for remote control

Claims (7)

A remote control display device for displaying an environment viewed from the mobile device when the mobile device is remotely operated,
An image generation device for generating a three-dimensional CG image based on environment data representing an environment on the traveling direction side of the mobile device;
A display for displaying the generated CG image;
The image generating device is
Predict the future position of the mobile device at a future time based on the delay time considering the communication time between the mobile device and the remote control device,
A display device for remote control, which generates the CG image viewed from the predicted future position. The environment data is point cloud data measured by a laser scanner provided in the moving device, and the point cloud data sets the three-dimensional position of each measurement point on the surface of the object in the measurement range viewed from the moving device. Data represented by and transmitted from the mobile device,
The display device for remote control according to claim 1, further comprising: a communication unit that receives the point cloud data transmitted from the mobile device. A storage unit that stores map data as the environmental data;
A communication unit that receives position information transmitted from the mobile device as the current position of the mobile device;
The image generation device predicts the future position of the moving device at the future point in a set coordinate system representing the map data, based on the position information of the moving device and the delay time,
The display device for remote control according to claim 1, wherein the CG image viewed from the predicted future position is generated based on the map data. The image generation apparatus includes a position prediction unit that predicts the future position;
The speed of the moving device indicated by the operation performed on the remote control device, or the speed of the moving device measured at the moving device and transmitted to the display device for remote control is input to the position prediction unit,
The display device for remote control according to any one of claims 1 to 3, wherein the position prediction unit predicts the future position based on the input speed and the delay time. Time information indicating a time measured by a clock of the mobile device is transmitted from the mobile device when the time is measured,
The display device for remote control comprises a communication unit for communicating with the mobile device, and the image generation device comprises a delay time acquisition unit for obtaining the delay time,
The delay time acquisition unit
A clock that measures time,
The difference between the time indicated by the time information received by the communication unit and the time measured by the clock of the delay time acquisition unit when the communication unit receives the time information is calculated as the communication time The display unit for remote control according to any one of claims 1 to 4, further comprising: a calculation unit that determines a time including a time obtained by doubling a time as the delay time. Time information indicating a time measured by a clock of the mobile device is transmitted from the mobile device when the time is measured,
The display device for remote control includes a communication unit for communicating with the mobile device, and the image generation device includes a delay time acquisition unit for obtaining the delay time.
The delay time acquisition unit
A clock that measures time,
A difference between a time indicated by the time information received by the communication unit and a time measured by the clock of the delay time acquisition unit when the communication unit receives the time information is calculated as a first communication time. And a calculation unit,
Time information indicating the time measured by the clock of the delay time acquisition unit is transmitted from the communication unit when the time is measured, and the time when the mobile device receives the time information, and the time indicated by the time information The second communication time is calculated in the mobile device as a second communication time, and the communication unit receives the second communication time from the mobile device,
The display device for remote control according to any one of claims 1 to 4, wherein the calculation unit obtains a time including a sum of the first communication time and the second communication time as the delay time. A remote control display method for displaying an environment viewed from a mobile device when the mobile device is remotely operated.
(A) Generate a three-dimensional CG image based on environmental data representing the environment on the traveling direction side of the mobile device,
(B) displaying the generated CG image on a display;
In the above (A),
Predict the future position of the mobile device at a future time based on the delay time considering the communication time between the mobile device and the remote control device,
A display method for remote control, generating the CG image viewed from the predicted future position.

Images