JP2018180771A - Remote control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote control device preventing an update of an image displayed to an operator of the remote control device from becoming unstable.SOLUTION: A remote control device (10) can remotely control a vehicle (20) as an operation target. The remote control device includes: image acquisition means (12) that indicates an image around a vehicle via a wireless communication and sequentially acquires image data whose data amount increases as a speed of the vehicle increases, and stores the acquired image data in a buffer in a predetermined capacity; display means (12, 13) for reading the image data stored in the buffer and displaying the image indicated by the image data as a display image; and control means (12) for performing a predetermined operation leading to deceleration of the vehicle on condition that a time during which the display image can be displayed by the image data stored in the buffer has become less than the predetermined time.SELECTED DRAWING: Figure 1

Description

  The present invention relates to the technical field of a remote control device whose operation target is, for example, a vehicle such as a car.

  This type of device performs data communication with the operation target. As a technology relating to data communication, for example, in order to suppress the occurrence of problems due to fluctuations in the communication load of a network connecting a plurality of ECUs (Electronic Control Units) mounted on a vehicle, according to the fluctuations in the communication load An apparatus for changing the communication speed has been proposed (see Patent Document 1). Alternatively, the system includes a server and a client that mutually communicate wirelessly, and the client displays a route to a destination generated based on map data, road data, and client position information in the server, and the communication speed is A system has been proposed in which the server transmits a guidance screen using a map image that changes the scale to fit the entire route on one screen to the client (see Patent Document 2).

JP, 2008-211644, A Unexamined-Japanese-Patent No. 2010-223691

  In the remote control device, an image showing the surroundings of the operation target is acquired through communication with the operation target, and the acquired image is displayed to the operator of the remote control device. For example, when the update of the image displayed to the operator becomes unstable due to the communication environment, it becomes difficult for the operator to remotely control the operation target. The techniques described in Patent Documents 1 and 2 do not consider remote control.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide a remote control device capable of suppressing instability of updating of an image displayed to the operator of the remote control device. I assume.

  The remote control device according to the present invention is a remote control device capable of remotely operating a vehicle to be operated in order to solve the above-mentioned problems, and shows an image of the surroundings of the vehicle via wireless communication, and The image acquisition means for sequentially acquiring image data whose data amount increases as the speed of the vehicle increases, and storing the acquired image data in a buffer of a predetermined capacity, and reading out the image data stored in the buffer A display means for displaying an image indicated by the read out image data as a display image, and a condition that a time in which the display image can be displayed by the image data stored in the buffer has become less than a predetermined time And control means for performing a predetermined operation leading to deceleration of the vehicle.

  In the remote control device, the amount of image data necessary to display a display image (that is, an image displayed to the operator of the remote control device) increases as the speed of the vehicle to be operated increases. This is because as the speed of the vehicle increases, the data amount of image data for realizing the image quality and the image range (field of view) required to operate the vehicle safely becomes larger.

  When the communication environment deteriorates and the amount of data that can be acquired per unit time through communication decreases, the time taken to acquire image data necessary to display a display image increases. Then, regular updating of the displayed image becomes difficult.

  Therefore, in the remote control device, on the condition that the time in which the display image can be displayed by the image data stored in the buffer is less than the predetermined time, the predetermined operation leading to the deceleration of the vehicle is performed. The "predetermined operation" may include a warning operation that issues a warning for decelerating the vehicle. Further, the "predetermined operation" may include a control operation of automatically controlling the vehicle so as to decelerate. If the vehicle decelerates according to a predetermined operation, the data amount of image data required to display a display image decreases, so even if the communication environment is relatively poor, the image data can be acquired relatively early. Can. Therefore, according to the said remote control apparatus, it can suppress that the update of a display image becomes unstable.

  The operation and other advantages of the present invention will be apparent from the embodiments to be described below.

It is a block diagram showing composition of a remote control system concerning an embodiment. It is a flowchart which shows the image processing which concerns on embodiment. It is a figure which shows an example of the relationship between the driving | running | working scene and data which concern on embodiment. It is a figure showing the concept of image processing concerning an embodiment.

  An embodiment according to the remote control device of the present invention will be described with reference to FIGS. 1 to 4.

(System configuration)
The configuration of a remote control system including the remote control device according to the embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of the remote control system according to the embodiment.

  In FIG. 1, the remote control system 1 includes a remote control device 10 and a vehicle 20 that is an operation target of the remote control device 10. The remote control device 10 includes a wireless device 11, a decoder and superimposition display unit 12, a display device 13, an operation device ECU (Electronic Control Unit) 14, an accelerator pedal 15, a brake pedal 16, and a steering wheel 17. The vehicle 20 includes a wireless device 21, an encoder 22, a central camera 23C, a left camera 23L, a right camera 23R, a vehicle control ECU 24, an engine control ECU 25, a brake control ECU 26, and a steering control ECU 27.

  The remote control device 10 and the vehicle 20 can wirelessly communicate with each other via the wireless devices 11 and 21. For wireless communication between the remote control device 10 and the vehicle 20, for example, UDP (User Datagram Protocol) or TCP (Transmission Control Protocol) is used. In the remote control system 1, since real-time property is emphasized, it is desirable to use UDP.

  The central camera 23C captures the front of the vehicle 20. The left camera 23L captures an image of the left side of the vehicle 20. The right camera 23R captures the right side of the vehicle 20. Images captured by each of the central camera 23C, the left camera 23L, and the right camera 23R are encoded by the encoder 22 and transmitted to the remote control device 10 via the wireless device 21. The decoder and superimposition display unit 12 decodes the encoded image and displays it on the display device 13. As a result, an image of the surroundings of the vehicle 20 is displayed on the operator of the remote control device 10.

  The operator of the remote control device 10 operates the accelerator pedal 15, the brake pedal 16, and the steering wheel 17 while viewing the image displayed on the display device 13. The operating device ECU 14 detects the operating conditions of the accelerator pedal 15, the brake pedal 16, and the steering wheel 17, and transmits a signal indicating the detected operating conditions to the vehicle 20 via the wireless device 11. The vehicle control ECU 24 controls the engine control ECU 25, the brake control ECU 26, and the steering control ECU 27 based on the operation status indicated by the received signal. As a result, the vehicle 20 is remotely controlled by the operator of the remote control device 10.

  The existing technology can be applied to the technology related to the remote control of the vehicle 20 in the remote control system 1, and thus the detailed description thereof is omitted. In the present embodiment, a technique related to image processing for displaying an image around the vehicle 20 for the operator of the remote control device 10 will be described in detail.

(Image processing)
Image processing according to the embodiment will be described with reference to the flowchart of FIG.

1. The encoder 22 of the vehicle-side operating vehicle 20 acquires vehicle information based on outputs from various sensors such as a vehicle speed sensor (not shown) (step S101). Subsequently, the encoder 22 performs estimated traveling scene determination based on the acquired vehicle information (step S102). In the estimated traveling scene determination, it is determined which of the traveling scenes shown in FIG. 3 the traveling state of the vehicle 20 corresponds to.

  Specifically, when the braking force is applied to the vehicle 20 by the brake control ECU 26 and the speed of the vehicle 20 is, for example, 10 km / hour or less, “Stop from extremely low speed in the estimated traveling scene determination Is determined. If braking force is applied to the vehicle 20 by the brake control ECU 26 and the speed of the vehicle 20 is, for example, greater than 10 km / h and 30 km / h or less, it is determined in the estimated traveling scene determination as "stop from low speed" . If braking force is applied to the vehicle 20 by the brake control ECU 26 and the speed of the vehicle 20 is, for example, greater than 30 km / hr and not more than 60 km / hr, it is determined in the estimated traveling scene determination as "stop from medium speed" Ru.

  If the braking control ECU 26 does not apply a braking force to the vehicle 20 (that is, the vehicle 20 is accelerating or coasting) and the speed of the vehicle 20 is, for example, 10 km / hour or less, the estimated traveling scene determination It is determined that the vehicle travels at a very low speed. If no braking force is applied to the vehicle 20 by the brake control ECU 26 and the speed of the vehicle 20 is, for example, greater than 10 km / hr and equal to or less than 30 km / hr, then it is determined to be "low speed travel" in the estimated travel scene determination. If no braking force is applied to the vehicle 20 by the brake control ECU 26 and the speed of the vehicle 20 is, for example, greater than 30 km / hr and equal to or less than 60 km / hr, it is determined as "medium-speed travel" in the estimated travel scene determination. .

  In parallel with the processes of steps S101 and S102, the encoder 22 receives data from the remote control device 10 via the wireless device 21 (step S103). The data received in the process of step S103 is the determination result of each of the communication environment determination (step S221) and the buffer determination (step S222) described later. Each of the determination result of the communication environment determination and the determination result of the buffer determination is represented by, for example, a four-step evaluation of “best”, “good”, “bad”, and “worst”.

  Next, the encoder 22 performs image / vehicle / display information determination based on the result of the process of step S102 and the data received in the process of S103 (step S104). The image / vehicle / display information determination will be specifically described with reference to FIG. FIG. 3 is a view showing an example of the relationship between a traveling scene and data according to the embodiment.

  Here, the “total data amount” of FIG. 3 is necessary to realize each traveling state mentioned as “traveling scene” while appropriately updating the image displayed to the operator of the remote control device 10 It indicates the amount of data per unit time required to transmit the minimum data appropriately (hereinafter referred to as “transmission speed” as appropriate). Also, “type of data” indicates the type of minimum data necessary to realize each traveling state listed as “traveling scene”.

  In the image / vehicle / display information determination, the upper limit of the communication speed is determined by the determination result of the communication environment determination and the determination result of the buffer determination. For example, when the determination result of the communication environment determination and the determination result of the buffer determination are both “best”, the upper limit of the communication speed is, for example, about 25 Mbps.

  In the image / vehicle / display information determination, the communication speed required to realize the traveling state corresponding to the traveling scene indicated by the estimation result of the estimated traveling scene determination is the communication environment determination determination result and the buffer determination determination result. If it is below the upper limit of the communication speed determined based on the data type, the type of data is determined such that as much data as possible is transmitted from the vehicle 20 to the remote control device 10 below the upper limit of the communication speed.

  For example, when the upper limit of the communication speed is 25 Mbps and the determination result of the estimated traveling scene determination is "medium speed traveling", the central camera 23C is not the image quality described in the type of data corresponding to medium speed traveling in FIG. The image quality of the image captured by each of the left camera 23L and the right camera 23R is high.

  On the other hand, the minimum communication speed necessary to realize the traveling state corresponding to the traveling scene indicated by the judgment result of the estimated traveling scene judgment is determined based on the judgment result of the communication environment judgment and the judgment result of the buffer judgment. If it is larger than the upper limit of the communication speed, data necessary for realizing the traveling state corresponding to the traveling scene indicated by the judgment result of the estimated traveling scene judgment is transmitted from the vehicle 20 to the remote control device 10 The type of data is determined.

  For example, when the upper limit of the communication speed is 17 Mbps and the determination result of the estimated traveling scene determination is “medium-speed traveling”, the image quality of the image captured by each of the central camera 23C, the left camera 23L and the right camera 23R is shown in FIG. The image quality is described in the type of data corresponding to medium speed running of 3.

  Thereafter, the encoder 22 designates the resolution of each of the central camera 23C, the left camera 23L and the right camera 23R based on the result of the image / vehicle / display information determination (step S105), and the central camera 23C, the left camera 23L and the right An image captured by each of the cameras 23R is acquired (step S106). Further, the encoder 22 acquires vehicle information and display information in parallel with the processes of steps S105 and S106 (step S107). The vehicle information and the display information correspond to, for example, "vehicle information" and "display information" described in the type of data corresponding to "medium-speed travel" in FIG.

  Thereafter, the encoder 22 transmits the image and vehicle information and display information acquired in the processing of steps S106 and S107 to the remote control device 10 via the wireless device 21 (step S108). The encoder 22 starts the process of step S101 after a first predetermined period (for example, several milliseconds to several tens of milliseconds) has elapsed.

2. When the decoder and superimposition display unit 12 of the remote control device side operation remote control device 10 receives data from the vehicle 20 via the wireless device 11 (step S211), an image included in the received data (that is, the central camera For example, information related to traveling of the vehicle 20 is added as superimposed display to the images captured by the left camera 23C, the left camera 23L, and the right camera 23R (step S212). In the process of step S212, a so-called augmented reality (AR) function is implemented.

  Thereafter, the decoder and superimposition display unit 12 stores the image to which the superimposition display is given in a buffer memory (not shown). It is assumed that image data of, for example, three images (about 100 milliseconds) can be stored in the buffer memory. The decoder and superimposition display unit 12 reads the image stored in the buffer memory at a constant cycle (step S213). Subsequently, the decoder and superimposition display unit 12 displays the read image on the display device 13 (step S214). After the second predetermined period (for example, several milliseconds to several tens of milliseconds) has elapsed, the decoder and superimposing display unit 12 starts the process of step S211.

  By the way, if the communication environment is sufficiently good, data transmitted from the vehicle 20 can be received appropriately (for example, with a minimum delay time). In this case, the buffer memory stores an image having a data amount necessary and sufficient for displaying the image on the display device 13 and sufficient for updating the image displayed on the display device 13 appropriately. .

  The data amount of the image necessary and sufficient for displaying the image on the display device 13 increases as the speed of the vehicle 20 increases. For example, as shown in FIG. 3, in "very low speed travel", the sum of the data amounts of overhead view information, vehicle information, right camera (low image quality) and left camera (low image quality) displays an image on the display device 13. In “medium-speed travel”, the data of each of the central camera (low image quality), vehicle information, right camera (high image quality), left camera (high image quality), and display information The sum of the amounts is the data amount of the image necessary and sufficient for displaying the image on the display device 13. This is because as the speed of the vehicle 20 increases, the amount of image data for realizing the image quality and the image range (field of view) required for the operator of the remote control device 10 to safely operate the vehicle 20 increases. is there.

  When the communication environment deteriorates, depending on the traveling state of the vehicle 20, a relatively large delay may occur in reception of data transmitted from the vehicle 20. For example, when the vehicle 20 is traveling at "medium speed", communication of about 22 Mbps is required as shown in FIG. 3 in order to properly receive the minimum data necessary to realize "medium speed traveling". You need speed. If the communication environment deteriorates and the actual communication speed becomes slower than 22 Mbps, a relatively large delay occurs in reception of the minimum data necessary to realize "medium-speed travel". As a result, the image stored in the buffer memory is reduced.

  Therefore, when the delay time of data reception becomes longer than a predetermined time (for example, 300 milliseconds), the decoder and superimposition display unit 12 reduces the communication speed necessary to properly receive the data transmitted from the vehicle 20. In order to do so, for example, a warning notice such as "Please run at low speed!" Is given (step S215). This is because, as shown in FIG. 3, the lower the speed of the vehicle 20, the lower the communication speed necessary to properly transmit the minimum data necessary to realize the traveling state of the vehicle 20. .

  The fact that the delay time of data reception is longer than the predetermined time means that an image having a sufficient amount of data necessary to display an image on display 13 is not stored in the buffer memory (ie, stored in buffer memory). The data amount of the image is less than a predetermined amount).

  If the operator of the remote control device 10 decelerates the vehicle 20 in response to the warning notification, the amount of image data necessary and sufficient for displaying an image on the display device 13 also decreases, so the communication speed is relatively slow. Even in this case, data relating to an image necessary and sufficient for displaying the image on the display device 13 can be properly received. That is, the amount of data necessary for displaying the image on the display device 13 and sufficient for updating the image displayed on the display device 13 appropriately is stored in the buffer memory.

  The decoder and superimposition display unit 12 performs the process of steps S221 to S223 in parallel with the process of steps S211 to S215. That is, the decoder / superimposition display unit 12 determines the communication environment based on the communication status between the remote control device 10 and the vehicle 20 (step S221). As described above, the determination result of the communication environment determination is represented by, for example, a four-step evaluation of “best”, “good”, “bad”, and “worst”.

  In parallel to the process of step S221, the decoder and superimposition display unit 12 performs buffer determination based on the image stored in the buffer memory (step S222). For example, it is assumed that image data of three images (about 100 milliseconds) can be stored in the buffer memory. When image data for three images is stored in the buffer memory, the determination result of the buffer determination is “best”. When image data for two images is stored in the buffer memory, the determination result of the buffer determination is “good”. When image data of one image is stored in the buffer memory, the determination result of the buffer determination is “bad”. When image data is not stored in the buffer memory, the determination result of the buffer determination is “worst”.

  Next, the decoder / superimposition display unit 12 transmits the determination result of the communication environment determination and the determination result of the buffer determination to the vehicle 20 via the wireless device 11 (step S223). After the third predetermined period (for example, several milliseconds to several tens of milliseconds) has elapsed, the decoder and superposition display unit 12 starts the process of step S221.

(Technical effect)
When an image captured by each of the central camera 23C, the left camera 23L and the right camera 23R of the vehicle 20 is transmitted to the remote control device 10 via wireless communication between the remote control device 10 and the vehicle 20, as shown in FIG. Thus, although the vehicle 20 periodically transmits the image, the remote control device 10 receives the image with an irregular delay (two steps from the top and the top of FIG. 4). See the eyes). Each rectangle in FIG. 4 represents an image, and the numbers assigned to the rectangles indicate frame numbers.

  In the remote control device 10, the received image is temporarily stored in the buffer memory, and the stored image is read from the buffer memory at a predetermined cycle. At this time, the image displayed to the operator of the remote control device 10 can be periodically updated if the cycle of reading the image from the buffer memory is later than the cycle of the vehicle 20 transmitting the image (Fig. 4). See bottom).

  As described above, even if the buffer memory is used, the reception of the image transmitted from the vehicle 20 is significantly delayed when the communication environment is deteriorated if the data amount for one image is always constant. It is possible that the images stored in the memory may be reduced or lost, and the image displayed to the operator of the remote control device 10 (i.e., the image displayed on the display device 13) may not be updated regularly.

  In the remote control system 1, when the delay time of data reception is longer than a predetermined time, a warning is issued to urge the vehicle 20 to decelerate in order to make the traveling condition of the vehicle 20 match the current communication environment or the like. As a result, if the operator of the remote control device 10 decelerates the vehicle 20, the delay time of data reception can be shortened. Therefore, the deterioration of the image stored in the buffer memory can be suppressed.

  According to the remote control system 1, the operator of the remote control device 10 is used by using a buffer memory and issuing a warning to the operator of the remote control device 10 when the delay time of data reception is longer than a predetermined time. It is possible to prevent the image displayed on the screen from becoming unstable.

<Modification>
In the above-described embodiment, in the process of step S215 of FIG. 3, when the delay time of data reception is larger than the predetermined time, a warning for prompting the vehicle 20 to decelerate is issued. Instead of the process of step S215, the decoder and superimposition display unit 12 can display an image on the display device 13 for a predetermined time by the image stored in the buffer memory based on the determination result of the buffer determination, for example. If less than, a warning may be issued prompting the vehicle 20 to decelerate.

  Alternatively, the decoder and superimposition display unit 12 may use the vehicle, for example, when the time in which the image can be displayed on the display device 13 is less than a predetermined time by the image stored in the buffer memory based on the determination result of the buffer determination. A signal indicating that the speed 20 should be decelerated may be transmitted to the vehicle 20 via the wireless device 11 (as a result, for example, the brake control ECU 26 is controlled by the vehicle control ECU 24 to decelerate the vehicle 20).

  The present invention is not limited to the embodiment described above, but can be suitably modified without departing from the scope or spirit of the invention as can be read from the claims and the specification as a whole, and a remote control device with such a modification Also within the technical scope of the present invention.

  1 ... remote control system, 10 ... remote control device, 20 ... vehicle

Claims (1)

A remote control device capable of remotely operating a vehicle to be operated,
An image of the surroundings of the vehicle is shown via wireless communication, and image data whose amount of data increases as the speed of the vehicle increases is sequentially acquired, and the acquired image data is stored in a buffer of a predetermined capacity. Image acquisition means to
Display means for reading out the image data stored in the buffer and displaying the image indicated by the read out image data as a display image;
Control means for performing a predetermined operation leading to the deceleration of the vehicle on condition that the displayable time of the display image is less than a predetermined time by the image data stored in the buffer;
A remote control device comprising:

Images