JP5174636B2 - Remote control system and remote control device - Google Patents

Description

  The present invention relates to a remote operation system and a remote operation device, and more particularly to a remote operation system and a remote operation device for remotely operating a moving body.

In general, a remote operation system is known in which a moving body such as an automobile is remotely operated by a remote operation device located away from the moving body. Conventionally, various techniques for facilitating remote operation of a moving body in such a remote operation system have been proposed. As an example, Patent Document 1 discloses a technique in which a moving body image is combined with an environment image previously captured by a camera provided on a moving body and displayed on a monitor of a remote control device. The operator remotely operates the moving body using the input device of the remote operation device while confirming the image displayed on the monitor.
JP 2005-208857 A

  However, when performing remote operation as in Patent Document 1, since the actual behavior of the moving body is displayed with a delay on the monitor, it is not easy to grasp the appropriate input amount to the input device from the monitor image. It is necessary, and it is difficult for an unfamiliar operator to remotely control the moving body as desired. In particular, when the mobile body is turned right or left, if the operator is not accustomed, there is a possibility that the mobile body may be meandered by repeating a sudden input (rapid operation) to the input device to correct the orientation.

  SUMMARY OF THE INVENTION Therefore, a main object of the present invention is to provide a remote operation system and a remote operation device that allow an inexperienced operator to easily remotely operate a moving body.

In order to achieve the above object, an image processing system according to claim 1 is a remote operation system including a moving body having a pair of left and right front wheels and a remote operation device for remotely operating the moving body, The moving body includes an imaging unit that captures an environment image around the moving body, and an information detection unit that detects information related to the moving body and the imaging unit. Input means for steering the vehicle, determination means for determining a virtual viewpoint based on the detection result of the information detection means, the environment image captured from the virtual viewpoint by the imaging means, and the environment image at the time of imaging First generation means for generating, in a virtual space, a three-dimensional environment image representing the environment image in three dimensions and the virtual viewpoint based on a detection result of the information detection means; Second generating means for generating the moving body model in the virtual space based on a detection result of the information detecting means after capturing the environmental image and data relating to the moving body model representing the moving body, and the virtual Based on the detection result of the information detection unit and the input amount from the input unit after capturing the environmental image from the viewpoint, the input amount from the input unit at the time of forward movement, left turn and right turn of the moving body Generating a composite image by perspectively projecting the three-dimensional environment image viewed from the virtual viewpoint, the moving body model, and the auxiliary model; a fourth generating unit, e Bei and display means for displaying the composite image, the auxiliary model, left front wheel and right perpendicular to the traveling direction of the mobile body model and the mobile body model Characterized in that it comprises a first auxiliary model that extends transversely to the moving body model while through the midpoint between the front wheel.

A remote operation system according to a second aspect is the remote operation system according to the first aspect, wherein the auxiliary model includes a second auxiliary model representing a movement path of the moving body model.

The remote control device according to claim 3 , comprising: an imaging unit that captures an environment image around a moving body having a pair of left and right front wheels ; and an information detection unit that detects information about the moving body and the imaging unit. A remote control device for remotely operating a mobile body, wherein an operator steers the mobile body, a determination means for determining a virtual viewpoint based on a detection result of the information detection means, and the imaging A three-dimensional environment image representing the environment image in three dimensions and the virtual viewpoint based on the environment image picked up from the virtual viewpoint by the means and the detection result of the information detection means at the time of picking up the environment image First generation means for generating in a virtual space, detection result of the information detection means after imaging the environmental image from the virtual viewpoint, and data relating to a moving body model representing the moving body Based on the second generation means for generating the moving body model in the virtual space, the detection result of the information detection means after imaging the environmental image from the virtual viewpoint, and the input amount from the input means Based on the virtual viewpoint, the third generating means for generating an auxiliary model serving as an index of the input amount from the input means at the time of forward movement, left turn and right turn of the mobile body, and the 3 viewed from the virtual viewpoint e Bei fourth generation means for generating a composite image as the dimension environmental image with the mobile model and the auxiliary model perspective projection to, and display means for displaying the composite image, the auxiliary model, the movable body characterized in that it comprises a first auxiliary model that extends transversely to the perpendicular to the traveling direction of the model and the mobile body model while through an intermediate point between the left front wheel and right front wheel of the moving body model

In the remote operation system according to claim 1, a virtual viewpoint is determined based on a detection result of the information detection unit, and an environment image captured from the virtual viewpoint by the imaging unit and detection of the information detection unit at the time of capturing the environment image Based on the result, a three-dimensional environment image and a virtual viewpoint are generated in the virtual space. In addition, the moving body model is generated in the virtual space based on the detection result of the information detecting unit after capturing the environmental image from the virtual viewpoint and the data related to the moving body model. Further, based on the detection result of the information detection unit and the input amount from the input unit after capturing the environmental image from the virtual viewpoint, an auxiliary model serving as an index of the input amount from the input unit is generated in the virtual space. Then, a three-dimensional environment image viewed from the virtual viewpoint, the moving body model, and the auxiliary model are perspective-projected to generate a composite image, and the composite image is displayed on the display unit. Since the auxiliary model appearing in the composite image is generated by the remote control device in accordance with the input amount from the input means, it substantially corresponds to the input amount from the current input means. By confirming the positional relationship between such an auxiliary model and the shoulder of the road in the composite image, the operator can input the amount of input to the input means necessary for moving the moving body model and thus the moving body in an appropriate direction. Easy to grasp. Therefore, even an inexperienced operator can easily remotely control the moving body, and can quickly move the moving body to the destination. In addition, a first auxiliary model that is orthogonal to the traveling direction of the moving body model and extends in the lateral direction with respect to the moving body model is included in the auxiliary model. The operator can easily grasp an appropriate input amount to the input means by confirming how the road shoulder and the first auxiliary model intersect in the composite image. The first auxiliary model is particularly effective when the road is sharply bent. In this case, by adjusting the input amount to the input means so that the first auxiliary model is substantially orthogonal to the shoulder of the road, even an unfamiliar operator can easily and stably operate the moving body. it can. The same applies to the remote control device according to claim 3 .

The remote control system according to claim 2, the second auxiliary model that extends along a moving path of the mobile body model is included in the auxiliary model. The operator can easily grasp the appropriate input amount to the input means by confirming the gap between the shoulder of the road and the second auxiliary model in the composite image. The second auxiliary model is particularly effective when the road is gently curved. In this case, by adjusting the input amount to the input means so that the distance between the road shoulder and the second auxiliary model is constant, even an unfamiliar operator can easily and stably operate the moving body. Can do.

  According to the present invention, it is possible to obtain a remote operation system and a remote operation device that enable an inexperienced operator to easily remotely operate a moving body.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a remote control system 10 according to an embodiment of the present invention. FIG. 2 is a left side view showing the moving body 12.

Referring to FIG. 1, a remote operation system 10 includes a moving body 12 and a remote operation device 14 for remotely operating the moving body 12.
Referring also to FIG. 2, the moving body 12 is, for example, a four-wheel buggy for traveling on rough land and is used for farm work, surveying, etc., and includes a microcomputer (hereinafter referred to as a microcomputer) 16, a speed detector 18, a camera 20, It includes a positioning unit 22, an attitude sensor 24, an orientation sensor 26, a hard disk drive (including a hard disk: hereinafter referred to as HDD) 28, a drive unit 30, and a communication device 32.

Speed detector 18 is attach to the rotation axis near the left front wheel 30a, for detecting the speed of the moving body 12 based on the rotational speed of the rotary shaft.
The camera 20 is provided slightly above the center of the front surface of the moving body 12 and captures an environmental image in front of the moving body 12. The camera 20 is a digital CCD camera having a lens 20a. It is desirable that the lens 20a has a wide angle. In this embodiment, an all-around fisheye lens having an angle of view of 180 ° is used. As a result, the camera 20 can capture a wide-angle image. Here, the center point of the lens 20 a is the viewpoint A of the camera 20, and the position of the viewpoint A is the position of the camera 20.

  The positioning unit 22 is provided at a reference point B indicating the position of the moving body 12 in order to detect the position of the moving body 12. In this embodiment, the reference point B is set at the intersection of a straight line drawn from the viewpoint A directly behind in the horizontal direction and a perpendicular passing through an intermediate point between the two rear wheels 30b. The positioning unit 22 detects the position of the reference point B by receiving a signal from a GPS (Global Positioning System) satellite.

The posture sensor 24 is, for example, a 3D gyro sensor or a tilt sensor, and detects the left / right tilt (roll angle) of the moving body 12 and the front / back tilt (pitch angle) of the moving body 12.
The direction sensor 26 detects the forward direction (azimuth angle) of the moving body 12.

  The HDD 28 serving as storage means is a device for writing and reading programs, control data, text data, image data, and the like to and from the hard disk in accordance with instructions from the microcomputer 16. The stored environmental image data and various programs are stored (stored).

  In addition to the two front wheels 30a and the two rear wheels 30b, the drive unit 30 includes a steering motor for moving the steering mechanism of the moving body 12, a throttle motor for moving the throttle, a brake motor for moving the brake lever, and the like. .

  The communication device 32 is used for communicating the mobile body 12 and the remote control device 14 via the communication line T. The communication line T may be either wireless or wired. For example, data communication is performed using a wireless LAN or a mobile phone line. The communication speed of the communication line T is, for example, about 250 kbps.

  The microcomputer 16 includes a CPU, a RAM, a ROM, a clock, and the like, and controls the operation of the moving body 12. The microcomputer 16 acquires the detection results of the positioning unit 22, the attitude sensor 24, and the direction sensor 26, for example, every 0.02 seconds. At the same time, the microcomputer 16 calculates the cumulative moving distance of the moving body 12 after the microcomputer 16 is activated based on the detection result of the speed detector 18. The position, orientation, orientation, and cumulative moving distance of the moving body 12 are stored in the HDD 28 as “moving body information” together with the data acquisition time when these are acquired, and transmitted from the communication device 32 to the remote control device 14. . For example, the microcomputer 16 acquires image data of the environment image from the camera 20 every 3 seconds, for example, and acquires camera information (consisting of the position, orientation, and orientation of the camera 20) at the time of capturing the environment image. These image data and camera information are stored in the HDD 28 as “past information” and transmitted from the communication device 32 to the remote control device 14 together with the cumulative moving distance and imaging time of the moving body 12 when the environmental image is captured. .

In this embodiment, the camera 20 is fixed to the moving body 12, and the distance between the viewpoint A indicating the position of the camera 20 and the reference point B indicating the position of the moving body 12 is constant. Accordingly, the coordinate position shifted from the position of the moving body 12 by the distances in the front and rear, left and right, and up and down directions (here, only in the front and rear direction) from the position of the moving body 12 to the position of the camera 20 is defined as the position of the camera 20. Can be acquired. Moreover, since the camera 20 and the moving body 12 have the same posture and orientation, the posture and orientation of the moving body 12 can be used as they are. Therefore, camera information at the time of capturing an environmental image can be obtained based on moving body information at that time.
The drive unit 30 is also controlled by the microcomputer 16.

Returning to FIG. 1, the remote control device 14 will be described.
The remote operation device 14 includes, for example, a control unit 34 composed of a personal computer, a monitor 36 composed of, for example, a liquid crystal monitor for displaying an image or the like for supporting the operation of the movable body 12 by the operator, and the operator controls the movable body 12. An input device 38 for operation and a communication device 40 for communicating with the communication device 32 of the mobile unit 12 via the communication line T are included.

The control unit 34 includes a CPU 44 and a ROM 46, a RAM 48 and an HDD 50 which are storage means connected to each other via a bus 42.
The CPU 44 executes various programs stored in the ROM 46, the HDD 50, etc., which are storage means, gives instructions to each component, and controls the operation of the remote operation device 14.

  The HDD 50 is a device for writing and reading programs, control data, text data, image data, and the like to and from the hard disk in accordance with instructions from the CPU 44. The HDD 50 is a table showing the correspondence between the image processing program for executing the operation shown in FIG. 3, data relating to the moving body model M, the steering angle of the steering wheel of the operation unit 52a (described later) and the steering angles of the two front wheels 30a. The data and past information transmitted from the mobile body 12 are stored (stored).

  The ROM 46 stores a startup program and the like. The activation program is executed by the CPU 44 when the control unit 34 is powered on. As a result, a program such as an operating system (OS) recorded in the HDD 50 is loaded into the RAM 48, and various processes and controls can be executed.

  In the RAM 48, an image processing program or the like for executing the operation shown in FIG. 3 is developed, and processing results by this program, temporary data for processing, display data (image data, text data, etc.) and the like are held. . The RAM 48 is used as a work area for the CPU 44. The display data developed on the RAM 48 is transmitted to the monitor 36. The monitor 36 displays display contents (images, texts, etc.) corresponding to the display data from the RAM 48.

The input device 38 includes a steering unit 52 for steering the left and right front wheels 30a of the moving body 12. The steering unit 52 includes an operation unit 52a having a handle (steering wheel) operated by an operator, and a steering angle sensor 52b that detects a steering angle (cutting angle) of the handle in the operation unit 52a. A command related to the steering of the two front wheels 30a is transmitted to the moving body 12 based on the detection result of the steering angle sensor 52b, that is, the input amount from the steering unit 52. As a result, the steering mechanism of the moving body 12 is driven, and the steering angles of the two front wheels 30a are changed. As the steering mechanism of the moving body 12, an Ackermann mechanism is used in which the steering angle of the inner wheel is larger than the steering angle of the outer wheel in the two front wheels 30a.
The input device 38 includes units for throttle operation, brake operation, and the like, and commands based on operation inputs to these are transmitted from the remote operation device 14 to the moving body 12.

  In this embodiment, the camera 20 corresponds to an imaging unit. The information detection means includes a speed detector 18, a positioning unit 22, an attitude sensor 24 and an azimuth sensor 26. The steering unit 52 corresponds to input means. The CPU 44 of the remote operation device 14 also functions as a determination unit and first to fourth generation units. The monitor 36 corresponds to display means.

  Next, an example of an image processing operation in the remote operation device 14 of the remote operation system 10 will be described with reference to FIG. In the remote control device 14, image processing is performed to display on the monitor 36 a state in which the traveling moving body 12 is viewed from behind.

  First, the CPU 44 of the remote control device 14 determines whether or not past information from the moving body 12 has been received (step S1). If past information is received, the past information is stored in the HDD 50 (step S3). The process proceeds to step S5. On the other hand, if past information is not received in step S1, the process proceeds directly to step S5. In step S5, the CPU 44 determines whether or not the latest moving body information at the current time has been received. If the mobile body information from the mobile body 12 is not received, the process returns to step S1. On the other hand, if the mobile body information is received, the CPU 44 stores the past information stored in the HDD 50 based on the mobile body information. Past information is selected from among them, and a virtual viewpoint V is determined (step S7). The virtual viewpoint V refers to a viewpoint set behind the current position of the moving body 12, and in this embodiment, the position of the imaging means (position of the viewpoint A of the camera 20) at the time of environmental image capturing up to the present time. It is selected from the inside.

Here, a selection method of past information (mainly environment image and virtual viewpoint) will be described with reference to FIG.
FIG. 4 is an illustrative view for explaining a method of selecting past information. In FIG. 4, the horizontal axis is the cumulative moving distance of the moving body 12. F1, F2, and F3 indicated by white circles indicate the cumulative movement distances included in the past information, and the cumulative movement distance increases as the direction of the arrow moves. The environmental image captured when the cumulative moving distance is F1, F2, and F3 and the position of the viewpoint A of the camera 20 are included in the past information and stored in the HDD 50, respectively. N is the cumulative moving distance at the current position of the moving body 12, and FIG. 4 (b) shows a state later in time than FIG. 4 (a).

In this embodiment, past information obtained at a position closest to a point in front of the current position of the moving body 12 by a predetermined distance G (for example, 6 m) is selected, and the shooting point of the environmental image included in this past information is a virtual point. Viewpoint V is assumed. That is, in step S7, the CPU 44 obtains a distance obtained by subtracting the predetermined distance G from the accumulated movement distance N included in the moving body information acquired in step S5, and selects past information including the accumulated movement distance closest to the distance. Is done.
Specifically, it is selected as follows.
The distances D1 and D2 shown in FIG.

  Since D1 <D2 at the time of FIG. 4A, the past information including the cumulative movement distance F1 is selected. That is, the environmental image captured when the cumulative movement distance is F1 is selected, and the position of the viewpoint A of the camera 20 when the cumulative movement distance is F1 is determined as the virtual viewpoint V. At the time of FIG. 4B when the current cumulative travel distance N becomes larger than that in FIG. 4A, since D1> D2, past information including the cumulative travel distance F2 is selected. That is, the environment image captured when the cumulative movement distance is F2 is selected, and the position of the viewpoint A of the camera 20 when the cumulative movement distance is F2 is determined as the virtual viewpoint V.

Returning to FIG. 3, thereafter, the CPU 44 generates a three-dimensional environment image K and a virtual viewpoint V based on the environment image I in the global coordinate system GC as a virtual space based on the past information selected in step S7. (Step S9).
This process will be described with reference to FIGS.

  FIG. 5 is an illustrative view showing a two-dimensional orthogonal coordinate system C1 composed of a y-axis and a z-axis on which an environmental image I captured by the camera 20 is represented. The environment image I is a two-dimensional fisheye image (circular image) including the entire landscape in front of the lens 20a. The center point of the environmental image I corresponds to the viewpoint A (see FIG. 2) of the camera 20. The camera photographing center direction (x-axis direction in the coordinate system C2 shown in FIG. 6) is a direction orthogonal to the paper surface of FIG.

  FIG. 6 is an illustrative view showing a three-dimensional orthogonal coordinate system C2 on which a three-dimensional environment image K obtained by three-dimensionally converting the two-dimensional environment image I is displayed. In the coordinate system C2, the origin O is the viewpoint A of the camera 20, the x-axis direction is the camera photographing center direction, and the y-axis direction is the horizontal direction orthogonal to the x-axis direction when the x-axis direction is one of the horizontal directions. The other direction, the z-axis direction, is the vertical direction when the x-axis direction is one of the horizontal directions.

  FIG. 7 is an illustrative view showing a global coordinate system GC as a virtual space. The global coordinate system GC is a three-dimensional orthogonal coordinate system including an X axis, a Y axis, and a Z axis. The origin O of the global coordinate system GC is the position of the viewpoint A of the camera 20 when the microcomputer 16 of the moving body 12 is activated. The X-axis direction is the north-south direction, the Y-axis direction is the east-west direction, and the Z-axis direction is the vertical direction.

  The planar-shaped environment image I shown in FIG. 5 is converted into a three-dimensional environment image K having a three-dimensional shape (here, hemispherical) shown in FIG. 6 and generated in the global coordinate system GC shown in FIG. The coordinates P (x, y, z) of the three-dimensional environment image K in the coordinate system C2 are obtained by Expression 2 based on the coordinates P (py, pz) of the environment image I in the coordinate system C1. P (x, y, z) is a point on the hemisphere (three-dimensional environment image K) shown in FIG.

  Here, φ is an angle formed by the line segment OP in the y-axis plus direction from the origin O, and the range is 0 ≦ φ <2π. φ can be obtained by using (py, pz) and an inverse function (arctan) of tan. Further, θ is an angle formed by a line segment PO connecting the origin O and the point P and the x axis, as shown in FIG. The shape of the three-dimensional environment image K is determined by the characteristics of the lens 20a. From this, θ is obtained by calculating using a predetermined mathematical formula based on the coordinates P (py, pz) of the environment image I or by obtaining from the table data stored in the HDD 50. θ is 0 ≦ θ ≦ π / 2. Further, J is the distance from the origin O to the point P (radius of the three-dimensional environment image K) as shown in FIG. 6, and is set to an arbitrary value. In this embodiment, J is set to 10 m, for example.

Specifically, the coordinates P (x, y, z) of the three-dimensional environment image K in the coordinate system C2 are obtained as follows.
First, the environment image I is arranged in the coordinate system C1. The environment image I is arranged so that the center point thereof coincides with the origin O of the coordinate system C1. The environment image I is divided into a plurality of square image pieces IP (for example, vertical 16 × horizontal 16 = 256: vertical 6 × horizontal 6 = 36 in FIG. 5 for simplification of the drawing), and the coordinate system C1. The coordinate P (py, pz) of the vertex of each image piece IP in is acquired. Subsequently, based on the acquired coordinates P (py, pz), the coordinates P (x, y, z) in the coordinate system C2 are obtained by Equation 2.

In this way, the coordinates P (x, y, z) in the coordinate system C2 are obtained for the four vertices of the image piece IP. Then, the coordinates P (x, y, z) of the four vertices of the image piece IP in the coordinate system C2 are respectively the postures of the camera 20 (roll angle r, pitch angle p) included in the past information at the time of shooting from the virtual viewpoint V. ) And orientation (azimuth angle y) and the position (x _c , y _c , z _c ) of the camera 20 are converted into coordinates (x _G , y _G , z _G ) in the global coordinate system GC. Such coordinate conversion can be generally performed by rotation and translation around an axis, and is performed by Equation 3.

  Here, the roll angle r, the pitch angle p, and the azimuth angle y are called Euler angles, respectively. In the global coordinate system GC, the rotation angle around the x axis, the rotation angle around the y axis, and the z axis around the coordinate system C2. It is a rotation angle, and the direction in which each axis right-hand thread advances is positive. The origin of roll and pitch is the vertical upward direction of the global coordinate system GC, and the origin of orientation is the true north direction of the global coordinate system GC.

In this way, the coordinates of the four vertices of the image piece IP in the global coordinate system GC are obtained. That is, the position of the image piece IP in the global coordinate system GC is specified. Then, the image piece IP is generated (arranged) at the specified position. By performing this process for all image pieces IP, as shown in FIG. 7, a three-dimensional environment image K representing the environment image I in three dimensions in the three-dimensional space of the global coordinate system GC is generated. Along with the generation of the three-dimensional environment image K, the virtual viewpoint V located at the origin O of the coordinate system C2 is generated at the position (x _c , y _c , z _c ) of the camera 20 in the global coordinate system GC.

  A graphic library that can be generally used can be used for the processing in step S9. In this embodiment, OpenGL (registered trademark) provided by Silicon Graphics is used as the graphic library. The process of step S9 is a function {glTexcoord2D (px, py) and glVertex3D (x, y, z) provided by OpenGL (registered trademark): px, py are coordinate values in the coordinate system C1 shown in FIG. , X, y, z are coordinate values in the coordinate system GC shown in FIG.

  The actual three-dimensional environment image K becomes a polyhedral (256-faced) dome shape by connecting the planar image pieces IP together. 6 and 7 show a hemispherical three-dimensional environment image K for simplification of the drawings.

Returning to FIG. 3, after step S9, the moving body model M, the first auxiliary model M1, the second auxiliary model M2, the reference point B, and the cut-out center point Q are generated in the global coordinate system GC by the CPU 44 (see FIG. 3). Step S11). Based on the latest mobile body information received in step S5 and data related to the mobile body model M stored in the HDD 50, the CPU 44 uses the mobile body model M representing the mobile body 12 and the reference point B indicating the position of the mobile body 12. , And a cut-out center point Q, which is the center when cutting out the projected image, is generated in the global coordinate system GC. The CPU 44 also serves as a first indicator of the input amount from the steering unit 52 based on the latest moving body information received in step S5 and the detection result of the steering angle sensor 52b (input amount from the steering unit 52). The auxiliary model M1 and the second auxiliary model M2 are generated in the global coordinate system GC.
These processes will be described with reference to FIGS.

First, a method for generating the moving object model M will be described with reference to FIG. FIG. 8 is an illustrative view for explaining a method of generating a mobile object model. In order to simplify the description, the moving body model shown in FIG.
As shown in FIG. 8, the moving body model can express an image of the shape of the moving body by a point group and a straight line connecting two corresponding points among them. The moving body model is arranged in the three-dimensional orthogonal coordinate system C3 such that its center is located at the origin O, its front faces the x-axis direction, and its upper side faces the z-axis direction. A coordinate value is given to each vertex of the moving body model. For example, if the length of each side of the cube indicating the moving body model is 2, the vertex S1 is (1, 1, 1) and the vertex S2 is (1,- 1,1).
Then, each vertex of the cube is coordinate-transformed into the global coordinate system GC by Equation 4.

Here, (x _m , y _m , z _m ) is the current position of the moving object 12 in the global coordinate system GC obtained from the moving object information. (X _b , y _b , z _b ) are the coordinate values (coordinate values to be subjected to coordinate conversion) of the points constituting the moving body model in the coordinate system C3. (X _a , y _a , z _a ) are coordinate values after (x _b , y _b , z _b ) are transformed into the global coordinate system GC. r, p, and y are the current roll angle, pitch angle, and azimuth angle of the moving body 12 in the global coordinate system GC obtained from the moving body information. In this embodiment, the roll angle r, pitch of the camera 20 It is equivalent to the angle p and the azimuth angle y.

  Then, a cuboid moving body model is generated in the global coordinate system GC by connecting the vertices whose coordinates are converted to the global coordinate system GC with straight lines. That is, the entire moving body model is converted into an image of the global coordinate system.

  The actual moving body model M forms a model of the moving body 12 by a point group and a straight line connecting two corresponding points. The actual moving body model M is arranged in the coordinate system C3 such that the reference point B is located at the origin O, the front thereof faces the x-axis direction, and the upper side faces the z-axis direction. As data relating to the moving body model M, coordinate values in the coordinate system C3 are assigned to at least each point of the point group. As in step S9, the moving body model M can be generated in the global coordinate system GC by performing coordinate transformation of each point of the point group using Equation 4 and connecting the corresponding two points with a straight line (see FIG. 7).

  In this way, the moving body model M is generated in the global coordinate system GC as a virtual space. Similarly, the reference point B indicating the position of the moving body 12 and the cut-out center point Q that is the center when the projected image is cut out are also generated in the global coordinate system GC. In this embodiment, the cut-out center point Q is set at a position that is a predetermined distance away in front of the moving object model M. Specifically, for example, the cut center point Q is set at a position 6 m ahead of the reference point B.

  If the moving body model does not include a straight line and is expressed only by the point group, the coordinate values of all the points in the point group may be transformed into the global coordinate system GC by Equation 4.

  Further, the coordinate values of the point group representing the front wheel and the rear wheel of the moving body model M are changed according to the input amount from the input device 38. As a result, it is possible to express a state in which the front wheel of the moving body model M is tilted to the left or right.

Next, a method for generating the first auxiliary model M1 will be described with reference to FIG.
FIG. 9 is an illustrative view for explaining a method of generating the first auxiliary model M1. FIG. 9 shows a state in which the coordinate system C3 is viewed from the positive side of the z axis (see FIG. 8). FIG. 9 shows a state in which the handle of the operation unit 52a is turned to the left (state when turning left). In order to simplify the explanation, first, the x-axis coordinate value and the y-axis coordinate value in the coordinate system C3 will be considered.

  As shown in FIG. 9, the first auxiliary model M1 is expressed as a linear image by connecting a point EL on the left side of the moving body model M and a point ER on the right side of the moving body model M. In this embodiment, the point EL is set to the turning center point R of the moving body model M when making a left turn. In general, in a four-wheeled vehicle such as the moving body 12, the turning center point is a point where the extension lines of the rotation axes of the left and right rear wheels and the left and right front wheels that are steering wheels intersect. The extension line of the rotation axis of the rear wheel of the moving body model M coincides with the y axis, and the turning center point R is the point where the extension lines of the rotation axes of the left and right front wheels intersect on the y axis. From this, the turning center point R is represented by (0, r). The y-axis coordinate value r of the turning center point R is obtained by Equation 5.

  Here, L is the distance between the reference point B and the intermediate point FM between the center point FL of the left front wheel and the center point FR of the right front wheel. That is, L is the wheel base (distance between the front and rear axles) of the moving body 12. W is the distance between the center point FL and the center point FR. That is, W is the distance between the center points of the left and right front wheels. The coordinate values of the center points FL, FR and the intermediate point FM, and the values of L and W are stored in the HDD 50 in advance. αL is the rudder angle of the left front wheel. It can be seen from FIG. 9 that the angle formed between the straight line connecting the center point FL and the turning center point R and the y axis is also αL. The CPU 44 acquires the steering angle αL from the table data in the HDD 50 based on the input amount from the steering unit 52. In this way, the y-axis coordinate value r of the turning center point R and then the point EL (one end point of the first auxiliary model M1) is obtained by Equation 5.

  Further, the point ER (the other end point of the first auxiliary model M1) is on the extended line of the straight line connecting the point EL and the intermediate point FM and at a position separated from the intermediate point FM by the distance from the point EL to the intermediate point FM. Is set. Therefore, the point ER can be represented by (2L, -r).

  The traveling direction of the moving body model M predicted at the time of right / left turn (here, at the time of left turn) is a direction orthogonal to a straight line connecting the turning center point R and the intermediate point FM. Therefore, the first auxiliary model M1 expressed by a straight line connecting the point EL and the point ER is orthogonal to the traveling direction of the moving body model M and extends in the lateral direction with respect to the moving body model M.

Note that the y-axis coordinate value r of the turning center point R (point EL) may be obtained using the steering angle αR of the right front wheel instead of the steering angle αL. In this case, in Equation 5, αR may be used instead of αL, and W / 2 may be subtracted instead of adding W / 2.
Further, when turning right, the point ER is set as the turning center point R to obtain the y-axis coordinate value r of the point ER, and the coordinate value of the point EL may be (2L, r).

  On the other hand, when the steering angle of the handle of the operation unit 52a is 0 °, that is, when the moving body model M goes straight, the point EL is represented by (L, r), and the point EL is represented by (L, -r). . In this case, a fixed value stored in advance in the HDD 50 is used as the y-axis coordinate value r. In FIG. 9, the point EL and the point ER in this case are indicated by white circles, and a straight line connecting them is indicated by a one-dot chain line. Also in this case, the first auxiliary model M1 has an image that is orthogonal to the traveling direction of the moving body model M and extends in the lateral direction with respect to the moving body model M.

  Here, as the z-axis coordinate values of the points EL and ER in the coordinate system C3, fixed values stored in advance in the HDD 50 may be used. In this embodiment, the z-axis coordinate values of the point EL and the point ER are set so as to correspond to the difference Hz (see FIG. 2) between the height of the reference point B and the height of the lower end of the front wheel 30a. Therefore, in this embodiment, the first auxiliary model M1 is expressed as a straight line passing below the moving body model M.

  The first auxiliary model M1 is generated in the global coordinate system GC by converting the coordinates of the point EL and the point ER obtained in this way using Equation 4 in the same manner as described above and connecting them with a straight line.

Next, a method for generating the second auxiliary model M2 will be described with reference to FIGS.
10 and 11 are illustrative views for explaining a method for generating the second auxiliary model M2. 10 and 11 show a state in which the coordinate system C3 is viewed from the positive side of the z axis (see FIG. 8). FIG. 10 shows a state when turning left, and FIG. 11 shows a state when going straight. Similar to the first auxiliary model M1, the x-axis coordinate value and the y-axis coordinate value in the coordinate system C3 are first considered.

  As shown in FIG. 10, the second auxiliary model M2 includes a point group GL and a point group GR that represent the movement path of the moving body model M. The point group GL includes points PL1 to PL8, and the point group GR includes points PR1 to PR8.

  When turning right or left (here, when turning left), the center point FL moves on the circumference of the radius r1 with the turning center point R as the center. The points PL1 to PL8 constituting the point group GL are arranged in order from the center point FL side with a predetermined interval IN on the circumference so as to represent the movement path of the center point FL. The coordinate values of such points PL1 to PL8 are obtained by Equation 6.

Here, n is an integer (in this case, 1 to 8) indicating what number point PL is seen from the center point FL. Further, r1 which is the turning radius of the center point FL is obtained by the square root of L ² + (r−W / 2) ² using r obtained by the above equation 5. θL is an angle formed by a straight line connecting the center point FL and the turning center point R and a straight line connecting the point PL1 and the turning center point R, and is obtained by IN / r1 [rad]. The predetermined interval IN is a fixed value stored in advance in the HDD 50, and is 0.5 m, for example. It should be noted that the predetermined interval IN at the time of turning left and right is the length of an arc connecting point PLn and point PLn + 1.

  Further, when turning right or left (here, when turning left), the center point FR moves on the circumference of the radius r2 centering on the turning center point R. The points PR1 to PR8 constituting the point group GR are sequentially arranged from the center point FR side with a predetermined interval IN on the circumference so as to represent the movement path of the center point FR. The coordinate values of such points PR1 to PR8 are obtained by Equation 7.

Here, n is an integer (1 to 8 in this case) indicating what number point PR is seen from the center point FR. Further, r2 which is the turning radius of the center point FR is obtained by the square root of L ² + (r + W / 2) ² using r obtained by the above equation 5. ΘR is an angle formed by a straight line connecting the center point FR and the turning center point R and a straight line connecting the point PR1 and the turning center point R, and is obtained by IN / r2 [rad].

  On the other hand, as shown in FIG. 11, when the turning angle of the handle of the operation unit 52a is 0 °, that is, when the moving body model M goes straight, the center point FL and the center point FR move on a straight line parallel to the x axis. Will do. In this case, the coordinate values of the points PL1 to PL8 are obtained by Equation 8.

Note that the predetermined interval IN when going straight is the length of a straight line connecting the points PLn and PLn + 1.
Similarly, the coordinate values of the points PR1 to PR8 in this case are obtained by Equation 9.

  Thus, when the moving body model M turns right and left, the coordinate values of the points PL1 to PL8 and the points PR1 to PR8 are obtained by the equations 6 and 7. When the moving body model M goes straight, the coordinate values of the points PL1 to PL8 and the points PR1 to PR8 are obtained by the equations 8 and 9.

  Note that, similarly to the first auxiliary model M1 described above, the z-axis coordinate values of the points PL1 to PL8 and the points PR1 to PR8 in the coordinate system C3 are stored in the HDD 50 in advance. In this embodiment, the z-axis coordinate values of the points PL1 to PL8 and the points PR1 to PR8 are set to the same value as the z-axis coordinate value of the point EL (ER) described above. Therefore, in this embodiment, the second auxiliary model M2 is expressed as a point group arranged forward from below the moving body model M.

  The second auxiliary model M2 is generated in the global coordinate system GC by performing coordinate conversion of the points PL1 to PL8 and the points PR1 to PR8 obtained in this manner using the equation 4 in the same manner as described above.

  Returning to FIG. 3, after step S <b> 11, the angle of view from the virtual viewpoint V is calculated by the CPU 44 so that the size of the moving body model M is substantially constant in the display image provided to the operator ( Step S13). Here, it is assumed that a display image of a predetermined size (for example, vertical 480 pixels × horizontal 800 pixels) is displayed on the screen of the monitor 36 (for example, vertical 768 pixels × horizontal 1024 pixels). Information regarding the size of the display image (for example, vertical 480 pixels × horizontal 800 pixels) and information regarding the aspect ratio of the display image (for example, vertical: horizontal = 3: 5) are stored in the HDD 50 in advance.

FIG. 12 is an illustrative view showing a state in which the moving object model M generated in the global coordinate system GC is viewed from the lateral direction.
In FIG. 12, d is the distance from the virtual viewpoint V to the reference point B of the moving body 12 (current position of the moving body 12), β is the vertical angle of view from the virtual viewpoint V, and h is the height of the moving body model M. , H is the cutout width at the position of the distance d (vertical plane passing through the reference point B) when cut out from the virtual viewpoint V at the vertical angle of view β. The distance d can be obtained from the position of the virtual viewpoint V and the position of the reference point B.

A method for calculating the vertical angle of view β will be described with reference to FIG.
When the moving body model M is displayed on the monitor 36, in order to make the vertical size of the moving body model M a constant value k (0 <k <1) times the vertical size of the display image, k = h / It is sufficient if the relationship of H is established. For example, k = h / H = 0.33.
From FIG. 12, d * tan (β / 2) = H / 2 is established.
From these two mathematical expressions, the vertical angle of view β is obtained by Equation 10.

  FIG. 12B shows a state where the moving body 12 has moved from the state of FIG. 12A and the distance d from the virtual viewpoint V to the reference point B of the moving body 12 has become longer. Although H, h, and k are fixed values, since d increases, the vertical angle of view β in FIG. 12B is smaller than that in FIG.

  By setting the vertical angle of view β in this way, the vertical size of the moving object model M in the display image can be kept substantially constant. Unlike the case where the vertical angle of view β is fixed, the moving object model M Can be prevented from becoming smaller in the display image and thus in the screen of the monitor 36.

When the line connecting the virtual viewpoint V and the reference point B shown in FIG. 12B is on the xy plane of the three-dimensional orthogonal coordinate system C2, the horizontal angle of view γ from the virtual viewpoint V is shown in FIG. It becomes like this.
FIG. 13 shows the three-dimensional environment image K generated in the global coordinate system GC, the moving body model M, the first auxiliary model M1, and the second auxiliary model M2 on the plus side in the z-axis direction of the three-dimensional orthogonal coordinate system C2 ( It is an illustration figure which shows the state seen from FIG. In FIG. 13, the three-dimensional environment image K is a hemispherical wide-angle image with an angle of view of 180 ° taken at the virtual viewpoint V.

  Similar to the vertical angle of view β, the horizontal angle of view γ can be calculated by setting the magnification of the horizontal size of the moving object model M with respect to the horizontal size of the display image. The horizontal angle of view γ can also be calculated based on the calculated vertical angle of view β and a preset aspect ratio of the display image (for example, vertical: horizontal = 3: 5).

  Note that the cut-out range of the projected image is set to one of a cut-out vertical width based on the vertical angle of view β and a cut-out horizontal width based on the horizontal view angle γ, and an aspect ratio of the display image (for example, vertical: horizontal = 3: 5). Can be determined based on. In this embodiment, the cutout range of the projection image is determined based on the vertical angle of view β and the aspect ratio of the display image. Therefore, in step S13, the process for obtaining the horizontal angle of view γ is not performed.

Returning to FIG. 3, a synthesized image is then generated by the CPU 44 (step S <b> 15).
A generally used three-dimensional graphic library can be used for generating a composite image. In this embodiment, OpenGL (registered trademark) and the gluLookAt function and gluPerspective function provided free of charge for use in Microsoft Windows (registered trademark) were used. As the argument to the gluLookAt function, the position of the virtual viewpoint V and the position of the cut-out center point Q in the global coordinate system GC are given, and as the argument to the gluPerspective function, the vertical angle of view β and the aspect ratio of the display image (for example, vertical: horizontal) = 3: 5). As a result, it is possible to draw a composite image obtained by perspective projection including the three-dimensional environment image K generated in the global coordinate system GC, the moving body model M, the first auxiliary model M1, and the second auxiliary model M2.

  The composite image obtained in this way is obtained by perspectively projecting the three-dimensional environment image K, the moving body model M, the first auxiliary model M1, and the second auxiliary model M2 from the virtual viewpoint V toward the center point Q. The obtained projection image is cut out according to the cut-out range based on the obtained vertical angle of view β and the aspect ratio of the display image.

  Thereafter, the CPU 44 converts the size of the synthesized image so that the synthesized image can be displayed on the monitor 36 at a predetermined size (for example, vertical 480 pixels × horizontal 800 pixels), and a display image is obtained (step S17). Then, the obtained display image (composite image whose size has been converted) is displayed on the monitor 36 and presented to the operator (step S19).

  As shown in FIG. 13, when the moving body model M moves forward with the virtual viewpoint V behind, the reference point B is located on the line connecting the virtual viewpoint V and the cut-out center point Q. Therefore, in this case, a display image with the reference point B as the center position is presented.

  However, when the mobile body model M is bent, the image on the traveling direction side of the mobile body model M is presented to the operator more widely from the positional relationship between the cut-out center point Q and the mobile body model M. For example, when the moving body model M turns to the left, the cutout range is determined with the line connecting the virtual viewpoint V and the cutout center point Q as the centerline as shown in FIG.

  Therefore, as the orientation of the moving body model M with respect to the virtual viewpoint V changes, the moving direction model image of the moving body model M is gradually presented along with it, and the operator feels comfortable. In addition, the operator can easily recognize the environment in the traveling direction of the moving body 12 and can remotely operate the moving body 12 more easily.

  Such processing of steps S1 to S19 is repeated at intervals of about 0.1 seconds.

  According to the remote operation device 14 and the remote operation system 10, it is possible to generate the first auxiliary model M1 and the second auxiliary model M2 according to the current turning angle of the steering wheel of the operation unit 52a (input amount from the steering unit 52). Then, the operator confirms the positional relationship between the first auxiliary model M1 and the second auxiliary model M2 and the shoulder of the road in the display image (the composite image whose size has been converted), thereby moving the moving body. It is possible to easily grasp the turning angle of the handle necessary for moving the model M and the moving body 12 in an appropriate direction. Therefore, even an inexperienced operator can easily remotely control the moving body 12 and can quickly move the moving body 12 to the destination.

  The operator can easily grasp the appropriate turning angle of the handle by confirming how the road shoulder and the first auxiliary model M1 intersect in the display image. The first auxiliary model M1 is particularly effective when the road is bent sharply (when the road curvature is large) as shown in FIG. That is, it is particularly effective when the traveling direction of the moving body 12 is changed by 90 ° or more to the right or left. In this case, by adjusting the turning angle of the handle so that the first auxiliary model M1 is substantially orthogonal to the road shoulder, even an unfamiliar operator can easily and stably operate the moving body 12 remotely. .

  Further, the operator can easily grasp the appropriate turning angle of the handle by confirming the gap between the road shoulder and the second auxiliary model M2 in the display image. The second auxiliary model M2 is particularly effective when the road is gently bent as shown in FIG. 16 (when the road curvature is small). In this case, even the inexperienced operator can move the moving body 12 by adjusting the turning angle of the handle so that the distance between the road shoulder (the side line on the road surface in FIG. 16) and the second auxiliary model M2 is constant. Remote operation can be easily and stably performed.

  In the above-described embodiment, the past information is selected based on the distance, but the present invention is not limited to this. The past information may be selected in the same manner as in the case of the distance based on the time. In other words, the imaging time included in the past information may be compared with the time 5 seconds before the current time, and the past information captured at the imaging time closest to the time 5 seconds before may be employed. In addition, the operator may arbitrarily select from past information transmitted from the communication device 32 to the remote operation device 14.

  Information regarding the moving body 12 and information regarding the camera 20 may be detected by separate detection means. The position, orientation, and orientation of the moving body 12 and / or the camera 20 may be obtained by an existing three-dimensional self-position estimation method.

  In the above-described embodiment, the case where the point EL at the time of turning left is set as the turning center point R and the point ER at the time of turning right is set as the turning center point R has been described, but the present invention is not limited to this. The coordinate values of the point EL and the point ER can be arbitrarily set as long as the straight line connecting the point EL and the point ER overlaps the straight line connecting the turning center point R and the intermediate point FM. That is, the length of the first auxiliary model M1 can be set arbitrarily.

  In the above-described embodiment, the case where the point group GL and the point group GR of the second auxiliary model M2 are each configured by eight points has been described, but the number of points that configure the point group GL and the point group GR is as follows. Can be set arbitrarily.

  The z-axis coordinate values of the point EL, the point ER, the points PL1 to PL8, and the points PR1 to PR8 in the coordinate system C3 are not limited to the above-described embodiment, and can be arbitrarily set. For example, these z-axis coordinate values may be set to the same value as the z-axis coordinate values of the center point FL and the center point FR. That is, the heights of the first auxiliary model M1 and the second auxiliary model M2 may be set to the same height as the center of the front wheel of the moving body model M.

  In the above-described embodiment, the case where the first auxiliary model M1 and the second auxiliary model M2 are generated in the global coordinate system CG has been described. However, only one of them may be generated. Based on the input amount from the steering unit 52, the auxiliary model to be generated may be switched such that the first auxiliary model M1 is generated when going straight and the second auxiliary model M2 is generated when turning right or left.

  In the above-described embodiment, the case where a camera having an angle of view of 180 ° is used as the imaging unit has been described, but the imaging unit is not limited to this. The larger the angle of view of the image pickup means, the better. For example, if a camera with an angle of view of about 270 ° is used as the image pickup means, it is possible to prevent the image edge of the environmental image from appearing in the display image when turning right or left.

  The moving body in the present invention is not limited to the above-described four-wheel buggy vehicle. Other examples of the moving body include construction machines such as excavators and ships.

It is a block diagram which shows the remote control system of one Embodiment of this invention. It is a left view which shows an example of a mobile body. It is a flowchart which shows an example of the image processing operation in the remote control apparatus of the remote control system of FIG. It is an illustration figure for demonstrating the selection method of past information. It is an illustration figure which shows the two-dimensional orthogonal coordinate system by which an environmental image is represented. It is an illustration figure which shows the three-dimensional orthogonal coordinate system in which a three-dimensional environmental image is displayed. It is an illustration figure which shows the global coordinate system as virtual space. It is an illustration figure for demonstrating the production | generation method of a mobile body model. It is an illustration figure for demonstrating the production | generation method of a 1st auxiliary model. It is an illustration figure for demonstrating the production | generation method of the 2nd auxiliary | assistant model at the time of a left turn. It is an illustration figure for demonstrating the production | generation method of the 2nd auxiliary model at the time of straight ahead. It is an illustration figure which shows the vertical angle of view in embodiment of this invention. It is an illustration figure which shows the state which looked at the three-dimensional environment image produced | generated to the global coordinate system, the mobile body model, the 1st auxiliary model, and the 2nd auxiliary model from the top. It is an illustration figure which shows the case where a mobile body model turns left in embodiment of this invention. It is an illustration figure which shows an example of the synthesized image displayed in embodiment of this invention. It is an illustration figure which shows the other example of the synthesized image displayed in embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Remote operation system 12 Mobile body 14 Remote operation device 16 Microcomputer 18 Speed detector 20 Camera 20a Lens 22 Positioning unit 24 Attitude sensor 26 Direction sensor 28, 50 Hard disk drive 32, 40 Communication equipment 34 Control unit 36 Monitor 38 Input equipment 44 CPU
46 ROM
48 RAM
52 Steering Unit 52a Operation Unit 52b Steering Angle Sensor C1 2D Cartesian Coordinate System C2, C3 3D Cartesian Coordinate System GC Global Coordinate System I Environment Image K 3D Environment Image M Mobile Model M1 First Auxiliary Model M2 Second Auxiliary Model V Virtual viewpoint

Claims (3)

A remote operation system comprising a mobile body having a pair of left and right front wheels and a remote control device for remotely operating the mobile body,
The moving body is
Imaging means for capturing an environmental image around the moving body;
Comprising information detecting means for detecting information relating to the moving body and the imaging means,
The remote control device is:
An input means for an operator to steer the moving body;
Determining means for determining a virtual viewpoint based on a detection result of the information detecting means;
A three-dimensional environment image representing the environment image in three dimensions and the virtual viewpoint based on the environment image picked up from the virtual viewpoint by the image pickup means and the detection result of the information detection means at the time of picking up the environment image First generating means for generating a virtual space,
Second generation means for generating the moving body model in the virtual space based on a detection result of the information detection means after imaging the environmental image from the virtual viewpoint and data relating to the moving body model representing the moving body. When,
Based on the detection result of the information detection unit and the input amount from the input unit after capturing the environmental image from the virtual viewpoint, the input unit from the input unit during forward movement, left turn, and right turn Third generation means for generating in the virtual space an auxiliary model serving as an input quantity index;
Fourth generation means for generating a composite image by perspective-projecting the three-dimensional environment image viewed from the virtual viewpoint, the moving body model, and the auxiliary model;
E Bei and display means for displaying the composite image,
The auxiliary model is a first auxiliary model that is orthogonal to the traveling direction of the moving body model and extends laterally with respect to the moving body model while passing through an intermediate point between the left front wheel and the right front wheel of the moving body model. Including remote control system.   The remote operation system according to claim 1, wherein the auxiliary model includes a second auxiliary model that represents a movement path of the moving body model. A remote control device for remotely operating the moving body including an imaging unit that captures an environmental image around a moving body having a pair of left and right front wheels, and an information detection unit that detects information about the moving body and the imaging unit. There,
An input means for an operator to steer the moving body;
Determining means for determining a virtual viewpoint based on a detection result of the information detecting means;
A three-dimensional environment image representing the environment image in three dimensions and the virtual viewpoint based on the environment image picked up from the virtual viewpoint by the image pickup means and the detection result of the information detection means at the time of picking up the environment image First generating means for generating a virtual space,
Second generation means for generating the moving body model in the virtual space based on a detection result of the information detection means after imaging the environmental image from the virtual viewpoint and data relating to the moving body model representing the moving body. When,
Based on the detection result of the information detection unit and the input amount from the input unit after capturing the environmental image from the virtual viewpoint, the input unit from the input unit during forward movement, left turn, and right turn Third generation means for generating in the virtual space an auxiliary model serving as an input quantity index;
Fourth generation means for generating a composite image by perspective-projecting the three-dimensional environment image viewed from the virtual viewpoint, the moving body model, and the auxiliary model;
E Bei and display means for displaying the composite image,
The auxiliary model is a first auxiliary model that is orthogonal to the traveling direction of the moving body model and extends laterally with respect to the moving body model while passing through an intermediate point between the left front wheel and the right front wheel of the moving body model. Including remote control devices.