JP2022146888A - Display system, communication system, display control method, and program - Google Patents

Abstract

To allow users to appropriately grasp the movement state of a mobile body.SOLUTION: A display system for displaying an image captured by a mobile body 10 (10A, 10B, or 10C) moving within a predetermined base receives a captured image of the base transmitted from the mobile body 10 (10A, 10B, or 10C), and superimposes and displays a virtual route image 711, 713, or 715 on a movement route of the mobile body 10 (10A, 10B, or 10C) in the base shown in the received captured image.SELECTED DRAWING: Figure 13

Description

The present disclosure relates to display systems, communication systems, display control methods, and programs.

2. Description of the Related Art A robot that is installed in a base such as a factory or a warehouse and can move autonomously within the base is known. Such robots are used, for example, as inspection robots or service robots, and can perform tasks such as inspection of facilities in a base instead of workers.

A system is also known in which a user at a remote location can manually operate a robot that can autonomously move within a site according to the state of the robot, the situation of the site, the intended use, or the like. For example, in Patent Document 1, an unmanned vehicle automatically switches between autonomous driving and remote control based on the mixing ratio of the driving environment based on the distance measurement data and the communication environment of the remote control device, and presents the result to the user. Content is disclosed. Further, Patent Document 2 discloses the use of a user interface to manually drive a robot to a desired destination or autonomously navigate the robot.

JP 2011-150516 A Japanese translation of PCT publication No. 2014-503376

However, the conventional method has a problem that it is difficult for the user to appropriately grasp the movement state of the moving body such as a robot when the user wants to switch between autonomous movement and manual operation of the moving body.

In order to solve the above-described problems, the invention according to claim 1 is a display system for displaying an image captured by a mobile object moving within a predetermined base, wherein the image transmitted from the mobile object is receiving means for receiving a photographed image; and display control means for superimposing and displaying a virtual route image on the movement route of the mobile object at the base shown in the received photographed image. display system.

ADVANTAGE OF THE INVENTION According to this invention, it is effective in the ability to let a user grasp|ascertain the movement state of a mobile body appropriately.

It is a figure which shows an example of the whole structure of a communication system. It is a figure which shows an example of a schematic of a structure of a moving body. It is a figure which shows an example of the hardware constitutions of a mobile body. It is a figure which shows an example of the hardware constitutions of a display apparatus. It is a figure which shows an example of the functional structure of a communication system. 4 is a conceptual diagram showing an example of a map information management table; FIG. 4 is a conceptual diagram showing an example of a destination sequence management table; FIG. 4 is a conceptual diagram showing an example of a route information management table; FIG. FIG. 10 is a sequence diagram showing an example of movement control processing of a moving body; FIG. 10 is a sequence diagram showing an example of processing up to the start of movement of a moving body; It is a figure which shows an example of a route input screen. It is a figure which shows an example of a route input screen. FIG. 10 is a sequence diagram showing an example of switching processing between autonomous movement of a moving object and manual operation using an operation screen; It is a figure which shows an example of an operation screen. It is a figure which shows an example of an operation screen. It is a figure which shows an example of an operation screen. It is a figure which shows an example of an operation screen. 4 is a flowchart showing an example of switching processing between an autonomous movement mode and a manual operation mode in a mobile body; It is a flowchart which shows an example of the autonomous movement process of a mobile body. FIG. 10 is a sequence diagram showing an example of manual operation processing of a moving body; It is a figure which shows an example of an operation command input screen. It is a figure which shows the modification 1 of an operation screen. It is a figure which shows the modification 1 of an operation screen. It is a figure which shows the modification 2 of an operation screen. It is a figure which shows the modification 3 of an operation screen. It is a figure which shows the modification 4 of an operation screen. FIG. 12 is a diagram showing a modification 5 of the operation screen; FIG. 12 is a diagram showing a modification 6 of the operation screen; It is a figure showing an example of functional composition of a communications system concerning modification 1 of an embodiment. FIG. 11 is a sequence diagram showing an example of switching processing between autonomous movement and manual operation of a moving body using an operation screen according to Modification 1 of the embodiment; It is a figure which shows an example of the whole structure of the communication system which concerns on the modification 2 of embodiment. It is a figure which shows an example of a functional structure of the communication system which concerns on the modified example 2 of embodiment. FIG. 11 is a sequence diagram showing an example of processing up to the start of movement of a moving object according to Modification 2 of the embodiment; FIG. 11 is a sequence diagram showing an example of switching processing between autonomous movement and manual operation of a moving object using an operation screen according to Modification 2 of the embodiment; It is a figure which shows an example of the functional structure of a communication system.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

●Embodiment●
●System Configuration FIG. 1 is a diagram showing an example of the overall configuration of a communication system. A communication system 1 shown in FIG. 1 is a system that allows a user to remotely control a mobile object 10 that moves within a predetermined location.

A communication system 1 includes a mobile unit 10 and a display device 50 located at a predetermined base. The mobile object 10 and the display device 50 that configure the communication system 1 can communicate via the communication network 100 . A communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), or the like. The communication network 100 includes not only wired communication but also 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave). Access) or LTE (Long Term Evolution) or other wireless communication network may be included.

The moving body 10 is a robot that is installed at the target site and can move autonomously within the target site. This autonomous movement includes an action of performing imitation learning (machine learning) on the route traveled in the past within the target site and autonomously moving within the target site using the result of the imitation learning. Also, the autonomous movement may be an action of autonomously moving within the target site according to a preset movement route or an action of autonomously moving within the target site using a technique such as line tracing. Furthermore, the mobile object 10 can also be moved manually by a remote user. That is, the moving body 10 can move within the target site while switching between autonomous movement and manual operation by the user. In addition, the mobile body 10 can perform predetermined tasks such as inspection, maintenance, transportation, or light work while moving within the target site. Here, the moving body 10 means a robot in a broad sense, and may be a robot capable of both autonomous movement and movement by remote control by a user. The mobile body 10 also includes an automobile that can travel by switching between automatic operation and manual operation by remote operation. Furthermore, the mobile object 10 also includes flying objects such as drones, multicopters, and unmanned flying objects.

In addition, the target site where the mobile object 10 is installed is, for example, an office, a factory, a construction site, a substation, a farm, a field, a plantation, a cultivated land, an outdoor site such as a disaster site, or an office, a school, a factory, a warehouse, or the like. , commercial facilities, hospitals or nursing homes. That is, the target site may be any site where there is a need for the mobile body 10 to perform tasks that have conventionally been performed manually.

The display device 50 is a computer such as a notebook PC (Personal Computer) located at a management site different from the target site and used by an operator (user) who performs a predetermined operation on the mobile body 10 . The operator uses the operation screen displayed on the display device 50 at a management site such as an office to move the mobile object 10 or perform an operation for executing a predetermined task. For example, the operator remotely operates the moving body 10 while viewing the image of the target site displayed on the display device 50 .

Note that FIG. 1 shows an example in which one mobile body 10 and a display device 50 are connected via a communication network 100, but the display device 50 is connected to a plurality of mobile bodies 10 located at one target site. , or may be configured to be connectable to a moving body 10 located at a different target site. FIG. 1 shows an example in which the display device 50 is located at a remote management site different from the target site where the mobile unit 10 is installed. It may be a configuration located in. Furthermore, the display device 50 is not limited to a notebook PC, and may be, for example, a desktop PC, a tablet terminal, a smart phone, a wearable terminal, or the like.

Conventionally, for example, when a mobile body becomes unable to travel due to an obstacle or the like while it is moving autonomously, an operator manually performs a recovery operation to return the mobile body to autonomous movement. However, it has been difficult for the operator to make an accurate switching judgment for returning from the manual operation to the autonomous movement only with the information presented to the conventional operator. In addition, when performing autonomous movement by learning during manual operation, if the learning results so far cannot be used appropriately due to changes in the environment such as weather or buildings in the base, manual operation for re-learning It was difficult for the operator to make a decision to switch to . In other words, when the operator wants to switch between autonomous movement and manual operation of the mobile body, it is difficult for the operator to appropriately grasp the movement state of the mobile body with the conventional method. In addition, along with this, it is difficult for the operator to appropriately determine whether to switch between autonomous movement and manual operation.

Therefore, the communication system 1 displays a virtual By superimposing and displaying the route image, the operator can appropriately grasp the movement state of the moving body 10 . In addition, since the communication system 1 can switch between the autonomous movement and the manual operation of the moving body 10 using the operation screen displayed on the display device 50, it is possible to easily operate whether or not to switch between the autonomous movement and the manual operation. In addition, it is possible to improve the operability of the user when switching between autonomous movement and manual operation. Furthermore, the communication system 1 can allow the operator to appropriately determine the necessity of learning by manual operation even for the mobile object 10 that learns the movement route of autonomous movement using manual operation. .

●Configuration of Moving Body Next, a specific configuration of the moving body 10 will be described with reference to FIG. FIG. 2 is a diagram showing an example of a schematic configuration of a moving body. Note that components may be added or deleted from the configuration of the moving body 10 shown in FIG. 2 as necessary.

The mobile body 10 shown in FIG. and a movable arm 16 for causing the moving body 10 to perform a predetermined work (operation). Among them, the housing 11 is located in the body portion of the moving body 10 and incorporates the control device 30 and the like for controlling the processing or operation of the moving body 10 .

The photographing device 12 obtains a photographed image by photographing a subject such as a person, an object, or a landscape located at a base where the mobile body 10 is installed. The photographing device 12 is a digital camera (general photographing device) such as a digital single-lens reflex camera or a compact digital camera capable of acquiring a planar image (detailed image). Note that the captured image acquired by the imaging device 12 may be a moving image, a still image, or both a moving image and a still image. Also, the captured image acquired by the imaging device 12 may include audio data together with the image data. Furthermore, the imaging device 12 may be a wide-angle imaging device capable of acquiring an omnidirectional (360°) panoramic image. A wide-angle photographing device is, for example, a omnidirectional photographing device for photographing a subject and obtaining two hemispherical images that are the basis of a omnidirectional (panoramic) image. Furthermore, the wide-angle imaging device may be, for example, a wide-angle camera or a stereo camera capable of acquiring a wide-angle image having an angle of view equal to or larger than a predetermined value. That is, the wide-angle photographing device is photographing means capable of acquiring an image (omnidirectional image, wide-angle image) photographed using a lens having a focal length shorter than a predetermined value. Further, the moving body 10 may be configured to include a plurality of imaging devices 12 . In this case, the moving object 10 includes both a wide-angle imaging device and a general imaging device capable of acquiring a detailed image (planar image) by photographing a part of the subject photographed by the wide-angle imaging device as the imaging device 12. It may be a configuration.

The support member 13 is a member for installing (fixing) the photographing device 12 on the moving body 10 (housing 11). The support member 13 may be a pole or the like fixed to the housing 11 or a pedestal fixed to the housing 11 . Further, the support member 13 may be a movable member capable of adjusting the photographing direction (orientation) and position (height) of the photographing device 12 .

The moving mechanism 15 is a unit that moves the moving body 10, and includes wheels, a traveling motor, a traveling encoder, a steering motor, a steering encoder, and the like. Regarding movement control of the moving body 10, since it is an existing technology, a detailed description will be omitted. , moves the moving body 10 based on the received travel instruction. The moving mechanism 15 may be a bipedal foot type or a single-wheel type. Further, the shape of the mobile body 10 is not limited to the vehicle type shown in FIG. good too.

The movable arm 16 has a motion means that enables additional motions other than the movement of the moving body 10 . As shown in FIG. 2, the movable arm 16 has, for example, a hand at the tip of the movable arm 16 for gripping an object such as a part as an operating means. By rotating or deforming the movable arm 16, the moving body 10 can perform a predetermined work (movement). In addition to the configuration described above, the moving body 10 may have various sensors capable of detecting information around the moving body 10 . Various sensors are sensor devices, such as a barometer, a thermometer, a photometer, a human sensor, a gas sensor, an odor sensor, or an illuminance meter, for example.

●Hardware Configuration Next, the hardware configuration of the devices or terminals constituting the communication system according to the embodiment will be described with reference to FIGS. 3 and 4. FIG. Components may be added or deleted from the hardware configuration of the device or terminal shown in FIGS. 3 and 4 as necessary.

○Hardware configuration of moving body○
FIG. 3 is a diagram illustrating an example of a hardware configuration of a mobile object; The mobile 10 includes a controller 30 that controls the processing or operation of the mobile 10 . The control device 30 is provided inside the housing 11 of the moving body 10 as described above. Note that the control device 30 may be provided outside the housing 11 of the mobile body 10 or may be provided as a device separate from the mobile body 10 .

The control device 30 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HDD (Hard Disk Drive) 304, a media I/F (Interface) 305, an input/output I/ F 306 , sound input/output I/F 307 , network I/F 308 , short-range communication circuit 309 , antenna 309 a of the short-range communication circuit 309 , external device connection I/F 311 , and bus line 310 .

The CPU 301 controls the moving body 10 as a whole. The CPU 301 is an arithmetic unit that implements each function of the moving body 10 by reading programs or data stored in a ROM 302 or HD (Hard Disk) 304a or the like onto a RAM 303 and executing processing.

The ROM 302 is non-volatile memory that can retain programs or data even when the power is turned off. A RAM 303 is a volatile memory used as a work area or the like for the CPU 301 . The HDD 304 controls reading or writing of various data to the HD 304 a under the control of the CPU 301 . The HD 304a stores various data such as programs. A media I/F 305 controls reading or writing (storage) of data to a recording medium 305a such as a USB (Universal Serial Bus) memory, memory card, optical disk, or flash memory.

The input/output I/F 306 is an interface for inputting/outputting characters, numerical values, various instructions, etc., to/from various external devices. The input/output I/F 306 controls display of various information such as cursors, menus, windows, characters, and images on a display 14 such as an LCD (Liquid Crystal Display). Note that the display 14 may be a touch panel display provided with input means. In addition to the display 14, the input/output I/F 306 may be connected to input means such as a pointing device such as a mouse and a keyboard. Sound input/output I/F 307 is a circuit for processing input/output of sound signals between microphone 307a and speaker 307b under the control of CPU 301 . The microphone 307a is a type of built-in sound collecting means for inputting sound signals under the control of the CPU 301 . The speaker 307b is a kind of reproduction means for outputting sound signals under the control of the CPU 301. FIG.

A network I/F 308 is a communication interface that communicates (connects) with other devices or devices via the communication network 100 . A network I/F 308 is, for example, a communication interface such as a wired or wireless LAN. The short-range communication circuit 309 is a communication circuit such as NFC (Near Field communication) or Bluetooth (registered trademark). The external device connection I/F 311 is an interface for connecting other devices to the control device 30 .

A bus line 310 is an address bus, a data bus, or the like for electrically connecting the components described above, and transmits an address signal, a data signal, various control signals, and the like. CPU 301 , ROM 302 , RAM 303 , HDD 304 , media I/F 305 , input/output I/F 306 , sound input/output I/F 307 , network I/F 308 , short-range communication circuit 309 , and external device connection I/F 311 connect bus line 310 . are connected to each other through

Furthermore, the control device 30 includes a drive motor 101, an actuator 102, an acceleration/direction sensor 103, a GPS (Global Positioning System) sensor 104, an imaging device 12, a battery 120, and an obstacle detection device via an external device connection I/F 311. A sensor 106 is connected.

The driving motor 101 rotates the moving mechanism 15 based on the command from the CPU 301 to move the moving body 10 along the ground. Actuator 102 deforms movable arm 16 based on a command from CPU 301 . The acceleration/azimuth sensor 103 is a sensor such as an electronic magnetic compass, a gyrocompass, an acceleration sensor, or the like that detects geomagnetism. GPS sensor 104 receives GPS signals from GPS satellites. The battery 120 is a unit that supplies necessary power to the entire mobile body 10 . In addition, the battery 120 may include an external battery that plays a role of supplementary power supply from the outside, in addition to the battery that is built in the main body of the mobile body 10 . The obstacle detection sensor 106 is a detection sensor that detects surrounding obstacles when the moving body 10 moves. The obstacle detection sensor 106 is, for example, a stereo camera, an image sensor such as a camera equipped with an area sensor in which photoelectric conversion elements are arranged in a plane, a TOF (Time Of Flight) sensor, or a LIDAR (Light Detection and Ranging) sensor. sensors, radar sensors, laser range finders, ultrasonic sensors, depth cameras or ranging sensors such as depth sensors.

○Hardware configuration of the display device○
FIG. 4 is a diagram illustrating an example of a hardware configuration of a display device; Each hardware configuration of the display device 50 is indicated by a reference numeral in the 500s. The display device 50 is constructed by a computer, and as shown in FIG. Line 510 , keyboard 511 , pointing device 512 , sound input/output I/F 513 , microphone 514 , speaker 515 , camera 516 , DVD-RW (Digital Versatile Disk Rewritable) drive 517 , and media I/F 519 are provided.

Among these, the CPU 501 controls the operation of the entire display device 50 . The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). A RAM 503 is used as a work area for the CPU 501 . The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data to/from the HD 504 under the control of the CPU 501 . A display 506 displays various information such as cursors, menus, windows, characters, and images. Note that the display 506 may be a touch panel display provided with input means. Also, the display 506 is an example of a display unit. A display unit as the display 506 may be an external device connected to the display device 50 and having a display function. The display unit in this case projects an image from an external display such as an IWB (Interactive White Board), or a PJ (Projector) or HUD (Head-Up Display) connected as an external device. (For example, the ceiling or wall of a management base, the windshield of an automobile, etc.). The external device connection I/F 507 is an interface for connecting various external devices. A network I/F 508 is an interface for data communication using the communication network 100 . A bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

Also, the keyboard 511 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. A pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, or the like. The input means may be not only the keyboard 511 and pointing device 512 but also a touch panel, voice input device, or the like. Input means such as the keyboard 511 and the pointing device 512 may be an external UI (User Interface) of the display device 50 . Sound input/output I/F 513 is a circuit that processes input/output of sound signals with microphone 514 and speaker 515 under the control of CPU 501 . The microphone 514 is a type of built-in sound collecting means for inputting sound. The speaker 515 is a type of built-in output means for outputting audio signals. The camera 516 is a type of built-in image capturing means that captures an image of a subject and obtains image data. Note that the microphone 514, the speaker 515, and the camera 516 may be external devices instead of built-in types of the display device 50. FIG. A DVD-RW drive 517 controls reading or writing of various data to a DVD-RW 518 as an example of a removable recording medium. Note that the detachable recording medium is not limited to the DVD-RW, and may be a DVD-R, a Blu-ray (registered trademark) Disc (Blu-ray Disc), or the like. A media I/F 519 controls reading or writing (storage) of data to a recording medium 521 such as a flash memory.

Each of the above programs may be recorded in a computer-readable recording medium in an installable or executable format and distributed. Examples of recording media include CD-Rs (Compact Disc Recordable), DVDs (Digital Versatile Disks), Blu-ray Discs, SD cards, USB memories, and the like. Also, the recording medium can be provided domestically or internationally as a program product. For example, the display device 50 implements the display control method according to the present invention by executing the program according to the present invention.

●Functional Configuration Next, the functional configuration of the communication system according to the embodiment will be described with reference to FIGS. 5 to 8. FIG. FIG. 5 is a diagram illustrating an example of a functional configuration of a communication system; It should be noted that FIG. 5 shows the devices or terminals shown in FIG. 1 that are related to processing or operations described later.

○ Functional configuration of moving body (control device) ○
First, with reference to FIG. 5, the functional configuration of the control device 30 that controls the processing or operation of the moving body 10 will be described. The control device 30 includes a transmission/reception unit 31, a determination unit 32, an imaging control unit 33, a state detection unit 34, a map information management unit 35, a destination series management unit 36, a self-position estimation unit 37, a route information generation unit 38, route information Management unit 39, destination setting unit 40, movement control unit 41, mode setting unit 42, autonomous movement processing unit 43, manual operation processing unit 44, accuracy calculation unit 45, image generation unit 46, learning unit 47, and storage/reading unit 49. Each of these units is a function realized by any of the components shown in FIG. is. The control device 30 also has a storage unit 3000 constructed by the ROM 302, HD 304a, or recording medium 305a shown in FIG.

Transmitting/receiving unit 31 is mainly implemented by processing of CPU 301 for network I/F 308 , and transmits/receives various data or information to/from other devices or terminals via communication network 100 .

The determination unit 32 is realized by the processing of the CPU 301 and makes various determinations. The imaging control unit 33 is mainly realized by the processing of the CPU 301 for the external device connection I/F 311 and controls imaging processing for the imaging device 12 . The imaging control unit 33 instructs the imaging device 12 to perform imaging processing, for example. Further, the imaging control unit 33 acquires, for example, a captured image obtained by imaging processing by the imaging device 12 .

The state detection unit 34 is mainly realized by the processing of the CPU 301 for the external device connection I/F 311, and detects the state of the moving body 10 or the surroundings of the moving body 10 using various sensors. The state detection unit 34 uses, for example, the obstacle detection sensor 106 or the like to measure the distance to an object (obstacle) existing around the moving body 10 and outputs it as distance data.

Also, the state detection unit 34 detects the position of the moving body 10 using, for example, the GPS sensor 104 or the like. Specifically, the state detection unit 34 acquires the position on the environment map stored in the map information management DB 3001 using the GPS sensor 104 or the like. Further, the state detection unit 34 uses distance data measured using the obstacle detection sensor 106 or the like, applies SLAM (Simultaneous Localization and Mapping), and acquires a position by matching with an environmental map. may Here, SLAM is a technology capable of simultaneously estimating self-location and creating an environment map.

Furthermore, the state detection unit 34 detects the direction in which the moving body 10 is facing, for example, using the acceleration/direction sensor 103 or the like.

The map information management unit 35 is mainly realized by the processing of the CPU 301, and uses the map information management DB 3001 to manage map information indicating the environmental map of the target site where the moving body 10 is installed. The map information management unit 35 manages, for example, map information indicating an environment map downloaded from an external server or the like, or an environment map created by applying SLAM.

The destination series management unit 36 is mainly realized by the processing of the CPU 301, and manages the destination series on the moving route of the moving body 10 using the destination series management DB 3002. FIG. The destination series includes a final destination (goal) on the movement route of the mobile object 10 and a plurality of waypoints (subgoals) to the final destination. The destination series is, for example, data specified by position information indicating positions (coordinate values) on a map such as latitude and longitude. The destination series can be obtained, for example, by remotely operating the moving body 10 and specifying it. The designation method may be, for example, a GUI (Graphical User Interface) or the like on the environment map.

The self-position estimating unit 37 is mainly realized by the processing of the CPU 301, and calculates the current position of the moving object 10 based on the position information detected by the state detection unit 34 and the direction information indicating the direction in which the moving object 10 is facing. (Self-location) is estimated. The self-position estimation unit 37 uses, for example, a method such as an extended Kalman filter (EKF) to estimate the current position (self-position).

The route information generation unit 38 is mainly realized by the processing of the CPU 301 and generates route information indicating the moving route of the moving body 10 . The route information generating unit 38 uses the current position (self-position) of the mobile object 10 estimated by the self-position estimating unit 37 and the destination series managed by the destination series managing unit 36 to determine the final destination (goal ) and multiple waypoints (subgoals) to generate route information indicating the route from the current position to the final destination. The method of generating the route information includes, for example, connecting the waypoints from the current position to the final destination with a straight line, or using information about the obstacles obtained by the photographed image or the state detection unit 34 to avoid the obstacles. Use a method that minimizes travel time.

The route information management unit 39 is mainly realized by processing of the CPU 301 and manages the route information generated by the route information generation unit 38 using the route information management DB 3003 .

The destination setting unit 40 is mainly realized by the processing of the CPU 301 and sets the destination of the moving body 10 . The destination setting unit 40, for example, based on the current position (self-position) of the moving body 10 estimated by the self-position estimation unit 37, selects the current position from the destination series managed by the destination series management unit 36. Set a destination (current goal) or a waypoint (subgoal) as a destination. The method of setting the destination is, for example, setting the destination series closest to the current position (self-position) of the moving body 10 among the destination series that have not yet been reached (for example, the status is "unreached"). method, or a method of setting a destination series with the smallest data index among destination systems that have not yet been reached.

The movement control unit 41 is mainly realized by the processing of the CPU 301 for the external device connection I/F 311 and controls the movement of the moving body 10 by driving the movement mechanism 15 . The movement control unit 41 moves the moving body 10 according to a driving command from the autonomous movement processing unit 43 or the manual operation processing unit 44, for example.

The mode setting unit 42 is mainly realized by the processing of the CPU 301, and sets an operation mode indicating an operation for moving the moving object 10. Any manual operation mode for moving the body 10 by manual operation of the operator is set. The mode setting unit 42 switches between the autonomous movement mode and the manual operation mode, for example, in response to a switching request transmitted from the display device 50 .

The autonomous movement processing unit 43 is mainly realized by the processing of the CPU 301 and controls the autonomous movement processing of the moving body 10 . The autonomous movement processing unit 43 outputs, for example, a command to drive the moving body 10 to the movement control unit 41 so as to follow the movement route indicated by the route information generated by the route information generation unit 38 .

The manual operation processing unit 44 is mainly realized by the processing of the CPU 301 and controls manual operation processing of the moving body 10 . The manual operation processing unit 44 outputs a command to drive the moving body 10 to the movement control unit 41 according to the manual operation command transmitted from the display device 50, for example.

The accuracy calculation unit 45 is mainly realized by the processing of the CPU 301 and calculates the accuracy of autonomous movement of the moving body 10 . Here, the accuracy of autonomous movement of the moving body 10 is information indicating the degree of certainty (confidence) as to whether or not the moving body 10 can move autonomously. indicates that there is The accuracy of autonomous movement is calculated by, for example, decreasing the numerical value based on the likelihood value of the self-position estimated by the self-position estimating unit 37 when the likelihood is low, using the variance of various sensors, etc. Using the elapsed travel time which is the operating state by the autonomous travel processing unit 43, the longer the elapsed travel time in the autonomous travel mode, the lower the numeric value. It is possible to use a method such as lowering the numerical value as it becomes larger, or lowering the numerical value when there are many obstacles according to the obstacle information detected by the state detection unit 34 .

The image generator 46 is mainly implemented by the processing of the CPU 301 and generates a display image to be displayed on the display device 50 . The image generating unit 46 generates, for example, a route image showing the destination sequence managed by the destination sequence managing unit 36 on the captured image captured by the capturing control unit 33 . The image generation unit 46 draws the generated route image on the movement route of the moving body 10 for the photographed image data acquired by the photographing control unit 33 . The method of drawing the route image on the photographed image data is based on, for example, the self-position (current position) of the mobile body 10 estimated by the self-position estimation unit 37, the installation position of the photographing device 12, and the angle of view of the photographed image data. , a method of drawing a path image by perspective projection transformation, and the like. Note that the photographed image data may include PTZ (Pan-Tilt-Zoom) parameters for specifying the photographing direction of the photographing device 12 and the like. The captured image data including the PTZ parameters is stored (saved) in the storage unit 3000 of the moving body 10 or the like. In addition, the PTZ parameters are stored in the storage unit 3000 in association with the position information of the destination candidates, that is, the final destination (goal) forming the destination series and a plurality of waypoints (subgoals) leading to the final destination. may be In addition to the position information of the destination candidate, the coordinate data (x, y, θ) indicating the attitude of the moving body 10 when the photographed image data of the destination candidate is obtained is stored in the storage unit 3000 at the same time. There may be. This makes it possible to correct the posture of the moving body 10 using the PTZ parameters and the coordinate data (x, y, θ) when the actual stop position of the moving body 10 with respect to the destination deviates. Become. Some data, such as the data of the autonomous movement route (GPS trajectory) of the mobile body 10 and the photographed image data of the destination candidates used for display on the display device 50, are stored in Amazon Web Services (AWS) (trademark), for example. ) may be stored in a cloud computing service.

Further, the image generating unit 46, for example, stores the current position (self-position) of the moving object 10 estimated by the self-position estimating unit 37 and the destination series managed by the destination series managing unit 36 into the map information managing unit. Render to the environment map managed by 35. The drawing method on the environment map is, for example, a method using position information such as longitude and latitude of GPS, a method using coordinate information obtained by SLAM, or the like.

The learning unit 47 is mainly realized by the processing of the CPU 301 and learns a movement route for autonomous movement of the moving body 10 . For example, the learning unit 47 performs imitation learning (machine learning )I do. The autonomous movement processing unit 43 performs autonomous movement of the moving body 10 based on learning data that is the result of imitation learning by the learning unit 47, for example.

The storage/readout unit 49 is mainly realized by the processing of the CPU 301 , stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000 .

○Map Information Management Table FIG. 6 is a conceptual diagram showing an example of the map information management table. The map information management table is a table for managing map information, which is an environmental map of a target site where the mobile object 10 is installed. A map information management DB 3001 configured by a map information management table as shown in FIG. 6 is constructed in the storage unit 3000 .

The map information management table manages map information in which the site ID and site name that identify the target site where the moving body 10 is installed and the storage position of the environment map of the target site are associated. Among these, the storage location is, for example, destination information for accessing a storage area in the mobile body 10 in which the environment map is stored or an external server, and is indicated by a URL (Uniform Resource Locator) or a URI (Uniform Resource Identifier). be

O Destination Sequence Management Table FIG. 7 is a conceptual diagram showing an example of the destination sequence management table. The destination sequence management table is a table for managing a destination sequence including a final destination or a plurality of waypoints on the movement route for specifying the movement route of the moving object 10 . A destination series management DB 3002 configured by a destination series management table as shown in FIG. 7 is constructed in the storage unit 3000 .

The destination series management table includes a series ID for identifying a destination series and an environment map of the destination series for each base ID for identifying a base where the mobile body 10 is installed and for each route ID for identifying the movement route of the mobile body 10. Position information indicating the upper position and status information indicating the movement state of the moving body 10 with respect to the destination series are managed in association with each other. Of these, the positional information is represented by latitude and longitude coordinate information indicating the position of the destination-series mobile object 10 on the environmental map. Also, the status indicates information as to whether or not the moving body 10 has reached the destination series. The status includes, for example, "arrived", "current destination", and "not reached". The status is updated according to the current position and movement state of the mobile object 10 .

○ Route Information Management Table FIG. 8 is a conceptual diagram showing an example of the route information management table. The route information management table is a table for managing route information indicating the movement route of the moving body 10 . A route information management DB 3003 configured by a route information management table as shown in FIG. 8 is constructed in the storage unit 3000 .

The route information management table associates and manages the route ID that identifies the movement route of the mobile body 10 and the route information that indicates the movement route of the mobile body 10 for each base ID that identifies the base where the mobile body 10 is installed. there is Of these, the route information indicates the future movement route of the mobile body 10 in the order of the destination sequence that will be the future destination. The route information is generated by the route information generator 38 when the moving body 10 starts to move.

○ Functional configuration of the display device ○
Next, the functional configuration of the display device 50 will be described with reference to FIG. The display device 50 has a transmission/reception section 51 , a reception section 52 , a display control section 53 , a determination section 54 , a sound output section 55 and a storage/readout section 59 . Each of these units is a function realized by any of the components shown in FIG. is. The display device 50 also has a storage unit 5000 constructed by the ROM 502, HD 504, or recording medium 521 shown in FIG.

The transmitting/receiving unit 51 is mainly implemented by the processing of the CPU 501 for the network I/F 508, and transmits/receives various data or information to/from other devices or terminals.

The reception unit 52 is mainly realized by processing of the keyboard 511 or the pointing device 512 by the CPU 501, and receives various selections or inputs from the user. The display control unit 53 is mainly implemented by the processing of the CPU 501 and causes the display unit such as the display 506 to display various screens. The determination unit 54 is realized by the processing of the CPU 501 and makes various determinations. The sound output unit 55 is mainly realized by the processing of the CPU 501 for the sound input/output I/F 513 , and outputs an audio signal such as a warning sound from the speaker 515 according to the state of the moving body 10 .

The storage/readout unit 59 is mainly realized by the processing of the CPU 501 , stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

● Processing or operation of the embodiment ○ Movement control processing ○
Next, processing or operation of the communication system according to the embodiment will be described with reference to FIGS. 9 to 21. FIG. First, with reference to FIG. 9, the overall flow of the moving operation of the moving body 10 will be schematically described. FIG. 9 is a sequence diagram illustrating an example of movement control processing of a moving object. Details of each process shown in FIG. 9 will be described later with reference to FIGS. 10 to 19 .

First, the moving body 10 sets the current destination to which the moving body 10 moves by the destination setting unit 40 (step S1). In this case, the destination setting unit 40 sets the destination based on the position and status of the destination series stored in the destination series management DB 3002 (see FIG. 7). The moving object 10 starts moving to the destination set in step S1 along the moving route indicated by the route information generated by the route information generating unit 38 (step S2). During the movement along the moving route set in step S1, the moving body 10 estimates its own position by the self-position estimating unit 37, A destination is set (step S3).

Next, the display device 50 displays an operation screen for operating the mobile body 10 on the display unit such as the display 506 based on various data or information transmitted from the mobile body 10 moving within the target site. (step S4). Then, when the moving body 10 switches between autonomous movement and manual operation based on a request from the display device 50 (YES in step S5), the process proceeds to step S6. On the other hand, when the moving body 10 does not switch between autonomous movement and manual operation (NO in step S5), the process proceeds to step S7. In step S6, the mobile body 10 switches the operation mode of the mobile body 10 by the mode setting unit 42, and moves based on the corresponding operation mode (autonomous movement mode or manual operation mode).

Then, when the moving body 10 reaches the final destination indicated by the route information generated by the route information generating unit 38 (YES in step S7), the process ends and the moving body 10 stops at the final destination. On the other hand, the moving body 10 continues the processing from step S3 until it reaches the final destination indicated in the route information (NO in step S7). Even if the moving body 10 does not reach the final destination, it may be detected that a predetermined time or more has passed since the start of movement, that an obstacle is detected on the movement route, or that a stop command is received from the operator. 3, the movement of the moving body 10 may be temporarily stopped or the operation of the movement process may be terminated in the middle.

O Processing up to Start of Movement of Moving Body Next, processing up to start of movement of the moving body 10 will be described with reference to FIGS. 10 and 11. FIG. FIG. 10 is a sequence diagram showing an example of processing up to the start of movement of the moving object.

First, the transmitting/receiving unit 51 of the display device 50 transmits a route input request indicating a request for inputting a moving route of the moving body 10 to the moving body 10 in response to a predetermined input operation or the like of the operator. (Step S11). This route input request includes a site ID that identifies the site where the moving body 10 is located. Thereby, the transmitting/receiving unit 31 of the control device 30 included in the moving body 10 receives the route input request transmitted from the display device 50 .

Next, the map information management unit 35 of the control device 30 searches the map information management DB 3001 (see FIG. 6) using the base ID received in step S11 as a search key, and finds a base ID that is the same as the received base ID. is read out via the storage/readout unit 49 (step S12). Here, as shown in FIG. 6, the map information management DB 3001 stores an environment map downloaded in advance from an external server or the like or an environment map created by remotely operating the moving body 10 by applying SLAM. position is indicated. The map information management unit 35 accesses the storage location indicated by the read map information and reads the corresponding map image data.

Next, the transmitting/receiving unit 31 transmits the map image data corresponding to the map information read in step S12 to the display device 50 that made the request (step S13). Thereby, the transmitting/receiving unit 51 of the display device 50 receives the map image data transmitted from the moving body 10 .

Next, the display control unit 53 of the display device 50 causes the display unit such as the display 506 to display the route input screen 200 including the map image data received in step S13 (step S14). FIG. 11 is a diagram showing an example of a route input screen. A route input screen 200 shown in FIG. 11 is a display screen for inputting a route on which the operator wishes to move the mobile body 10 .

A map image related to the map image data received in step S13 is displayed on the route input screen 200 . The route input screen 200 also includes a display selection button 205 that is pressed to enlarge or reduce the displayed map image, and a "complete" button 210 that is pressed to complete the route input process.

As shown in FIG. 11A, on the route input screen 200, the operator selects a predetermined position on the map image using an input means such as the pointing device 512 to display a destination series 250a. be. The operator selects positions on the map image while viewing the map image displayed on the route input screen 200 . As a result, a plurality of destination series 250a to 250h corresponding to the positions selected by the operator are displayed on the route input screen 200, as shown in FIG. 11B.

Then, as shown in FIG. 11B, when the operator selects a predetermined position on the map image and presses the "complete" button 210, the receiving unit 52 inputs the destination series 250a to 250h. is received (step S15). The transmitting/receiving unit 51 transmits the destination series data indicating the destination series 250a to 250h accepted in step S15 to the moving body 10 (step S16). This destination series data includes position information indicating the positions of the destination series 250a to 250h input in step S15 on the map image. Thereby, the transmitting/receiving unit 31 of the control device 30 included in the mobile body 10 receives the destination sequence data transmitted from the display device 50 .

Next, the destination series management unit 36 of the control device 30 stores the destination series data received in step S16 in the destination series management DB 3002 (see FIG. 7) in association with the base ID received in step S11. It is stored via the storage/reading unit 49 (step S17). The destination series management unit 36 identifies a plurality of destination series (for example, destination series 250a to 250h) indicated in the received destination series data by series ID, and identifies the corresponding destination for each series ID. Position information indicating the position on the map image of the series is associated and stored.

Next, the self-position estimation unit 37 estimates the current position of the moving body 10 (step S18). Specifically, the self-position estimation unit 37 uses a method such as an extended Kalman filter using position information indicating the position of the moving body 10 detected by the state detection unit 34 and direction information indicating the direction in which the moving body 10 is facing. The self-position (current position) of the moving body 10 is estimated by .

Next, the route information generating unit 38 generates route information indicating the movement route of the moving body 10 based on the self-position estimated in step S18 and the destination series data received in step S16 (step S19). . Specifically, the route information generator 38 uses the current position (self-position) of the mobile body 10 estimated in step S18 and the destination sequence data received in step S16 to determine the final destination of the mobile body 10. (goal) and set multiple waypoints (subgoals). Then, the route information generation unit 38 generates route information indicating the moving route of the moving body 10 from the current position to the final destination. The route information generation unit 38 avoids obstacles by, for example, connecting each waypoint from the current position to the final destination with a straight line, or by using a photographed image or obstacle information obtained by the state detection unit 34. Identify the travel route using a method that minimizes travel time while Then, the route information management unit 39 stores the route information generated by the route information generation unit 38 in the route information management DB 3003 (see FIG. 8) via the storage/readout unit 49 in association with the generated route ID. Let

Next, the destination setting unit 40 sets the destination of the moving body 10 based on the current position of the moving body 10 estimated in step S18 and the route information generated in step S19 (step S20). Specifically, based on the estimated current position (self-position) of the moving body 10, the destination setting unit 40 selects the destination to be currently headed from among the destination series indicated in the generated route information. Set (current goal) as the destination. For example, the destination setting unit 40 selects the destination series closest to the current position (self-position) of the mobile body 10 among the destination series that have not yet been reached (for example, the status is “unreached”). Set as 10 destinations. Then, the movement control unit 41 starts movement processing of the moving body 10 to the destination set in step S20 (step S21). Accordingly, the moving body 10 is autonomously moved.

In this way, the communication system 1 can autonomously move the moving body 10 based on the moving route generated according to the destination series input by the operator. Although an example of selecting a destination series by selecting a position on the map image displayed on the route input screen 200 in step S15 has been described, the route input screen 200 is based on the past data learned by the learning unit 47. A plurality of captured images may be displayed, and the operator may select a displayed captured image to select the destination series corresponding to the captured position of the captured image. In this case, the destination series data includes information identifying the selected captured image instead of position information. The destination series management DB 3002 stores the identification information of the captured image instead of the position information.

O Movement Control by Operator for Moving Body Next, control processing for the moving moving body 10 by remote control by the operator will be described with reference to FIGS. 12 to 19 . FIG. 12 is a sequence diagram showing an example of switching processing between autonomous movement of a moving body and manual operation using an operation screen. FIG. 12 shows an example in which the moving body 10 has started autonomous movement within the site by the processing shown in FIG.

First, the accuracy calculation unit 45 of the control device 30 included in the mobile body calculates the autonomous movement accuracy of the mobile body 10 (step S31). The accuracy calculator 45 calculates the autonomous movement accuracy based on the route information generated by the route information generator 38 and the current position of the moving body 10 estimated by the self-position estimator 37, for example. The accuracy of autonomous movement of the mobile object 10 is information indicating the degree of certainty (confidence) as to whether the mobile object 10 can move autonomously. show. Further, the accuracy calculation unit 45 may calculate the autonomous movement accuracy based on the learning data from the learning unit 47 and the current position of the moving body 10 estimated by the self-position estimation unit 37, for example. In this case, the accuracy of autonomous movement of the mobile body 10 is information indicating the learning accuracy regarding autonomous movement.

Further, the accuracy calculation unit 45 decreases the numerical value based on the likelihood value of the self-position estimated by the self-position estimation unit 37 when the likelihood becomes low, or uses the variance of various sensors, etc., to increase the variance. You may calculate autonomous movement precision so that a numerical value may be lowered. Further, the accuracy calculation unit 45 uses, for example, the elapsed travel time, which is the operating state of the autonomous travel processing unit 43, to lower the numerical value as the travel elapsed time in the autonomous travel mode increases, or the destination sequence and the moving object 10 The autonomous movement accuracy may be calculated so that the numerical value is lowered as the distance increases. Further, the accuracy calculation unit 45 may calculate the autonomous movement accuracy so as to lower the numerical value when there are many obstacles according to the obstacle information detected by the state detection unit 34, for example.

The photographing control unit 33 performs photographing processing using the photographing device 12 while moving within the base (step S32). The image generation unit 46 generates a virtual route image to be displayed on the photographed image obtained by the photographing process in step S32 (step S33). The route image is generated, for example, based on the current position of the mobile object 10 estimated by the self-position estimation unit 37 and the position information and status for each destination series stored in the destination series management DB 3002 . Further, the image generator 46 generates a photographed display image in which the route image generated in step S33 is drawn for the photographed image photographed in step S32 (step S34).

Further, the image generation unit 46 adds a current position display image indicating the current position (self-position) of the moving body 10 estimated by the self-position estimation unit 37 to the map image read out in step S12 and A map display image is generated in which a sequence image representing the received destination sequence is drawn (step S35).

Note that the order of the processes in steps S31 to S35 may be changed, or they may be performed in parallel. The moving body 10 continuously executes the processing of steps S31 to S35 while moving between bases. The moving body 10 generates various types of information for presenting to the operator whether the autonomous movement of the moving body 10 is going well or not through the processing from step S31 to step S35.

Next, the transmitting/receiving unit 31 transmits the notification information indicating the autonomous movement accuracy calculated in step S31, the photographed display image data generated in step S34, and the map display image generated in step S35 to the display device 50. Data is transmitted (step S36). Thereby, the transmitting/receiving unit 51 of the display device 50 receives the notification information, the photographed display image data, and the map display image data transmitted from the moving body 10 .

Next, the display control unit 53 of the display device 50 displays the operation screen 400 on the display unit such as the display 106 (step S37). FIG. 13 is a diagram showing an example of an operation screen. An operation screen 400 shown in FIG. 13 is an example of a GUI for the operator to remotely operate the mobile object 10 .

The operation screen 400 includes a map display image area 600 displaying the map display image data received in step S36, a shooting display image area 700 displaying the shooting display image data received in step S36, and a notification received in step S36. It includes a notification information display area 800 for displaying information, and a mode switching button 900 for accepting a switching operation for switching between the autonomous movement mode and the manual operation mode.

Of these, the map display image displayed in the map display image area 600 includes a current position display image 601 indicating the current position of the moving body 10, and series images 611 and 613 indicating the destination series forming the moving route of the moving body 10. , 615 and a trajectory display image showing the trajectory of the movement path of the moving body 10 are superimposed on the map image and displayed. The map display image area 600 also includes a display selection button 605 that is pressed when enlarging or reducing the displayed map image.

The series images 611, 613, and 615 display the destination series on the map image such that the operator can identify the movement history indicating the positions to which the moving body 10 has already moved, the current destination, and the future destination. there is Of these, the sequence image 611 indicates the sequence of destinations reached by the moving body 10 . Also, the series image 613 indicates the destination series, which is the current destination of the moving body 10 . Further, the series images 615 show destinations (future destinations) that the mobile body 10 has not yet reached. Series images 611, 613, and 615 are generated based on the status of the destination series stored in the destination series management DB 3002 in the process of step S35.

The photographed display image displayed in the photographed display image area 700 includes route images 711, 713, and 715 that virtually show the movement route of the moving body 10 generated in the process of step S33. The route images 711, 713, and 715 include a destination series corresponding to the positions of bases shown in the captured images, a movement history indicating positions to which the mobile body 10 has already moved, current destinations, and future destinations. displayed in an identifiable manner. Of these, the route image 711 indicates a series of destinations reached by the moving body 10 . Also, the route image 713 indicates the destination series, which is the current destination of the mobile object 10 . Furthermore, the route image 715 indicates destinations (future destinations) that the mobile body 10 has not yet reached. The route images 711, 713, 715 are generated based on the status of the destination series stored in the destination series management DB 3002 in the process of step S33. Note that the image captured by the image capturing device 12 may be displayed in the captured display image area 700 as a live streaming video distributed in real time through a computer network such as the Internet.

In the notification information display area 800, information about the accuracy of autonomous movement indicated in the notification information received in step S36 is displayed. It includes a numerical value display area 810 to be displayed and a degree display area 830 for displaying the degree of autonomous movement by discretizing the numerical value indicating the accuracy of autonomous movement. A numerical value display area 810 indicates the numerical value of the autonomous movement accuracy calculated in the process of step S31. Further, the degree display area 830 sets a predetermined threshold value for the numerical value of the autonomous movement accuracy, and indicates the degree of autonomous movement accuracy (“high, medium, low”) corresponding to the numerical value. Here, the numerical value indicating the accuracy of autonomous movement indicated in the numerical value display area 810 and the degree of autonomous movement indicated in the degree display area 830 are examples of notification information indicating the accuracy of autonomous movement. Note that the notification information display area 800 may be configured to include at least one of the numerical value display area 810 and the degree display area 830 .

The mode switching button 900 is an example of an operation accepting unit that accepts a switching operation for switching between the autonomous movement mode and the manual operation mode. The operator can switch between the autonomous movement mode and the manual operation mode of the moving body 10 by selecting the mode switching button 900 using a predetermined input means.

In the example shown in FIG. 13, the operation screen 400 shows a state in which the moving body 10 is autonomously moving with the position of the series image 613 and the route image 713 as the current destination. Further, the operation screen 400 shows that the current autonomous movement accuracy of the moving body 10 is "93.8%", which is relatively high autonomous movement accuracy.

FIG. 14 shows a state in which the moving body 10 has moved from the state shown in FIG. On the operation screen 400 shown in FIG. 14, the positions of the series image 613 and the route image 713 indicating the current destination have changed since the moving body 10 has moved from the state shown in FIG. Further, in the operation screen 400 shown in FIG. 14, the current autonomous movement accuracy of the mobile body 10 is "87.9%", and the numerical value of the autonomous movement accuracy is lower than the state shown in FIG. The degree of autonomous movement accuracy also changes from "high" to "medium". The operator can view the status of bases indicated in the map display image and the base display image displayed on the operation screen 400, and changes in the accuracy of autonomous movement indicated in the notification information display area. It can be determined whether to switch the body 10 between autonomous movement and manual operation.

Returning to FIG. 12, the reception unit 52 receives selection of the mode switching button 900 on the operation screen 400 in response to an input operation by the operator using input means such as the pointing device 512 (step S38). For example, the operation screen 400 is displayed in FIG. 15(B) when the operator selects a mode switching button 900 (displayed as "switch to manual operation") in the state shown in FIG. 15(A). The display changes to the mode switching button 900 (displayed as "resume autonomous movement") as shown. In this case, the operator selects the mode switching button 900 in order to switch the operation mode of the moving body 10 from the autonomous movement mode to the manual operation mode.

Then, the transmitting/receiving unit 51 transmits a mode switching request indicating a request to switch the mobile body 10 between the autonomous movement mode and the manual operation mode to the mobile body 10 (step S39). Thereby, the transmitting/receiving unit 31 of the control device 30 included in the moving body 10 receives the mode switching request transmitted from the display device 50 .

Next, the control device 30 executes the mode switching process of the moving body 10 in response to receiving the mode switching request in step S39 (step S40).

(Switching processing between autonomous movement and manual operation)
Here, the mode switching process in step S40 will be described in detail with reference to FIG. FIG. 16 is a flowchart showing an example of switching processing between an autonomous movement mode and a manual operation mode in a mobile object.

First, when the transmission/reception unit 31 receives the mode switching request transmitted from the display device 50 (YES in step S51), the control device 30 causes the process to proceed to step S52. On the other hand, the control device 30 continues the processing of step S51 until the mode switching request is received (NO in step S51).

Next, if the received mode switching request indicates switching to the manual operation mode (YES in step S52), the mode setting unit 42 causes the process to proceed to step S53. The movement control unit 41 stops the autonomous movement processing of the moving body 10 in response to the autonomous movement processing stop command from the autonomous movement processing unit 43 (step S53). Then, the mode setting unit 42 switches the operation of the moving body 10 from the autonomous movement mode to the manual operation mode (step S54). Then, the movement control unit 41 executes the movement of the moving body 10 by manual operation according to the drive command from the manual operation processing unit 44 (step S55).

On the other hand, if the received mode switching request does not indicate switching to the manual operation mode, that is, if it indicates switching to the autonomous movement mode, the mode setting unit 42 indicates that the switching request indicates switching to the autonomous movement mode. If so (NO in step S52), the process proceeds to step S56. The mode setting unit 42 switches the operation of the moving body 10 from the manual operation mode to the autonomous movement mode (step S56). Then, the movement control unit 41 moves the moving body 10 by autonomous movement in accordance with the driving command from the autonomous movement processing unit 43 (step S57).

In this manner, the display device 50 displays the operation screen 400 including the photographed display image in which the route images 711, 713, and 715 shown on the movement route of the moving object 10 are shown on the photographed image, thereby enabling the autonomous The operator can appropriately determine whether to switch between movement and manual operation. In addition, the display device 50 displays the map display image in which the series images 611, 613, and 615 indicating the destination series are shown on the map image together with the captured display image, so that the movement state of the moving body 10 can be more clearly understood. Since the operator can be visually grasped, the operability of the moving body 10 for the operator can be improved. Furthermore, the display device 50 allows the operator to switch between autonomous movement and manual operation by using a mode switching button 900 on the operation screen 400 on which the captured display image is displayed. It is possible to improve the operability when switching between. In addition, the moving object 10 performs movement control according to the operator's request by switching between the autonomous movement mode and the manual operation mode, which are operation modes, in response to a switching request transmitted from the display device 50. can be done.

Note that the moving body 10 not only switches the operation mode according to the switching request transmitted from the display device 50, but also, for example, according to the autonomous movement accuracy calculated by the accuracy calculation unit 45, the numerical value of the autonomous movement accuracy is The configuration may be such that the autonomous movement mode is switched to the manual operation mode when it falls below a predetermined threshold.

Further, the display device 50 may include means for not only displaying the operation screen 400 but also for notifying the operator of the degree of autonomous movement accuracy. For example, the sound output unit 55 of the display device 50 may be configured to output a warning sound from the speaker 515 when the numerical value of the autonomous movement accuracy is below a predetermined threshold. Further, the display device 50 may be configured to vibrate input means such as a controller used for manual operation of the moving body when the numerical value of the autonomous movement accuracy is below a predetermined threshold.

Furthermore, instead of directly displaying the autonomous movement accuracy on the operation screen 400, the display device 50 may display a predetermined message based on the numerical value or degree of the autonomous movement accuracy as notification information. In this case, the operation screen 400 may display a message prompting the operator to switch to manual operation, for example, when the numerical value or degree of autonomous movement accuracy falls below a predetermined threshold. Further, the operation screen 400 may display a message prompting the operator to switch from manual operation to autonomous movement, for example, when the numerical value or degree of autonomous movement accuracy exceeds a predetermined threshold.

○Autonomous Movement Processing Next, the autonomous movement processing of the moving body 10 executed by the processing shown in step S57 will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of autonomous movement processing of a moving object.

First, the destination setting unit 40 of the control device 30 provided in the mobile unit 10 determines the current position of the mobile unit 10 estimated by the self-position estimation unit 37 and the route information stored in the route information management DB 3003 (see FIG. 8). Based on this, the destination of the moving body 10 is set (step S71). Specifically, the destination setting unit 40 selects, for example, the current position of the moving body 10 estimated by the self-position estimation unit 37 from among the destination series indicated in the route information stored in the route information management DB 3003. and the position indicated by the closest destination sequence is set as the destination. In the example shown in FIG. 7, the position of the destination series with the series ID "P003" whose status is the current destination is set as the destination. Then, the destination setting unit 40 generates a travel route to the set destination. The method of generating the movement route by the movement destination setting unit 40 includes a method of connecting the current position and the movement destination with a straight line, or a method of avoiding obstacles using a photographed image or obstacle information obtained by the state detection unit 34, and calculating the movement time. A method that minimizes is used.

Then, the movement control unit 41 moves the moving body 10 to the destination set so as to pass through the movement route generated in step S71. In this case, the movement control unit 41 autonomously moves the moving body 10 according to the drive command from the autonomous movement processing unit 43 . The autonomous movement processing unit 43, for example, executes autonomous movement based on learning data that is the result of imitation learning by the learning unit 47 (step S72).

Then, when the moving body 10 reaches the final destination or the autonomous movement by the autonomous movement processing section 43 is interrupted (YES in step S73), the movement control section 41 ends the process. For example, when the autonomous movement is interrupted, the mode setting unit 42 switches from the autonomous movement mode to the manual mode in response to a request for switching from the autonomous movement mode to the manual operation mode as shown in FIG. is the case. On the other hand, the movement control unit 41 continues the autonomous movement processing in step S72 until the arrival of the moving body 10 at the final destination or the interruption of the autonomous movement by the autonomous movement processing unit 43 is detected (NO in step S73). ).

In this way, the moving body 10 operates autonomously using the generated route information and learning data learned during the manual operation mode during operation in the autonomous movement mode set in response to a switching request or the like from the operator. You can move. In addition, the mobile body 10 can perform autonomous movement using the learning data by learning about autonomous movement using various data acquired in the manual operation mode, and the mobile body 10 can Autonomous movement accuracy can be improved.

(Manual Operation Processing) Next, the manual operation processing of the moving body 10 executed by the processing shown in step S55 will be described with reference to FIGS. 18 and 19. FIG. FIG. 18 is a sequence diagram illustrating an example of manual operation processing of a moving body;

First, the reception unit 52 of the display device 50 receives a manual operation command in response to an operator's input operation on the operation command input screen 450 as shown in FIG. 19 (step S91). FIG. 19 is a diagram showing an example of an operation command input screen. An operation command input screen 450 shown in FIG. 19 displays icons for remotely operating the moving body 10 . The operation command input screen 450 is displayed on the operation screen 400, for example, when the operation mode of the moving body 10 is set to the manual operation mode. The operation command input screen 450 shows the state of the movement instruction key 455 that is pressed when requesting horizontal movement (forward, backward, right rotation, left rotation) of the mobile body 10 and the moving speed of the mobile body 10 . It includes a speed bar 457 where the speed of movement is displayed. When an operator who remotely operates mobile object 10 using display device 50 selects movement instruction key 455 , reception unit 52 receives a manual operation command for selected movement instruction key 455 .

In FIG. 19, an example of remotely controlling the movement of the mobile body 10 by receiving a selection for the movement instruction key 455 displayed on the operation command input screen 450 will be described. Alternatively, it may be configured to be performed by a dedicated controller such as a game pad equipped with a joystick. Further, in the input operation of the movement instruction key 455 by the operator, when the operator selects "retreat (↓)" while the moving body 10 is moving forward, the photographed image is displayed on the rear screen of the moving body 10. A configuration may be adopted in which the switching is performed, and the moving body 10 is moved backward (backward) from that point. Also, the transmission of the manual operation command from the display device 50 to the moving object 10 may be performed via a managed cloud platform such as AWS IoT Core (trademark), for example.

Next, the transmitting/receiving unit 51 transmits the manual operation command accepted in step S91 to the moving body 10 (step S92). As a result, the transmission/reception unit 31 of the control device 30 included in the moving body 10 receives the manual operation command transmitted from the display device 50 . Then, the manual operation processing unit 44 of the control device 30 outputs a drive command to the movement control unit 41 based on the manual operation command received in step S92. The movement control unit 41 executes movement processing of the moving body 10 according to the drive command from the manual operation processing unit 44 (step S93). Further, the learning unit 47 performs imitation learning (machine learning) of the movement route according to the manual operation by the manual operation processing unit 44 (step S94). The learning unit 47 performs imitation learning of a movement route related to autonomous movement, for example, based on the captured image acquired by the manual operation processing unit 44 during the movement operation in the manual operation mode and the detection data by the state detection unit 34 . Note that the learning unit 47 may be configured to perform imitation learning of the movement route using only the photographed image acquired during manual operation, or may use both the photographed image and the detection data by the state detection unit 34. A configuration may be adopted in which imitation learning of the moving route is performed by The photographed image used for imitation learning by the learning unit 47 may be a photographed image acquired during autonomous movement in the autonomous movement mode by the autonomous movement processing unit 43 .

In this way, the moving body 10 can move according to a manual operation command from the operator during operation in the manual operation mode set according to a switching request or the like from the operator. In addition, the moving body 10 can learn about autonomous movement using various data such as captured images acquired in the manual operation mode.

○Modified example of operation screen○
Next, modified examples of the operation screen 400 displayed on the display device 50 will be described with reference to FIGS. 20 to 25. FIG. FIG. 20 is a diagram showing Modification 1 of the operation screen. In addition to the configuration of the operation screen 400, the operation screen 400A shown in FIG. 20 displays notification information indicating autonomous movement accuracy in the map display image area 600 and the shooting display image area 700. FIG.

The map display image displayed in the map display image area 600 of the operation screen 400A includes, in addition to the configuration displayed in the map display image area 600 of the operation screen 400, an accuracy display image 660 indicating the degree of autonomous movement accuracy on the map image. including. Similarly, the photographed display image displayed in the photographed display image area 700 of the operation screen 400A has, in addition to the configuration displayed in the photographed display image area 700 of the operation screen 400, an accuracy level indicating the degree of accuracy of autonomous movement on the photographed image. Includes display image 760 . Accuracy display images 660 and 760 display the degree of autonomous movement accuracy by the size of a circle. Accuracy display images 660 and 760 show the uncertainty of autonomous movement or self-position, for example, by making the size of the circle smaller as the accuracy of autonomous movement is higher and the size of the circle larger as the accuracy of autonomous movement is lower. . Here, accuracy display image 660 and accuracy display image 760 are examples of notification information indicating the accuracy of autonomous movement. In addition, the accuracy display images 660 and 760 may be configured to indicate the degree of autonomous movement accuracy by a method such as changing the color of the circle according to the degree of autonomous movement accuracy.

Accuracy display image 660 is generated by drawing on the map image by the process of step S35 based on the numerical value of the autonomous movement accuracy calculated by accuracy calculator 45 . Similarly, the accuracy display image 760 is generated by drawing on the photographed image by the process of step S34 based on the numerical value of the autonomous movement accuracy calculated by the accuracy calculation unit 45 . The operation screen 400A displays a map display image in which an accuracy display image 660 is superimposed on a map image, and a photographed display image in which an accuracy display image 760 is superimposed on a photographed image.

In this way, the operation screen 400A displays an image indicating the accuracy of autonomous movement on the map image and the photographed image, so that the operator can intuitively understand the current movement of the moving body 10 while watching the moving situation of the moving body 10. The accuracy of autonomous movement of the body 10 can be grasped.

FIG. 21 is a diagram showing Modification 2 of the operation screen. Operation screen 400B shown in FIG. 21 displays, in addition to the configuration of operation screen 400, notification information indicating autonomous movement accuracy in map display image area 600 and shooting display image area 700, similar to operation screen 400A. ing.

The map display image displayed in the map display image area 600 of the operation screen 400B includes, in addition to the configuration displayed in the map display image area 600 of the operation screen 400, an accuracy display image 670 indicating the degree of autonomous movement accuracy on the map image. including. Similarly, the photographed display image displayed in the photographed display image area 700 of the operation screen 400B has, in addition to the configuration displayed in the photographed display image area 700 of the operation screen 400, an accuracy level indicating the degree of autonomous movement accuracy on the photographed image. Includes display image 770 . Accuracy display images 670 and 770 display degrees of autonomous movement accuracy in contour maps. The accuracy display images 670 and 770 show, for example, the degree of autonomous movement accuracy at each position on the map image and the photographed image using contour lines. Here, accuracy display image 670 and accuracy display image 770 are examples of notification information indicating the accuracy of autonomous movement. Note that the accuracy display images 670 and 770 may be configured to indicate the degree of autonomous movement accuracy by a method such as changing the color of contour lines according to the degree of autonomous movement accuracy.

The accuracy display image 670 is generated by drawing on the map image by the process of step S35 based on the numerical value of the autonomous movement accuracy calculated by the accuracy calculation unit 45 . Similarly, the accuracy display image 770 is generated by drawing on the captured image by the process of step S34 based on the numerical value of the autonomous movement accuracy calculated by the accuracy calculation unit 45. FIG. The operation screen 400B displays a map display image in which an accuracy display image 670 is superimposed on a map image, and a photographed display image in which an accuracy display image 770 is superimposed on a photographed image.

In this manner, the operation screen 400B displays an image of contour lines indicating the accuracy of autonomous movement on the map image and the photographed image, so that it becomes clear which area has the low accuracy of autonomous movement, and the manual operation of the operator becomes clear. It can be visually assisted to follow a route with high autonomous movement accuracy. Further, in the case of using machine learning or the like to improve autonomous movement performance each time a manual operation is performed, the communication system 1 allows the operator to move by manual operation where the autonomous movement accuracy is low by looking at a contour map showing the accuracy of autonomous movement. By accumulating learning data, it is possible to expand the area in which autonomous movement is possible.

FIG. 22 is a diagram showing a modified example 3 of the operation screen. In addition to the configuration of the operation screen 400, the operation screen 400C shown in FIG. 22 displays the degree of autonomous movement accuracy in the notification information display area 800 with stepwise different face images.

Notification information display area 800 of operation screen 400C includes, in addition to the configuration displayed in notification information display area 800 of operation screen 400, a degree display area 835 that indicates the degree of autonomous movement as a face image. Similar to the degree display area 830, the degree display area 835 discretizes the numerical value indicating the accuracy of autonomous movement and displays it as the degree of autonomous movement. The degree display area 835 sets a predetermined threshold value for the numerical value of the autonomous movement accuracy, and switches and displays the expression of the face image according to the numerical value of the autonomous movement accuracy calculated by the accuracy calculation unit 45 . Here, the face image shown in the degree display area 835 is an example of notification information indicating the accuracy of autonomous movement. Note that the degree display area 835 is not limited to the face image, and may be configured to display a predetermined illustrated image or the like that allows the operator to recognize the degree of autonomous movement accuracy step by step.

FIG. 23 is a diagram showing Modification 4 of the operation screen. An operation screen 400D shown in FIG. 23 displays the autonomous movement accuracy in the color of the screen frame of the operation screen in addition to the configuration of the operation screen 400 .

Operation screen 400D includes, in addition to the configuration of operation screen 400, screen frame display area 430 in which the degree of autonomous movement accuracy is converted into a color and displayed as a screen frame. The screen frame display area 430 changes the color of the screen frame according to the degree of autonomous movement accuracy. The screen frame display area 430 sets a predetermined threshold for the numerical value of the autonomous movement accuracy, and changes the color of the screen frame according to the numerical value of the autonomous movement accuracy calculated by the accuracy calculation unit 45 . For example, the screen frame display area 430 displays the screen frame in red when the accuracy of autonomous movement is low, and in blue when the accuracy of autonomous movement is high. Here, the color of the screen frame shown in the screen frame display area 430 is an example of notification information indicating the accuracy of autonomous movement. Note that the operation screen 400D may have a configuration in which not only the screen frame but also the color of the entire operation screen is changed according to the degree of autonomous movement accuracy.

FIG. 24 is a diagram showing Modification 5 of the operation screen. In addition to the configuration of the operation screen 400, the operation screen 400E shown in FIG. 24 displays the desired direction toward the moving body 10 during manual operation in the map display image area 600 and the shooting display image area 700. .

The map display image displayed in the map display image area 600 of the operation screen 400E includes, in addition to the configuration displayed in the map display image area 600 of the operation screen 400, the direction in which the moving body 10 wants to go during manual operation on the map image. and a direction display image 690 with an arrow pointing to . Similarly, the photographed display image displayed in the photographed display image area 700 of the operation screen 400E, in addition to the configuration displayed in the photographed display image area 700 of the operation screen 400, has the moving body 10 facing toward the manual operation on the photographed image. It includes a direction display image 790 that indicates the desired direction with an arrow. The desired direction during manual operation is, for example, a direction that indicates an area with high autonomous movement accuracy, and is a direction that leads to a position where there is a high possibility that autonomous movement can be resumed. Note that the direction display images 690 and 790 are not limited to displays using arrows, and may have a display configuration that allows the operator to identify the direction in which the moving body 10 is desired to move during manual operation.

In this way, the operation screen 400E allows the operator to visually grasp the direction in which the moving body 10 should move by displaying the desired direction of the moving body 10 during manual operation on the map image and the captured image. be able to.

FIG. 25 is a diagram showing Modification 6 of the operation screen. The operation screen 400F shown in FIG. 25 does not display the map display image area 600 that was displayed on each of the operation screens described above, and instead displays the photographing display image area 700, the notification information display area 800, and the mode switching button 900. is displayed.

Of these, the shooting display image displayed in the shooting display image area 700 of the operation screen 400F displays the accuracy display image 760 shown on the operation screen 400B and the direction display image 690 shown on the operation screen 400E on the shot image. there is Further, in the photographed display image displayed on the operation screen 400F, the route images 711, 713, and 715 are not displayed on the photographed image, unlike the operation screens described above. Notification information display area 800 and mode switching button 900 are the same as those displayed on operation screen 400 .

In this manner, the operation screen 400F displays at least the photographed image photographed by the moving body 10 and the notification information indicating the accuracy of autonomous movement of the moving body 10, so that the operator can use the minimum necessary information to control the movement of the moving body. 10 movement states can be grasped. In addition to or instead of the elements shown in FIG. 25, the operation screen 400F displays the elements displayed in the shooting display image area 700 and the notification information display area 800 in each of the operation screens described above. It may be configured to be

Effect of the Embodiment As described above, the communication system 1 allows the operator to display a photographed display image in which the route images 711, 713, and 715 shown on the movement route of the moving body 10 are shown on the photographed image. By displaying it on the operation screen to be used, the operator can appropriately grasp the movement state of the mobile body 10, and the operator can easily judge switching between autonomous movement and manual operation. In addition, the communication system allows the operator to switch between autonomous movement and manual operation using a mode switching button 900 on an operation screen on which a photographed display image or the like is displayed, whereby the operator can switch between autonomous movement and manual operation. It is possible to improve the operability when switching.

Furthermore, the communication system 1 can switch the mobile body 10 between the autonomous movement mode and the manual operation mode in response to a switch request from the operator. Manual operation switching control can be performed. In addition, the communication system 1 allows the operator to appropriately determine the necessity of learning by manual operation for the moving body 10 that performs learning about autonomous movement using captured images acquired in the manual operation mode. can be done.

Here, each of the operation screens described above may have a configuration in which at least the photographed display image shown in the photographed display image area 700 is displayed. Also, the mode switching button 900 shown on each operation screen may be replaced by input means such as the keyboard 511 of the display device 50 without being displayed on the operation screen. Further, the communication system 1 may be configured to include external input means such as a dedicated button for accepting switching operation between the autonomous movement mode and the manual operation mode outside the display device 50 . In these cases, an input means such as the keyboard 511 of the display device 50 or an external input means such as a dedicated button outside the display device 50 is an example of the operation receiving means. Furthermore, the system includes a display device 50 that displays an operation screen including the mode switching button 900, a display device 50 that accepts a switching operation using input means such as a keyboard 511, or a display device 50 and external input means such as a dedicated button. is an example of the display system in this embodiment. Furthermore, the operation accepting means includes means capable of accepting not only the switching operation between the autonomous movement mode and the manual operation mode using the mode switching button 900 or the like, but also the operation for performing predetermined control of the moving body 10. may be

●Modified example of the embodiment ○Modified example 1○
Next, Modification 1 of the communication system according to the embodiment will be described with reference to FIGS. 26 and 27. FIG. The same reference numerals are given to the same configurations and the same functions as those of the above-described embodiment, and the description thereof will be omitted. The communication system 1A according to Modification 1 is an example in which the calculation of the autonomous movement accuracy of the moving body 10A and the generation of various display images to be displayed on the operation screen 400 or the like are performed by the display device 50A.

26 is a diagram illustrating an example of a functional configuration of a communication system according to Modification 1 of the embodiment; FIG. A display device 50A according to Modification 1 shown in FIG. 26 includes an accuracy calculator 56 and an image generator 57 in addition to the configuration of the display device 50 shown in FIG.

The accuracy calculation unit 56 is mainly realized by the processing of the CPU 501, and calculates the accuracy of autonomous movement of the moving body 10A. The image generator 57 is mainly implemented by the processing of the CPU 501 and generates a display image to be displayed on the display device 50A. Accuracy calculator 56 and image generator 57 have the same configurations as accuracy calculator 45 and image generator 46 shown in FIG. 5, respectively. Along with this, the control device 30A that controls the processing or operation of the moving body 10A according to Modification 1 does not have the functions of the accuracy calculation unit 45 and the image generation unit 46 .

FIG. 27 is a sequence diagram illustrating an example of switching processing between autonomous movement of a moving body and manual operation using an operation screen according to Modification 1 of the embodiment. Similar to FIG. 12, FIG. 27 shows an example in which the moving body 10A starts autonomous movement within the site by the processing shown in FIG.

First, the photographing control unit 33 of the control device 30A provided in the moving body 10A performs photographing processing using the photographing device 12 while moving within the base (step S101). Then, the transmitting/receiving unit 31 sends the captured image data captured in step S101, the map image data read out in step S12, the route information stored in the route information management DB 3003, and the self-position estimation to the display device 50A. The position information indicating the current position (self-position) of the moving body 10A estimated by the unit 37 and the learning data by the learning unit 47 are transmitted (step S102). Thereby, the transmitting/receiving unit 51 of the display device 50A receives various data and information transmitted from the moving object 10A.

Next, the accuracy calculator 56 of the display device 50A calculates the autonomous movement accuracy of the mobile object 10A (step S103). The accuracy calculator 45 calculates autonomous movement accuracy based on, for example, the route information and position information received in step S102. Also, the accuracy calculation unit 56 may calculate the autonomous movement accuracy based on the learning data and the position information received in step S102, for example.

Next, the image generator 57 generates a route image to be displayed on the captured image received in step S102 (step S104). The route image is generated, for example, based on the position information received in step S102 and the position information and status for each destination sequence indicated in the route information received in step S102. Further, the image generator 57 generates a photographed display image in which the route image generated in step S104 is drawn for the photographed image received in step S102 (step S105). Furthermore, the image generation unit 57 generates a current position display image indicating the current position (self-position) of the moving body 10A indicated in the position information received in step S102 and a map image received in step S102. A map display image is generated in which a sequence image representing the destination sequence indicated in the route information received in step S102 is drawn (step S106).

Details of the processing of steps S103, S104, S105 and S106 are the same as the processing of steps S31, S33, S34 and S35 shown in FIG. 12, respectively. Note that the order of the processes in steps S103 to S106 may be changed, or they may be performed in parallel. The display device 50A receives the photographed image data and the like transmitted from the moving body 10A by the processing of step S102 as needed, and continuously executes the processing of steps S103 to S106.

Next, the display control unit 53 displays the operation screen 400 shown in FIG. 13 and the like on the display unit such as the display 106 (step S107). The display control unit 53 causes the operation screen 400 to display information calculated or generated in the processing of steps S103 to S106. Note that the display control unit 53 is not limited to the operation screen 400, and may be configured to display any one of the operation screens 400A to 400F described above. Since the subsequent steps S108 to S110 are the same as the steps S38 to S40 shown in FIG. 12, description thereof will be omitted.

As described above, the communication system 1A according to Modification 1 displays the operation screen 400 including the captured display image and the like on the display device 50A even when calculating the autonomous movement accuracy and generating various display screens on the display device 50A. , the operator can easily determine the movement state of the moving body 10A and the switching between autonomous movement and manual operation.

○Modification 2○
Next, Modification 2 of the communication system according to the embodiment will be described with reference to FIGS. 28 to 31. FIG. The same reference numerals are given to the same configurations and the same functions as those of the above-described embodiment, and the description thereof will be omitted. The communication system 1B according to Modification 2 is an example in which the information processing device 90 executes the calculation of the autonomous movement accuracy of the moving object 10B and the generation of various display images to be displayed on the operation screen 400 and the like.

FIG. 28 is a diagram illustrating an example of the overall configuration of a communication system according to Modification 2 of the embodiment. A communication system 1B according to Modification 2 includes an information processing device 90 capable of communicating with a mobile object 10B and a display device 50B via a communication network 100 in addition to the configurations of the above-described embodiments.

The information processing device 90 is a server computer for managing communication between the mobile body 10B and the display device 50B, various controls of the mobile body 10B, and generation of various display screens to be displayed on the display device 50B. The information processing apparatus 90 may be configured by one server computer, or may be configured by a plurality of server computers. Further, although the information processing device 90 is described as being a server computer that exists in a cloud environment, it may be a server that exists in an on-premises environment. Here, the hardware configuration of the information processing device 90 has the same configuration as that of the display device 50 as shown in FIG. Hereinafter, for convenience, the hardware configuration of the information processing apparatus 90 will be described using 900-series numerals for the configuration shown in FIG.

29 is a diagram illustrating an example of a functional configuration of a communication system according to Modification 2 of the embodiment; FIG. The configuration of the display device 50B according to Modification 2 shown in FIG. 29 is the same as the configuration of the display device 50 shown in FIG. Further, the control device 30B that controls the processing or operation of the moving object 10B according to the second modification includes the functions of the map information management unit 35, the accuracy calculation unit 45, and the image generation unit 46, and the map information built in the storage unit 3000. It becomes the structure which does not have management DB3001.

The information processing device 90 has a transmission/reception unit 91 , a map information management unit 92 , an accuracy calculation unit 93 , an image generation unit 94 and a storage/readout unit 99 . Each of these units has a function realized or functions by any one of the components shown in FIG. It is a means. The information processing apparatus 90 also has a storage unit 9000 constructed by the ROM 902, the HD 904, or the recording medium 921 shown in FIG.

The transmitting/receiving unit 91 is mainly realized by the processing of the CPU 901 with respect to the network I/F 908, and transmits/receives various data or information to/from other devices or terminals.

The map information management unit 92 is mainly realized by the processing of the CPU 901, and uses the map information management DB 9001 to manage map information indicating the environment map of the target site where the moving body 10B is installed. The map information management unit 92 manages, for example, map information indicating an environment map downloaded from an external server or the like, or an environment map created by applying SLAM.

The accuracy calculator 93 is mainly implemented by the processing of the CPU 901, and calculates the accuracy of autonomous movement of the moving body 10B. The image generator 94 is mainly implemented by the processing of the CPU 301 and generates a display image to be displayed on the display device 50B. Accuracy calculator 93 and image generator 94 have the same configurations as accuracy calculator 45 and image generator 46 shown in FIG. 5, respectively.

The storage/readout unit 99 is mainly realized by processing of the CPU 901 , stores various data (or information) in the storage unit 9000 , and reads out various data (or information) from the storage unit 9000 . A map information management DB 9001 is constructed in the storage unit 9000 . The map information management DB 9001 is composed of the map information management table shown in FIG.

FIG. 30 is a sequence diagram illustrating an example of processing up to the start of movement of the moving body according to Modification 2 of the embodiment. First, the transmission/reception unit 51 of the display device 50B transmits a route input request to the information processing device 90 in response to a predetermined input operation or the like by the operator. (step S201). This route input request includes a site ID that identifies the site where the moving body 10B is located. Thereby, the transmitting/receiving unit 91 of the information processing device 90 receives the route input request transmitted from the display device 50B.

Next, the map information management unit 92 of the information processing device 90 searches the map information management DB 9001 (see FIG. 6) using the base ID received in step S201 as a search key, and finds the same base as the received base ID. The map information associated with the ID is read out via the storage/readout section 99 (step S202). The map information management unit 92 accesses the storage location indicated by the read map information and reads the corresponding map image data.

Next, the transmitting/receiving unit 91 transmits the map image data corresponding to the map information read in step S202 to the display device 50B that made the request (step S203). Thereby, the transmitting/receiving unit 51 of the display device 50B receives the map image data transmitted from the information processing device 90 .

Next, the display control unit 53 of the display device 50B causes the display unit such as the display 506 to display the route input screen 200 (see FIG. 11) including the map image data received in step S203 (step S204). Then, when the operator selects a predetermined position on the map image and presses the "complete" button 210, the reception unit 52 accepts input of the destination series 250a to 250h in the same manner as in step S15 of FIG. Accept (step S205). The transmitting/receiving unit 51 transmits destination series data indicating the destination series 250a to 250h accepted in step S205 to the information processing device 90 (step S206). This destination series data includes position information indicating the positions of the destination series 250a to 250h input in step S205 on the map image. The transmitting/receiving unit 91 of the information processing device 90 transmits (transfers) the destination sequence data transmitted from the display device 50B to the moving body 10B (step S207). Thereby, the transmitting/receiving unit 31 of the control device 30B included in the moving body 10B receives the destination series data transmitted from the display device 50B. Since the subsequent steps S208 to S212 are the same as the steps S17 to S21 shown in FIG. 10, description thereof will be omitted.

FIG. 31 is a sequence diagram illustrating an example of switching processing between autonomous movement of a moving body and manual operation using an operation screen according to Modification 2 of the embodiment. Similar to FIG. 12, FIG. 31 shows an example in which the moving body 10B starts autonomous movement within the base by the processing shown in FIG.

First, the photographing control unit 33 of the control device 30B provided in the moving body 10B performs photographing processing using the photographing device 12 while moving within the base (step S231). Then, the transmitting/receiving unit 31 sends the captured image data captured in step S231, the route information stored in the route information management DB 3003, and the position of the moving body 10B estimated by the self-position estimating unit 37 to the information processing device 90. The position information indicating the current position (self-position) and the learning data by the learning unit 47 are transmitted (step S232). Thereby, the transmitting/receiving unit 91 of the information processing device 90 receives various data and information transmitted from the moving body 10B.

Next, the accuracy calculator 93 of the information processing device 90 calculates the autonomous movement accuracy of the moving body 10B (step S233). The accuracy calculator 45 calculates the autonomous movement accuracy, for example, based on the route information and position information received in step S232. Also, the accuracy calculator 56 may calculate the autonomous movement accuracy based on the learning data and the position information received in step S232, for example.

Next, the image generator 94 generates a route image to be displayed on the captured image received in step S232 (step S234). The route image is generated, for example, based on the position information received in step S232 and the position information and status for each destination sequence indicated in the route information received in step S232. Further, the image generator 57 generates a photographed display image in which the route image generated in step S234 is drawn for the photographed image received in step S232 (step S235). Further, the image generation unit 94 adds a current position display image indicating the current position (self-position) of the moving body 10B indicated in the position information received in step S232 to the map image read out in step S202. Then, a map display image is generated in which a sequence image representing the destination sequence indicated in the route information received in step S232 is drawn (step S236).

The details of the processing of steps S233, S234, S235 and S236 are the same as the processing of steps S31, S33, S34 and S35 shown in FIG. 12, respectively. Note that the order of the processes in steps S233 to S236 may be changed, or they may be performed in parallel. The information processing device 90 receives the photographed image data and the like transmitted from the moving body 10 by the processing of step S232 as needed, and continuously executes the processing of steps S233 to S236.

Next, the transmitting/receiving unit 91 transmits the notification information indicating the autonomous movement accuracy calculated in step S233, the photographed display image data generated in step S235, and the map display image generated in step S236 to the display device 50B. Data is transmitted (step S237). Thereby, the transmitting/receiving unit 51 of the display device 50</b>B receives the notification information, the photographed display image data, and the map display image data transmitted from the information processing device 90 .

Next, the display control unit 53 of the display device 50B displays the operation screen 400 shown in FIG. 13 and the like on the display unit such as the display 106 (step S238). Display control unit 53 displays the data and information received in step S<b>237 on operation screen 400 . Note that the display control unit 53 is not limited to the operation screen 400, and may be configured to display any one of the operation screens 400A to 400F described above.

Next, similarly to step S38 in FIG. 12, the reception unit 52 receives selection of the mode switching button 900 on the operation screen 400 in response to an input operation by the operator using input means such as the pointing device 512 (step S239). The transmitting/receiving unit 51 transmits a mode switching request indicating a request for switching between the autonomous movement mode and the manual operation mode of the moving object 10B to the information processing device 90 (step S240). Then, the transmission/reception unit 91 of the information processing device 90 transmits (transfers) the mode switching request transmitted from the display device 50B to the moving object 10B (step S241). Thereby, the transmitting/receiving unit 31 of the control device 30B included in the moving object 10B receives the mode switching request transmitted from the display device 50B. Then, in response to receiving the mode switching request in step S241, the control device 30B executes the mode switching process of the moving body 10B as shown in FIG. 16 (step S242).

As described above, the communication system 1B according to the second modification can display the operation screen 400 including the captured display image and the like even when the information processing device 90 calculates the autonomous movement accuracy and generates various display screens. Since it can be displayed on 50B, the operator can easily determine the movement state of the moving body 10 and switching between autonomous movement and manual operation.

32 is a diagram illustrating an example of a functional configuration of a communication system; FIG. In comparison with the functional configuration of the communication system shown in FIG. 29, the display device 50 has the same configuration as the display device 50 shown in FIG. The control device 30C for controlling the processing or operation of the moving object 10C is constructed from the control device 30B shown in FIG. It is a configuration in which the destination series management DB 3002 and the route information management DB 3003 that have been used are removed.

In the communication system 1C shown in FIG. 32, the information processing device 90 is compatible with, for example, cloud computing services such as AWS (trademark), and the communication system 1C includes the display device 50 and the mobile body 10C (control device 30C) communicate through information processing device 90 as indicated by arrows a and b. Also, the functions of the destination series management unit 36, the route information generation unit 38, the route information management unit 39, the destination series management DB 3002, and the route information management DB 3003 removed from the control device 30B are transferred to the information processing device 90. there is That is, the information processing device 90 includes a transmission/reception unit 91, a map information management unit 92, an accuracy calculation unit 93, an image generation unit 94, a destination series management unit 95, a route information generation unit 96, and a route information management unit 97. there is Further, in the storage unit 9000 of the information processing device 90, a map information management DB 9001, a destination series management DB 9002, and a route information management DB 9003 are constructed. It should be noted that the functions of the respective units transferred from the control device 30B (FIG. 29) to the information processing device 90 are the same as the functions explained with reference to FIG.

As described above, in the communication system 1C, communication between the display device 50 and the mobile object 10C (control device 30C) is performed via the information processing device 90 compatible with the cloud computing service. In such an information processing device 90, security of manual operation commands from the display device 50 and photographed image data from the moving body 10C can be enhanced by using authentication processing by a cloud computing service at the time of communication. can. In addition, by placing the generation function and management function of each data in the information processing device 90 (cloud service), it becomes possible to share the same data at multiple sites, P2P (Peer to Peer) communication (one-to-one communication) Direct communication), but also one-to-many point communication can be flexibly handled.

●Summary●
As described above, the display system according to one embodiment of the present invention is a display system for displaying an image captured by a moving body 10 (10A, 10B, 10C) moving within a predetermined location. On the movement route of the mobile body 10 (10A, 10B, 10C) at the base shown in the received photographed image, Virtual route images 711, 713 and 715 are superimposed and displayed. Thereby, the display system according to one embodiment of the present invention can allow the user to appropriately grasp the moving state of the moving body 10 (10A, 10B, 10C).

Further, in the display system according to one embodiment of the present invention, the route images 711, 713, 715 include images showing a plurality of points on the movement route, and the movement of the moving object 10 (10A, 10B, 10C) is displayed. Contains images showing the history and future destinations of the mobile 10 (10A, 10B, 10C). As a result, the display system according to one embodiment of the present invention provides a photographed display in which the route images 711, 713, and 715 shown on the moving route of the moving body 10 (10A, 10B, 10C) are shown on the photographed image. By displaying the image on the operation screen 400 or the like used by the operator, the operator can appropriately grasp the movement state of the moving body 10 (10A, 10B, 10C).

Further, the display system according to one embodiment of the present invention accepts input of route information indicating the movement route of the mobile body 10 (10A, 10B, 10C), and displays the accepted route information as the mobile body 10 (10A). , 10B, 10C), and moves the moving object 10 (10A, 10B, 10C) based on the transmitted route information. Further, the display system accepts the route information input on the map image showing the bases, and displays the map image superimposed with the sequence images 611, 613, 615 showing the route information as virtual route images 711, 713, 715. is displayed together with the photographed image on which is superimposed. As a result, the display system according to the embodiment of the present invention displays the map display image in which the series images 611, 613, and 615 indicating the route information are displayed on the map image together with the photographed display image. Since the moving state of the body 10 (10A, 10B, 10C) can be more visually recognized by the operator, the operability of the moving body 10 (10A, 10B, 10C) by the operator can be improved.

Moreover, the display system according to one embodiment of the present invention includes an operation reception unit that receives an operation for performing predetermined control on the moving body 10 (10A, 10B, 10C). Further, the operation receiving means switches between a manual operation mode in which the moving bodies 10 (10A, 10B, 10C) are moved by manual operation and an autonomous movement mode in which the moving bodies 10 (10A, 10B, 10C) are moved by autonomous movement. A mode switching button 900 for accepting a switching operation. As a result, the display system according to one embodiment of the present invention allows the operator to switch between autonomous movement and manual operation using the mode switching button 900, thereby allowing the operator to switch between autonomous movement and manual operation. Operability can be improved.

Furthermore, in the display system according to one embodiment of the present invention, the autonomous movement is autonomous movement based on learning, and when the moving body 10 (10A, 10B, 10C) switches from the autonomous movement mode to the manual operation mode, the autonomous Learning for movement becomes possible. Also, learning for autonomous movement is performed using captured images captured by the moving bodies 10 (10A, 10B, 10C). As a result, the display system according to one embodiment of the present invention performs learning for autonomous movement using the captured image, thereby allowing the moving body 10 (10A, 10B, 10C) to autonomously move using the learning data. While being able to perform, the autonomous movement precision of the mobile body 10 (10A, 10B, 10C) can be improved.

Further, a communication system according to an embodiment of the present invention includes a display system for displaying images captured by mobile bodies 10 (10A, 10B, 10C) moving within a predetermined location, , 10C), and position information indicating the current position of the mobile body 10 (10A, 10B, 10C) and the moving route of the mobile body 10 (10A, 10B) Based on the indicated route information, a display image is generated by superimposing the virtual route images 711, 713, 715 on the captured image, and the generated display image is displayed. As a result, the communication system 1 (1A, 1B, 1C) generates and displays a photographed display image visually showing the moving route of the moving body 10 (10A, 10B, 10C), thereby enabling the moving body 10 The operator can appropriately grasp the moving state of (10A, 10B, 10C).

Further, in the communication system according to an embodiment of the present invention, the moving objects 10 (10A, 10B, 10C) receive a switching request between the autonomous movement mode and the manual operation mode transmitted from the display system, and Based on the switching request, either the autonomous movement mode or the manual operation mode is set, and movement processing of the moving body 10 (10A, 10B, 10C) is executed based on the set mode. As a result, the communication system 1 (1A, 1B, 1C) allows the moving object 10 (10A, 10B, 10C) to switch between the autonomous movement mode and the manual operation mode according to the switching request transmitted from the display system. , movement control of the moving bodies 10 (10A, 10B, 10C) can be performed according to the user's request.

● Supplement ●
Each function of the embodiments described above may be implemented by one or more processing circuits. Here, the "processing circuit" in this embodiment means a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, and a processor designed to perform each function described above. ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit) and other devices such as conventional circuit modules .

In addition, the various tables in the embodiments described above may be generated by the learning effect of machine learning. may Here, machine learning is a technology that allows a computer to acquire human-like learning ability, and the computer autonomously generates algorithms necessary for judgment such as data identification from learning data taken in advance. It is a technology that makes predictions by applying this to new data. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a learning method combining these learning methods. Any learning method for

A display system, a communication system, a display control method, and a program according to one embodiment of the present invention have been described so far. Alternatively, deletion or the like can be changed within the range that a person skilled in the art can conceive, and as long as the action and effect of the present invention are exhibited in any aspect, it is included in the scope of the present invention.

1, 1A, 1B, 1C communication system 100 communication network 10, 10A, 10B, 10C moving bodies 30, 30A, 30B, 30C control device 31 transmitting/receiving unit (an example of switching request receiving means)
37 self-position estimation unit (an example of self-position estimation means)
38 Route information generation unit 42 Mode setting unit (an example of mode setting means)
43 Autonomous movement processing unit (an example of movement processing means)
44 manual operation processing unit (an example of movement processing means)
45 Accuracy calculator 46 Image generator (an example of image generator)
47 learning units 50, 50A, 50B display device 51 transmission/reception unit (an example of receiving means, an example of movement route transmission means, an example of acquisition means, an example of switching request transmission means)
52 Reception unit (an example of movement route reception means)
53 Display control unit (an example of display control means)
56 Accuracy calculator 57 Image generator (an example of image generator)
90 Information processing device 91 Transmission/reception unit 93 Accuracy calculation unit 94 Image generation unit (an example of image generation means)
200 Route input screen 250 Destination series 400, 400A, 400B, 400C, 400D, 400E, 400F Operation screen 600 Map display image areas 611, 613, 615 Series images 650, 660 Accuracy display image 700 Shooting display image areas 711, 713, 715 Route images 750, 760 Accuracy display image 800 Notification information display area 900 Mode switching button (an example of operation accepting means)

Claims (16)

A display system for displaying an image captured by a mobile object moving within a predetermined base,
a receiving means for receiving a photographed image of the base, which is transmitted from the mobile object;
display control means for superimposing and displaying a virtual route image on the moving route of the mobile object at the base shown in the received photographed image;
display system. 2. The display system according to claim 1, wherein said route image includes an image showing a plurality of points on said movement route. 3. The display system according to claim 1, wherein the route image includes an image showing a movement history of the mobile object and a future destination of the mobile object. A display system according to any one of claims 1 to 3, further comprising:
a moving route receiving means for receiving an input of route information indicating a moving route of the moving object;
a movement route transmission means for transmitting the route information for which the input has been received to the moving object;
A display system for moving the mobile object based on the transmitted route information. The movement route reception means receives the route information input on a map image showing the base,
5. The display system according to claim 4, wherein said display control means displays a map image superimposed with a sequence image indicating said route information together with said photographed image superimposed with said route image. A display system according to any one of claims 1 to 5, further comprising:
A display system comprising an operation receiving means for receiving an operation for performing predetermined control on the moving object. 7. The display system according to claim 6, wherein said operation receiving means receives a switching operation for switching between a manual operation mode in which said moving body is moved by manual operation and an autonomous movement mode in which said moving body is moved by autonomous movement. 8. The display system of claim 7, further comprising:
switching request transmission means for transmitting a switching request requesting switching between the autonomous movement mode and the manual operation mode to the moving object when the switching operation is accepted;
A display system that controls switching of the mobile body between the autonomous movement mode and the manual operation mode based on the transmitted switching request. The autonomous movement is learning-based autonomous movement,
9. The display system according to claim 7, wherein learning for the autonomous movement can be performed when the moving body switches from the autonomous movement mode to the manual operation mode. 10. The display system according to claim 9, wherein learning for said autonomous movement is performed using a photographed image photographed by said moving object. A communication system comprising: a display system for displaying an image captured by a mobile object moving within a predetermined location; and the mobile object,
Acquisition means for acquiring a photographed image in which the base is photographed;
display control means for superimposing and displaying a virtual route image on the moving route of the mobile object at the base shown in the received photographed image;
communication system. 12. The communication system of claim 11, further comprising:
An image generating means for generating a display image in which the route image is superimposed on the captured image based on position information indicating the current position of the moving object and route information indicating the moving route of the moving object,
The display control means is a communication system for displaying the generated display image. A communication system according to claim 12,
The moving body further
A self-position estimation means for estimating the current position of the mobile body,
The communication system, wherein the image generating means generates the display image based on the position information indicating the estimated current position and the route information. A communication system according to any one of claims 11 to 13, further comprising:
The moving body is
a switching request receiving means for receiving a switching request transmitted from the display system requesting switching between a manual operation mode for manually moving the moving body and an autonomous movement mode for autonomously moving the moving body;
mode setting means for setting either the autonomous movement mode or the manual operation mode based on the received switching request;
movement processing means for executing movement processing of the moving object based on the set mode;
communication system. A display control method executed by a display system for displaying an image captured by a mobile object moving within a predetermined base,
a receiving step of receiving a photographed image of the base, which is transmitted from the mobile object;
a display control step of superimposing and displaying a virtual route image on the movement route of the mobile object at the base shown in the received photographed image;
A display control method that performs A display system that displays an image taken by a mobile object moving within a predetermined base,
a receiving step of receiving a photographed image of the base, which is transmitted from the mobile object;
a display control step of superimposing and displaying a virtual route image on the movement route of the mobile object at the base shown in the received photographed image;
program to run.

Images