JPH11104984A - Real environment information display device and recording medium in which program for executing real environment information display process is recorded and which can be read by computer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to pertinently and easily grasp the state of real environment by displaying a three dimensional image of the real environment in which the visual point, time, range, etc., which the user desires to see, are modelled. SOLUTION: A real robot 12 acts in a real environment 10 to be able to recognize an external environment by a sensor. A virtual environment model generating part 16 generates a three dimensional virtual environmental model 18 modelling the real environment 10. A simulator 24 simulates the sensor function which is equivalent to that of the real robot 12 in the virtual environmental model 18 to generate a virtual robot 20, and allows the virtual robot to act in synchronism with the real robot 12. A difference detecting device 26 compares sensor information on the virtual robot by the simulator 24 to sensor information on the real robot to detect a difference, and a virtual environmental model renewal part 28 renews (generates, corrects, eliminates) the virtual environmental model 18 so that the difference between the sensor information is eliminated. A monitor processing part 30 displays the virtual environmental model 18 of optional visual point and time to the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to generation, correction, and deletion of an object on a real robot arranged in a real environment and a virtual robot arranged in a virtual environment model in accordance with a difference in sensor information between the real robot and the virtual robot. The present invention relates to a real environment information display device and a computer readable recording medium storing a program for executing a real environment information display process.

[0002]

2. Description of the Related Art In the near future, as robots spread to workplaces and homes, it is indispensable to have a technology that can adapt to a constantly changing environment such as a real environment. In this context, it is also important how to inform the user of the information of the real environment. For example, when it is desired to remotely monitor a home or office for security, it is useful to be able to easily view information at any time and place in a monitoring environment.

Also, by using an answering machine, the status of the house can be checked from any terminal, such as when the user forgets something at the house or office, and the robot can be requested to do work. In addition, it is possible to monitor in real time the status of an environment where humans cannot enter in extreme working spaces such as a nuclear waste disposal site, the deep sea, and outer space. Furthermore, such a technique can also be effectively used for a robot to help a home in an aging society.

As a method of presenting information of the current real environment to a user, there is a method of presenting an actual image or a planar map from an ITV camera. Also, as a device for a remote device of a robot in outer space, a simulated image generated by simulation is compared with a real image taken by an ITV camera, and either the simulated image or the real image is matched so that the two images match. There is also an apparatus in which one of them is modified (JP-A-3-55194 and JP-A-3-86484).

[0005]

However, the ITV
For displaying the real environment using camera images, use the ITV
The range that can be photographed by the camera is limited, and it is difficult to grasp the entire image of the real environment such as an office, a house, and a facility to be processed. In addition, ITV cameras can only be installed adjacent to physically existing objects such as walls, and cannot be freely arranged at any position in the space for constructing the real environment. Only a very limited real environment, such as a part of an empty room, can be displayed.

[0006] A planar map is convenient for grasping the whole image of the real environment, but it is naturally difficult to represent three-dimensional information, and it is difficult to grasp the situation when looking at the monitor display. Hard to do. Furthermore, it is difficult to update the map once created in real time as the environment changes,
It cannot be modified until there is a user input such as a layout change, and the map cannot represent the latest information.

On the other hand, when the video displayed on the monitor is a real image such as a camera image, the load of data transfer by the network is large, and it is difficult to transmit the video in real time. in this way,
At present, there is no real-world display device with a high degree of freedom that allows the user to easily see the situation from a desired viewpoint at a desired time in a changing real environment that changes every moment.

Similarly, a display device for remote monitoring and remote operation is mainly a device using an actual image of an ITV camera, and a pseudo image by simulation is also used. The real environment is a limited space that can be photographed by an ITV camera, and there is no device that allows the user to easily see the status of the real environment from any viewpoint at any time.

The present invention has been made in view of such conventional problems, and displays a three-dimensional image of a real environment modeled with respect to a viewpoint, a time, a range, and the like that a user wants to view, and appropriately and easily displays the three-dimensional image. It is another object of the present invention to provide a real environment information display device capable of grasping the status of a real environment and a computer readable recording medium storing a program for executing a real environment information display process.

[0010]

FIG. 1 is a diagram illustrating the principle of the present invention. The real environment display device of the present invention includes a real robot 12, a virtual environment model generation unit 16, a simulator 24, a difference detector 26, a virtual environment model update unit 28, and a monitor processing unit 30, as shown in FIG. Is done. The real robot 12 is placed in the real environment 10 and behaves, and can recognize the outside world with a sensor. The virtual environment model generation unit generates a virtual environment model 18 that models a real environment or a virtual environment model 18 that is modeled in three dimensions. The simulator 24 generates a virtual robot 20 by simulating a sensor function equivalent to that of the real robot 12 in the virtual environment model 18 and causes the virtual robot 20 to act in synchronization with the real robot 12.

The difference detector 26 compares the sensor information of the virtual robot 20 by the simulator 24 with the sensor information of the real robot 12 to detect a difference. When the difference detector detects a difference in sensor information, the virtual environment model updating unit updates the virtual environment model so as to eliminate the difference in sensor information. The monitor processing unit 30 outputs and displays the virtual environment model 18 to the user.

The real robot 12 recognizes the outside world by a plurality of sensors, arranges a plurality of the real robots 12 in the real environment 10, and uses the simulator 24 to
Can be generated and can be made to act in synchronization with each real robot 10. The virtual environment model updating unit 28 performs the following processing based on the difference in the sensor information. When an object in a real space that does not exist in the virtual space is recognized based on a difference in sensor information, a virtual object corresponding to the virtual space is newly generated.

When an object that exists in the virtual space but does not exist in the real space is recognized from the difference in sensor information, the corresponding virtual object in the virtual space is deleted. The monitor processing unit 30 includes a storage device 36 that stores information of the virtual environment model 18 before updating in a time-series manner every time the virtual environment model updating unit 28 performs an update. Update time information can be extracted and displayed.

As described above, according to the real environment display device of the present invention, the real robot 12 moving in the real environment 10 and the virtual robot 20 moving on the virtual environment model 18 are linked to each other and sent from the respective robots 12 and 20. If a difference is detected in the coming sensor information, the virtual environment is updated so as to be equal to the real environment 10, the real environment 10 that changes every moment is reflected on the virtual environment model 18, and this is displayed on the monitor. Therefore, the user can easily see the situation of the real environment at a desired time and from a desired viewpoint.

The monitor processing unit 30 applies model information other than visible information to the virtual environment model 18 initially generated by the virtual environment model
Behavior information and the like based on the route plan can be visualized and added. Furthermore, the real environment information display device of the present invention is:
The real robot 12, the virtual environment model generation unit 16, the simulator 24, the difference detector 26, the virtual model update unit 28, and the monitor processing unit 30 can be mounted on a plurality of computers connected on a computer network.

Further, the present invention provides a computer-readable recording medium on which a program for executing a real environment information display process is recorded, and a virtual environment model generating module for generating a virtual environment model which models a real environment. A simulator module that generates a virtual robot by simulating in a virtual environment model a sensor function equivalent to a real robot that behaves in a real environment and that can recognize the outside world with a sensor and act in synchronization with the real robot. A difference detection module that detects the difference by comparing the sensor information of the virtual robot and the sensor information of the real robot by the simulator module, and eliminates the difference in the sensor information when the difference detection module detects the difference in the sensor information. Update the virtual environment model A module, characterized by comprising a monitor processing module to output display the virtual environment model, a.

[0017]

FIG. 2 is a block diagram showing an embodiment of a real environment information display apparatus according to the present invention. In FIG. 2, a mobile robot 12 is disposed in a real environment 10 to be displayed by the present invention. The mobile robot 12 includes at least one or more sensors capable of recognizing the outside world, and can freely move inside the real environment 10. A virtual environment system 14 is constructed for the real environment 10.

The virtual environment system 14 includes a virtual environment model generation unit 16, a layout information storage unit 21, a virtual environment database 22, a simulator 24, a difference detector 26, a virtual environment model update unit 28, a monitor processing unit 30, and a storage device 36. Prepare. The monitor 32 is connected to the monitor processing unit 30.
Is connected, so that the user 34 can view the virtual environment model from any viewpoint and time.

The virtual environment model generation unit 16 generates a three-dimensional virtual environment model 18 modeled corresponding to the real environment 10 based on the three-dimensional layout information such as CAD stored in the layout information storage unit 21. Simulator 24
Generates a virtual robot 20 by simulating a sensor function equivalent to that of the mobile robot 12 in the real environment 10 in the virtual environment model 18, and causes the virtual robot 20 to act in synchronization with the mobile robot 12.

The difference detector 26 compares the sensor information of the virtual robot 20 by the simulator 24 with the sensor information of the mobile robot 12 in the real environment 10 to detect a difference between the two sensor information. When the difference detector 26 detects a difference in sensor information, the virtual environment model update unit 28 generates and deletes an object in the virtual environment model 18 so as to eliminate the difference in sensor information.

Further, the monitor processing unit 30 displays the virtual environment 18 on the monitor 32 based on an arbitrary viewpoint and time designated by the user 34. The monitor processing unit 30 stores the virtual environment model 18 before the update in the storage device 36 sequentially each time the virtual environment model update unit 28 updates the virtual environment model 18. Therefore, the user 34
An arbitrary time can be designated and the virtual environment model at that time can be viewed on the monitor 32 from an arbitrary viewpoint.

FIG. 3 is a block diagram of the mobile robot 12 arranged in the real environment 10 of FIG. Mobile robot 12
Has a plurality of types of sensors for the controller 37. In this embodiment, the range finder 3
8, an ultrasonic sensor 40, an infrared sensor 42, a contact sensor 44, a gyro 45, and an encoder 46 are provided.
The range finder 38 is known as a scanning type distance measuring device, and can detect three-dimensional information of an object by measuring a distance while optically scanning an arbitrary object. The ultrasonic sensor 46 can measure the distance from the ultrasonic wave propagation time to the object.

The infrared sensor 42 can detect a heat ray radiated from a human body or the like. The contact sensor 44 detects physical contact with the object. The gyro 45 detects the moving direction of the mobile robot 12. The encoder 46 detects the traveling distance of the mobile robot 12. Further, a communication interface 48 for exchanging commands, data, and the like with the outside is provided for the controller 37.

Further, a motor drive unit 50 for driving the moving motor and an actuator drive unit 52 for driving an appropriate actuator such as a robot hand are provided. As such a mobile robot 12 arranged in the real environment 10 of the present invention, for example, NO-MAD INC. Can be used.

FIG. 4 shows the configuration of a program module for realizing the real environment information display device of the present invention shown in FIG. In FIG. 4, a virtual environment simulator module 24A is provided on the virtual environment model 18 side, and a virtual robot simulator module 20A, a sensor module driver module 24B, and a motor driver module 20C are provided for the virtual environment simulator module 24A.

A virtual environment database 22A for constructing the virtual environment model 18 is provided, and a 3D monitor processing module 30A for monitoring and displaying the virtual environment model 18 is provided. For such a program module on the virtual environment model 18 side, a robot module 12A is provided corresponding to the mobile robot 12 in the real environment. A robot knowledge database 60 and a robot path planning module 62 are provided corresponding to the robot module 12A. The difference between the sensor information of the robot module 12A and the sensor information of the virtual environment model 18 is detected by the difference detection module 28A.

An instruction from the user can be given by the user command interface module 56.
Further, an object management module 58 is provided, and performs processing relating to objects created / deleted in the virtual environment model 18. All of such program modules are connected to the command transfer module 54, and the command transfer module 54 transfers commands between the respective program modules, and as shown in FIG. System 14 and monitor 32
A real environment information display device equipped with

FIG. 5 is a flowchart of the overall processing operation in the real environment information display device shown in FIG. In the process of the real environment information display device of the present invention, first, in step S1, a process of generating a three-dimensional virtual environment model that models a part of the external world such as a facility, a building, or a room to be displayed is performed. Do. Subsequently, the process proceeds to step S2, where a virtual robot is generated in the virtual environment model 18 by the sensor function corresponding to the mobile robot 12 in the real environment 10, and the virtual robot 20 is moved in the virtual environment model 18 in conjunction with the mobile robot 12. Then, a simulation process for realizing the same sensor function as that of the mobile robot 12 is executed. In this virtual robot simulation process, in step S3, the sensor information of the mobile robot 12 and the virtual robot 2
0 sensor information is compared.

When the difference between the sensor information of the mobile robot 12 and the sensor information of the virtual robot 20 is detected in step S4, the process proceeds to step S5, in which the virtual environment model 18 is modified and updated so as to eliminate the difference in sensor information. Subsequently, in step S6, it is checked whether or not there is a display request from the user 34. If there is a display request, in step S7, information of the virtual environment model 18 is read out and monitored based on an arbitrary viewpoint and time designated by the user. 32 is displayed.

Subsequently, in step S8, it is checked whether or not there is an end instruction to the apparatus.
The processing from step 2 is repeated. If there is an end instruction, a predetermined end process is performed in step S9, and then a series of processes is ended. FIG. 6 shows details of the processing for generating a three-dimensional virtual environment model in step S1 of FIG. FIG. 6 (A)
FIG. 3 is a block diagram of a function of generating a virtual environment model using three-dimensional CAD. The construction of the three-dimensional virtual environment model by the three-dimensional CAD is performed by the solid modulator 64, the object modeler 66, and the object layouter-68.
Done in

That is, as shown in the three-dimensional virtual model generation processing of FIG.
4 modulates the shape of the object based on the three-dimensional CAD information of the real environment. Subsequently, in step S2, the object modeler 66 sets attributes such as material, weight, and center of gravity of the object. Finally, the object layouter-68 arranges the objects in step S3, whereby a three-dimensional layout file for generating a three-dimensional virtual model can be created.

FIG. 7 shows a specific example of the three-dimensional virtual model creation processing of FIG. 6, and two-dimensional layout information (plan view) 7 showing the layout of the real environment to be displayed shown in FIG.
According to the three-dimensional CAD information given by 0, FIG.
A three-dimensional virtual environment model 18 as shown in FIG. This virtual environment model 18 takes an office layout as an example.

FIG. 8 is a simulation in which the virtual robot arranged in the virtual environment model in steps S2 to S5 of FIG. 5 is linked to the mobile robot in the real space. It is explanatory drawing of the update process which corrects. In FIG. 8, the real environment 10
The mobile robot 12 is traveling in the corridor of, and has hit the obstacle 72 placed in the corridor. On the other hand, in the virtual environment model 18, the virtual robot 20 runs along the corridor of the virtual environment in the virtual environment model 18 in conjunction with the mobile robot 12 in the real environment 10, No object corresponding to 72 exists.

Here, the virtual robot 20 arranged in the virtual environment model 18 has the same size, shape, and operation performance as the mobile robot 12 in the real environment 10.
The mobile robot 12 can move around in the same manner as the 0 mobile robot 12.
The mobile robot 12 in the real environment 10 includes, for example, a range finder 38, an ultrasonic sensor 40, and the like as shown in FIG.
Although a sensor such as an infrared sensor 42 is provided, the virtual robot 20 of the virtual environment model 18 also has the same sensor functions such as a range finder and an ultrasonic sensor.

In the simulator 24 shown in FIG. 2, the virtual robot 20 is connected to the mobile robot 12 in the real environment 10.
It simulates physical phenomena so that you can act like. That is, when the mobile robot 12 in the real environment 10 hits an object such as a wall and stops, or when a measurement result of sensor information is obtained by various sensors, the virtual robot 20 of the virtual environment model 18 similarly stops or Measure and send back.

In order to enable the virtual robot 20 to act in the virtual environment model 18 in the same manner as the real robot 12 in the real environment 10 as shown in FIG. Is created, attribute information such as weight, center of gravity, material, mirror coefficient, and reflectance is added to the object in the virtual environment model 18 in addition to the size and shape.

As shown on the left side of FIG. 8, the mobile robot 12 in the real environment 10 and the virtual robot 20 in the virtual environment model 18
Are linked, the sensor information sent from each of the robots 12 and 20 is compared, and when a difference in the sensor information is detected, the virtual environment model 18 is corrected so that the virtual environment model 18 is made equal to the real environment 10. . For example, in FIG. 8, the mobile robot 12 is approaching the obstacle 72 in the real environment 10, and the virtual robot 20 of the virtual environment model 18 is also moving in conjunction therewith.

As the robots 12 and 20 move, a plurality of types of sensor information are returned. When the sensor information is compared, the information on the obstacle 72 is included in the sensor information of the mobile robot 12 in the real environment 10. ing. On the other hand, the sensor information from the virtual robot 20 does not include information on the obstacle 72, and a difference in the sensor information is detected.

When the difference in the sensor information is detected as described above, the virtual environment model 18 is set to be equal to the real environment 10.
To correct. That is, the real environment 1 is included in the virtual environment model 18.
The virtual environment model 18 is corrected and updated so as to generate a virtual obstacle 72A corresponding to the 0 obstacle 72. The correction and update of the virtual environment model 18 by detecting the difference in sensor information between the mobile robot 12 and the virtual robot 20 include adding a new object to the virtual environment model 18 and deleting an unnecessary object from the virtual environment model 18. Yes, each is performed as follows.

Addition of Object When the object existing in the virtual environment model 18 does not exist in the real environment 10, the virtual environment model 18
A corresponding object is searched from the virtual environment database 22 based on information such as the center coordinates and radius of the object, and the searched object is deleted from the virtual environment database 22.

Addition of Object When an object that does not exist in the virtual environment model 18 exists in the real environment 10, a new object is generated in the virtual environment model 18 from the center coordinates and radius of the object at the detected position. The object initially generated in the virtual environment model 18 is a prototype having a basic shape. Thereafter, the object is further investigated by a sensor to obtain more detailed information of the object, and information close to the actual object is obtained. Generated in the environment model 18.

Here, the virtual environment database 22 shown in FIG. 2 is a database of information on objects existing in the virtual environment model 18, and describes the size, shape, position, attributes, etc. of the objects. . Attributes include weight, center of gravity, material, mirror coefficient, reflectance, and the like. This virtual environment database 22 has a size obtained as sensor information,
By searching a shape or the like as key information, a corresponding object can be searched.

Such a real environment 10 and a virtual environment model 1
The process of correcting and updating the virtual environment model 18 based on the difference in the sensor information of FIG. 8 depends on the sensor capability of the mobile robot 12, and a plurality of types of sensors are provided as shown in FIG.
By integrating sensor information of a plurality of sensors, generation based on more accurate object information can be realized. The correction based on the difference detection of the virtual environment model 18 is basically performed when a difference is found between the real environment 10 and the virtual environment model 18, but the difference between any one point is inaccurate. The difference between the sensor information of the mobile robot 12 and the sensor information of the virtual robot 20 in a predetermined fixed section is accumulated, and the difference information is processed to correct the virtual environment model 18 appropriately.

Furthermore, since the difference between the real environment 10 and the virtual environment model 18 is discovered by the respective robots 12 and 20, the amount of information to be updated by one robot is limited, and the latest information is always available in all places. Is not obtained. Therefore, if a sufficient number of robots can be prepared, the latest information can be reflected in the virtual environment model 18 almost in real time in all places, so that the user can always
The latest information on the real environment at all positions can be obtained.

FIG. 9 is a flowchart of a process of updating the virtual environment model 18 based on detection of a difference in sensor information between the mobile robot 12 and the virtual robot 20 as shown in FIG. First, for an update process of detecting a difference from the real environment 10 and correcting the virtual environment model 18, the mobile robot 12 is moved in the real environment 10, and the virtual robot 20 is moved in conjunction with the movement. Need to move within.

The movement of the mobile robot 12 in the real environment 10 may be performed, for example, on a planned route identified by the robot route planning module 62 in the program module configuration shown in FIG. A route may be set so as to make a prediction and turn around there, or may move based on a movement command given by the user.

When the mobile robot 12 behaves in the real environment 10 in this manner, the virtual environment model 18
The virtual robot 20 also acts on the above, and compares the sensor information of the mobile robot 12 with the sensor information of the virtual robot 20. By comparing the sensor information, step S1 in FIG.
When a difference between the virtual environment model 18 and the real environment 10 is detected, the process proceeds to step S2, and it is checked whether the object exists in the real environment but does not exist in the virtual environment.

If the object exists in the real environment but does not exist in the virtual environment, the flow advances to step S3 to instruct a process of adding a new object to the virtual environment. In response to the instruction to add the object, in step S4, the shape of the object is generated at the detection position from, for example, the difference between the sensor information. Specifically, (Sensor information of real environment)-(Sensor information of virtual environment)>
In the case of 0, (object information) = (sensor information of real environment) − (sensor information of virtual environment) (1) is obtained, and an object is generated as object information using the difference information as it is to detect a virtual environment model. Place in position.

Subsequently, in step S5, when various attributes of the object are obtained by the various sensor information, the attribute information is added to the object, and the attribute information is placed in the virtual environment database 22 together with the newly generated object information. . On the other hand, if the object exists in the virtual environment but does not exist in the real environment in step S2, an instruction to delete the object from the virtual environment is issued in step S6. Based on this, an object to be deleted is searched from the virtual environment database in step S7. That is, (sensor information of virtual environment) − (sensor information of real environment)>
When 0 is satisfied, the size of the object to be deleted from the virtual environment is assumed as (difference information) = (sensor information of virtual environment) − (sensor information of real environment), and further obtained from the sensor information at that time. The virtual environment database 22 is searched by adding various attribute information, and a corresponding object, that is, an object to be deleted is searched, and this is deleted from the virtual environment database in step S8.

FIG. 10 shows a specific state in the virtual environment model updating process of FIG.
In the virtual environment model 18, which is the office layout information displayed above, assuming that the mobile robot 12 is moving in the real environment 10 of FIG. 10B, FIG. When the virtual environment model 18 is displayed, it can be seen that the virtual robot 20 is moving in the virtual environment model 18 corresponding to the real environment 10 as shown in FIG.

The real environment 1 as shown in FIGS.
The update processing for correcting the virtual environment so that the virtual environment model 18 is made equal to the real environment 10 as shown in the flowchart of FIG. 9 by the linked movement of the mobile robot 12 of the virtual environment model 0 and the virtual robot 20 of the virtual environment model 18. Is done,
As a result, the virtual environment model 18 comes closer to the real environment 10.

FIG. 11 schematically shows the repetition of the updating process accompanying the robot movement of the real environment 10 and the virtual environment model 18 of FIG. That is, sensor information is obtained from each of the range finder, the ultrasonic sensor, the infrared sensor, the contact sensor, the gyro, and the encoder by the linked movement of the mobile robot 12 and the virtual robot 20 in the real environment 10 and the virtual environment model 18, respectively. , The respective differences are detected.

These differences in sensor information are integrated and reflected in the virtual environment model 18 through reference to a virtual environment database provided as object information in order to generate, delete, and correct objects. As a result, the actual environment 10
The virtual environment model 18 is approaching moment by moment. FIGS. 12 and 13 show the initial model 18 of FIG. 12A by updating the virtual environment model as shown in FIG.
The state changes from A to the model 18B in FIG. 12B, the model 18C in FIG. 12C, and further to the model 18D in FIG. 13D to approach the real environment.

That is, with respect to the initial model 18A of FIG. 12A, in the model 18B of FIG.
Is detected and an object is added.
In the case of 2 (C), the difference in the environment due to the wall is detected and added to the area 76, and finally, the left side section matches the real environment due to the arrangement of the new partition wall as shown in FIG. 13 (D). Fixes have been made.

Here, the object generation, deletion,
Simulations based on sensor information for correction include simulation of sensor information itself and simulation for checking object interference. First, regarding the simulation of the sensor information, as shown in FIG. 14A, the mobile robot 12 is regarded as a cylindrical body, and for example, the object 94 is calculated by calculating the intersection with the minimum sphere 96 containing the object 94 taking a desk as an example. Recognize the coordinate center and size of. Next, as shown in FIG. 14B, a minimum rectangular parallelepiped 98 containing the object 94 is set, and an intersection with each surface of the minimum rectangular parallelepiped 98 is obtained to generate detailed information on the object 94.

FIG. 15 shows a process of checking for interference between the mobile robot 12 and the object. In this interference check, the mobile robot 12 is approximated to a cylinder as shown in FIG. 15 (A), a minimum sphere 96 including the object 94 is set, and the coordinate center of the minimum sphere 96 and the cylinder center of the robot 12 are set. After calculating the interference check by calculating the distance L between intersections,
As shown in FIG. 15B, a minimum rectangular parallelepiped 98 including the object 94 is set, an intersection with each surface of the minimum rectangular parallelepiped 98 is calculated, and the presence or absence of interference is checked from the distance to the mobile robot 12.

Further, in the updating process accompanying the movement of the robot to bring the virtual environment model 18 closer to the real environment 10,
There are object simulation such as opening / closing of a door and movement of a person, and dynamic simulation of a physical phenomenon such as collision of an object. For the object simulation, it is necessary to describe the operation for each object registered in the virtual environment database.

For example, if the door is a door, it is described that the door moves about a fulcrum. The dynamic simulation is added as a function of calculating the amount of movement or deformation when an external force is applied from an attribute model of an object in real time by taking over physical laws. Next, the display processing function of the virtual environment model on the monitor 32 by the monitor processing unit 30 provided in the virtual environment system 14 of FIG. 2 will be described. FIG. 16 is an explanatory diagram of information storage of the virtual environment model by the storage device 36 provided in the monitor processing unit 30. In this embodiment, each time the virtual environment model 18 is updated, the storage device 36 stores the pre-update virtual environment model as time information, such as the initial model 18A in FIG. 12 to the model 18D in FIG. Together with the information stored in the storage device 36.

For example, the storage device 36 stores the three-dimensional virtual model information 76-1, 76-2,.
1, t2,... Tn are stored as indices. For this reason, the user 34 designates the time information Ti of the desired time to the monitor processing unit 30 so that the storage device 3
6, the virtual environment model information 76-i corresponding to an arbitrary specified time Ts can be read out and displayed on the monitor 32 as information of the real environment viewed from an arbitrary viewpoint specified at that time.

That is, when an arbitrary designated time Ts is set,
The specified time Ts is compared with the time information t1 to tn. For example, if ti <Ts <ti + 1, the virtual environment model information 76-i is read and displayed. Thereby, the user 34 can always see the real environment information of a favorite time and a viewpoint as needed. For example, when the present invention is applied to a security system such as theft monitoring or the like, the real environment information is viewed from the log information of the occurrence of the abnormality to find the time of the occurrence of the abnormality, and the virtual environment model information at that time is stored in the storage device 36. By reading out and displaying the information, the state of the real environment at the time of occurrence of an abnormality can be checked later.

FIG. 17 shows another embodiment of the monitor processing section 30 shown in FIG. 2. In this embodiment, in addition to the display of the real environment by the basic three-dimensional virtual environment model, the monitor is executed by the user. It is characterized in that information that is easy to understand can be additionally added corresponding to the task. FIG.
7, the monitor basic unit 30-1 filters the visible information corresponding to the virtual environment model on a one-to-one basis by the filter 78 with respect to the current virtual environment model 18 or the virtual environment model 18 stored as information in the storage device 36. Generated and displayed on monitor 32. Such a monitor basic unit 30-1
On the other hand, in this embodiment, the monitor adding unit 30-2
And a monitor addition unit 30-3.

The monitor adding unit 30-2 generates attribute information of an object arranged in the virtual environment model 18 by using the filter 80, for example, weight and center of gravity, and generates numerical information indicating the weight and center of gravity by the converter 82. Monitor 3
2 is displayed in correspondence with the second object. The monitor adding unit 30-3 includes, for example, the robot path planning module 62
And the landmark to be referred to is extracted by the filter 86, and the converter 88
Is converted to visible information and displayed on the monitor 32.

FIG. 18 shows the monitor basic unit 30-1 of FIG.
It is a specific example of the display processing by the monitor addition units 30-2 and 30-3. First, the monitor basic unit 30-1 converts the virtual environment model 18 at that time into visible information and displays it on the monitor 32. The monitor adding unit 30-2 determines, for example, the object 94, which is a desk in the virtual environment model 18,
The weight and the center of gravity are extracted as the attribute information 95, and the monitor 3
2, the position of the center of gravity is displayed as “weight 10 kg” as in an object 94 shown enlarged below.

Further, the monitor adding unit 30-3 extracts the path information 90a and the landmark information 92a included in the code information 84 obtained as the two-dimensional information from the robot path planning module 62. The information is converted into display information, and the route information 90 and the landmark 92 are additionally displayed on the monitor 32. As described above, since the virtual environment model 18 generated by the present invention is electronic information on a computer, any information required by the user can be easily and arbitrarily added as needed by the user.

FIG. 19 shows another embodiment of the real environment information display apparatus according to the present invention, wherein the processing functions of each unit shown in FIG. 2 are implemented on a plurality of computers connected on a computer network. I do. In FIG. 19, a layout information storage unit 21, a virtual environment model generation unit 16, and a virtual environment database 22 are arranged in a computer 100A.
A simulator 24 is arranged in the computer 100B, a difference detector 96 is arranged in the computer 100C, a computer 100D is arranged in the virtual environment model updating unit 28, and a monitor processing unit 30, a monitor 32 and a storage device are arranged in the computer 100E. 36 becomes the user terminal on which the real environment 10 is located.
Of mobile robots 12 are also arranged as independent computer devices.

The computers 100A to 100E and the mobile robot 12 in the real environment 10 are connected to a computer network 102,
2, the same functions as those of the apparatus configuration of FIG. 2 can be realized. FIG. 20 shows a plurality of computers 100A to 100E in the computer network 102 of FIG.
4 is a program configuration in an embodiment in which the mobile robot 12 and the mobile robot 12 are connected. In FIG. 20, the program module itself is the same as that of FIG. 4, except for the instruction passing module 54 in this embodiment,
Each module and database are connected through a network by, for example, a socks interface 104.

According to the real environment information display device of the present invention using such a network, the program modules and the database shown in FIG. 20 for constructing the real environment information display device are distributed to a plurality of machines and connected to the network. In addition, a real environment can be monitored based on a virtual environment model that moves a robot from a remote location or is updated at a remote location.

More specifically, it can be realized by a program configuration shown in FIG. 20 and a mechanism using, for example, JAVA or VRML for viewing information using various platforms through a network using the socks interface 104. it can. Especially JAVA
By using a language that supports multi-platforms such as and, it can be scanned from any machine independent of the platform,
For example, it is possible to operate the real environment information by remote control from a place where the user is away from home.

More specifically, when the real environment information display device of the present invention is constructed using a multi-platform language such as JAVA or VRML through a network, the format used in the field of CAD, for example, GXF or IGES Is modeled, and then an attribute is attached to the object to construct a format specific to the virtual environment model of the present invention. This format may be further converted into a language compatible with multi-platforms, for example, VRML, and simulation may be performed by the simulator of the present invention based on this data. FIG. 21 is an embodiment of a computer-readable recording medium on which a program for executing the real environment information display processing of the present invention is recorded.
A removable portable storage medium such as a ROM or a floppy disk, a storage device of a program provider that provides a program via a line, and a memory device such as a RAM or a hard disk of a processing device in which the program is installed. Also, the program provided by the recording medium,
It is loaded into the processing unit and executed on its main memory.

[0070]

As described above, according to the present invention, the change of the real environment due to the movement of the robot is reflected in the virtual environment model created as a three-dimensional model and monitored, so that the viewpoint desired by the user can be seen. In this case, the user can display the actual environment situation in a desired range, and can display an optimal three-dimensional image of the actual environment according to the user's request. Also, by automatically reflecting the result in a three-dimensional virtual environment model in real time in response to a change in the real environment, the user can always see the latest information of the real environment.

Further, by using a virtual environment model, IT
Additional information necessary for grasping a situation that cannot be seen with a V-camera or the like, such as robot path information and landmarks, can be visualized and added as information, thereby recognizing the real environment when viewing the monitor. Can be performed more appropriately and quickly. Further, since the file information of the three-dimensional virtual model to be monitored is digital information, it is easy to perform processing such as searching and editing. For example, by converting three-dimensional information into two-dimensional information, it is possible to intuitively change the situation. Can be caught.

Further, by storing information of a three-dimensional virtual environment model that is updated in accordance with a change in the real environment for each update, the user can specify not only the real time but also the past real time at an arbitrary time. You can easily see environmental information. Furthermore, the load on each computer can be reduced by realizing the device configuration by networking and distributing and distributing it to multiple computers. Furthermore, it is possible to easily grasp the state of the real environment via a network from a remote location, and Operations can be performed.

Further, when operating through a network, only the information that has changed in the three-dimensional virtual environment model needs to be sent, so that the load on the network can be reduced.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a block diagram of an embodiment of the present invention.

FIG. 3 is a block diagram of a mobile robot used in the present invention.

FIG. 4 is an explanatory diagram of a program configuration for realizing the device functions of FIG.

FIG. 5 is a flowchart of a real environment display process of FIG. 2;

FIG. 6 is an explanatory diagram of a three-dimensional virtual model generation process of the present invention.

FIG. 7 is a specific explanatory diagram of the three-dimensional model generation processing of FIG. 5 using CAD.

FIG. 8 is an explanatory diagram of a process of updating a virtual environment model accompanying a robot action according to the present invention.

FIG. 9 is a flowchart of a virtual environment model update process according to the present invention.

FIG. 10 is an explanatory diagram of a virtual environment and a real environment of the present invention viewed on a monitor.

FIG. 11 is an explanatory diagram schematically showing the virtual environment model updating process of FIG. 9;

FIG. 12 is an explanatory diagram showing a change in a virtual environment due to a virtual environment model update process.

FIG. 13 is an explanatory diagram showing a change in the virtual environment due to the virtual environment model update process (continued)

FIG. 14 is an explanatory diagram of a simulation of the present invention for recognizing an object based on sensor information.

FIG. 15 is an explanatory diagram of a simulation of the present invention for checking interference between a robot and an object.

FIG. 16 is an explanatory diagram of storage processing of a virtual environment model in the monitor processing of the present invention.

FIG. 17 is an explanatory diagram of a display process of an additional function in the monitor process of the present invention.

FIG. 18 is an explanatory diagram of additional information display processing of object attributes, landmarks, and route information in the monitor processing of the present invention.

FIG. 19 is a block diagram of another embodiment of the present invention constructed via a network.

FIG. 20 is an explanatory diagram of a program configuration for realizing the device functions of FIG. 19 constructed via a network.

FIG. 21 is an illustration of an embodiment of a computer-readable recording medium on which a program for executing the real environment information display processing of the present invention is recorded.

[Explanation of symbols]

 10: Real environment 12: Mobile robot (real robot) 14: Virtual environment system 16: Virtual environment model generator 18: Virtual environment model 20: Virtual robot 20A: Virtual robot simulator module 20B: Sensor driver module 20C: Motor driver module 21 : Layout information storage unit 22: virtual environment database 24: simulator 24 A: virtual environment simulator module 26: difference detector 26 A: difference detection module 28: virtual environment model update unit 30: monitor processing unit 30-1: monitor basic unit 30- 2, 3: monitor adding unit 30A: monitor processing module 32: monitor 34: user 36: storage device 37: controller 38: range finder (scanning distance measuring device) 40: ultrasonic sensor 42: infrared sensor 44: contact Sensor 45: Gyro (azimuth sensor) 46: Encoder (distance sensor) 48: Communication interface 50: Moving motor drive unit 52: Actuator drive unit 54: Command transfer module 56: User command interface module 58: Object management module 60: Robot knowledge Database 62: Robot path planning module 64: Solid model modulator 66: Object modeler 68: Object layouter 70: Two-dimensional layout information (drawing information) 72: Obstacle 72A: Virtual obstacle 76-1 to 76-n: Three-dimensional Virtual model information 80, 86: Filter 82, 88: Converter 84: Behavior information 90: Route information 92: Landmark 94: Object (desk) 95: Attribute information 96: Minimum sphere 98: Minimum rectangular Body 100A to 100D: Computer 102: Computer Network 104: Socks Interface

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Hiroyuki Okabe 4-1-1, Kamiodanaka, Nakahara-ku, Kawasaki-shi, Kanagawa Prefecture Inside Fujitsu Limited (72) Inventor Nobuo Watanabe 4-1-1, Kamiodanaka, Nakahara-ku, Kawasaki-shi, Kanagawa No. 1 Inside Fujitsu Limited

Claims (6)

[Claims] 1. A real robot arranged and acting in a real environment and capable of recognizing the outside world with a sensor, a virtual environment model generating unit for generating a virtual environment model that models the real environment, A simulator that generates a virtual robot by simulating a sensor function equivalent to that of the real robot and acts in synchronization with the real robot, and compares sensor information of the virtual robot and sensor information of the real robot by the simulator. And a difference detector for detecting a difference, and when the difference detector detects a difference in sensor information,
A real environment information display device comprising: a virtual environment model update unit that updates the virtual environment model so as to eliminate the difference in the sensor information; and a monitor unit that displays the virtual environment model. 2. The real environment information display device according to claim 1, wherein the real robot recognizes the outside world by a plurality of sensors, arranges the plurality of real robots in the real environment, and sets a plurality of the real robots by the simulator. A real environment information display device characterized in that virtual robots are generated and act in synchronization with each real robot. 3. The real environment information display device according to claim 1, wherein the virtual environment model updating unit newly adds the virtual space to the virtual space when recognizing an object in the real space that does not exist in the virtual space due to a difference in sensor information. A virtual object corresponding to the virtual environment, and when an object existing in the virtual space but not in the real space is recognized based on a difference in sensor information, the virtual object in the corresponding virtual space is deleted. apparatus. 4. The real environment information display device according to claim 1, wherein the monitor processing unit chronologically updates the information of the virtual environment model before the update every time the virtual environment model updating unit updates the virtual environment model. A real environment information display device having a storage device for storing, and extracting information of an arbitrary update time from the storage device and displaying the information. 5. The real environment information display device according to claim 1, wherein the monitor processing unit performs processing on the virtual environment model initially generated by the virtual environment model generation unit, except for visible information, as necessary. A real environment information display device for visualizing and adding model information, action information based on a path plan of the real robot, and the like. 6. A virtual environment model generation module for generating a virtual environment model that models the real environment, and a sensor function equivalent to that of a real robot that is placed in the real environment and acts and can recognize the outside world with a sensor. A simulator module that generates a virtual robot by simulating in a model and acts in synchronization with the real robot, and compares the sensor information of the virtual robot and the sensor information of the real robot by the simulator module to determine differences. A difference detection module for detecting, a difference between the sensor information detected by the difference detection module, a virtual environment model update module that updates the virtual environment model so as to eliminate the difference in the sensor information, and the virtual environment model And a monitor processing module for displaying. A computer-readable storage medium storing a program for executing a real environment information display process.