JP2023143222A - Information processing system and information processing method - Google Patents

Abstract

To provide an information processing system which allows a user to intuitively easily understand a learning state of a world model.SOLUTION: An information processing system comprises: a designation unit which designates a learned world model, environmental data about environment in which an agent operates, and a learned strategy model which specifies operation of the agent; a strategy execution unit which sequentially executes strategy in the environment, on the basis of the environmental data and the strategy model; an observation image processing unit which captures the environment by a virtual camera for each execution of the strategy, and generates a plurality of observation images; a latent state processing unit which compresses each of the plurality of observation images to derive each of a plurality of latent states, on the basis of the world model; and a display unit which displays the plurality of latent states.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to an information processing system and an information processing method.

In recent years, in the field of visual-based reinforcement learning, research on world models, which construct models of the environment so that agents can predict the results of their actions, has been progressing. In other words, the world model is a reinforcement learning method that learns a predictive model of the state transition of the environment. Regarding world models, for example, it is known that an agent learns a world model based on image input and actions (see Non-Patent Document 1).

Google AI Blog, The latest from Google Research, “Introducing PlaNet: A Deep Planning Network for Reinforcement Learning”,＜URL:https://webbigdata.jp/ai/post-2867＞

In research and development of world models, engineers are required to check how agents' world models are learned. However, because the world model is expressed as vector data, it is difficult for engineers to understand it intuitively. Furthermore, it is difficult to interactively check the restored image or the state of the world model in response to changes in time, changes in observation due to behavior, or changes in the observation position. Furthermore, conventionally, there is no way to check whether the learning results of the world model match the engineers' assumptions, that is, whether the learning results are good or bad.

The present disclosure provides an information processing system and an information processing method that make it easy to intuitively understand the learning state of a world model.

One aspect of the present disclosure includes a specification unit that specifies a learned world model, environmental data regarding an environment in which an agent operates, and a learned policy model that defines the behavior of the agent; a policy execution unit that sequentially executes policies in the environment based on a policy model; an observation image processing unit that images the environment with a virtual camera and generates a plurality of observation images each time the policy is executed; An information processing system comprising: a latent state processing unit that compresses each of the plurality of observed images and derives each of the plurality of latent states based on a world model; and a display unit that displays the plurality of latent states. be.

One aspect of the present disclosure provides a step of specifying a learned world model, environmental data regarding an environment in which an agent operates, and a learned policy model that defines the behavior of the agent, and the step of specifying the environmental data and the policy. sequentially executing strategies in the environment based on the model; capturing an image of the environment with a virtual camera each time the strategy is executed to generate a plurality of observation images; and based on the world model, The information processing method includes the steps of compressing each of the plurality of observed images to derive each of the plurality of latent states, and displaying the plurality of latent states on a display unit.

According to the present disclosure, the learning state of the world model can be easily understood intuitively.

Block diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure Block diagram showing an example of the configuration of the GUI section Block diagram showing an example of the configuration of the simulator section Diagram showing an example of a list screen Diagram showing an example of the model setting screen Diagram showing an example of the model settings screen with path names displayed Diagram showing an example of a camera setting screen Diagram showing an example of a camera setting screen when multiple virtual cameras are provided Diagram showing an example of latent state mapping Sequence diagram showing an example of the operation of the information processing system

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of well-known matters or redundant explanations of substantially the same configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

For example, the term "unit" or "device" used in the embodiments is not limited to a physical configuration mechanically realized by hardware, but also includes one whose functions are realized by software such as a program. Moreover, the functions of one configuration may be realized by two or more physical configurations, or the functions of two or more configurations may be realized by, for example, one physical configuration.

(Explanation of terms)
This embodiment deals with reinforcement learning such as a world model. First, we will explain terms related to reinforcement learning and world models.

"Reinforcement learning" is a machine learning method in which an agent learns tasks by repeatedly interacting with a dynamic environment through trial and error. Agents earn rewards for evaluating the goodness of their current state. The task is, for example, a robot device acting as an agent grasping an object. When a task is accomplished, you get a reward. Reinforcement learning learns the policy that yields the most rewards through a series of actions.

An "agent" is a subject that interacts with the environment and learns strategies. As a specific example, the agent is an AI (Artificial Intelligence), a robot device, or a controller.

"Status" indicates the state of the environment. As a specific example, the state is the position of the robot device, the angle of the arm of the robot device (robot arm), and the like. Therefore, the state is, for example, information about the controlled object.

"Action" is an action by which an agent influences the environment and changes its state. As a specific example, the action is movement of a robot device, torque applied to a robot arm, and the like.

"Reward" is an index that evaluates the goodness of an agent's state and actions. In reinforcement learning, a task is learned by learning to maximize the value of the reward. As a specific example, the reward is the distance between the goal and the robot device, that is, the distance between the target position and the current position of the robot device. Note that the magnitude of the reward value is set according to the quality evaluation criteria, and therefore does not necessarily match the magnitude of the parameter used to calculate the reward. For example, when the robot device learns that getting closer to a goal is a "good" result, the reward is calculated using an index (for example, the reciprocal of the distance) such that the closer the distance, the larger the reward value. Further, when the robot device learns that moving away from the goal is a "good" result, the reward is calculated using an index (for example, the distance value itself) such that the farther the distance, the larger the reward value.

"Policy" is a function that returns what action should be taken in a state (for example, an environment situation or an agent state). As a specific example, the policy is a strategy for the operation of the robotic device.

A "latent state" (latent variable) is used, for example, in a world model. The latent state is information (for example, vector information) that is obtained by compressing the observed information of the environment (for example, an observed image) and potentially expresses the situation of the environment.

"Latent space" is used, for example, in world models. Latent space is a vector space in which latent states are represented. Therefore, depending on how the latent state is positioned in the latent space, it is possible to understand how the agent recognizes observed information, that is, the learning state of the world model.

The “restored image” is used, for example, in a world model. The restored image is an image in which observation information (for example, an observed image) is restored again based on the compressed latent state. Therefore, from the restored image, it is possible to understand whether the agent correctly recognizes the observed image, that is, the learning state of the world model.

(Configuration of information processing system)
Next, the configuration of the information processing system will be explained.

FIG. 1 is a block diagram illustrating a configuration example of an information processing system 5 according to an embodiment of the present disclosure. The information processing system 5 includes a GUI (Graphical User Interface) section 100 and a simulator section 200.

The GUI unit 100 and the simulator unit 200 are communicably connected via a communication network or by direct communication. The GUI unit 100 is operated by a user and provides display information to the user. The simulator unit 200 is a physical calculation simulator used for reinforcement learning experiments. The simulator unit 200 uses an API (Application Programming Interface) or the like to acquire information on the environment (for example, images, the position and joints of the robot, time, or rewards), and inputs actions to act on the environment. or

The GUI unit 100 is, for example, a PC (Personal Computer) or other information processing device. The simulator unit 200 is, for example, a server device or other information processing device. Therefore, the GUI section 100 and the simulator section 200 may have a client-server relationship.

FIG. 2 is a block diagram showing an example of the configuration of the GUI section 100. The GUI section 100 includes a processor 110, a memory 120, a communication device 130, an operating device 140, and a display device 150.

The processor 110 may include an MPU (Micro Processing Unit), a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), or the like. The processor 110 may be configured with various integrated circuits (eg, LSI (Large Scale Integration), FPGA (Field Programmable Gate Array)). The processor 110 implements various functions by executing programs stored in the memory 120. The processor 110 centrally controls each section of the GUI section 100 and performs various processes.

Memory 120 includes a primary storage device (eg, RAM (Random Access Memory) or ROM (Read Only Memory)). The memory 120 may include a secondary storage device (for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive)), a tertiary storage device (for example, an optical disk or an SD card), or the like. Further, the memory 120 is an external storage medium, and may be removable from the GUI unit 100. The memory 120 stores various data, information, programs, etc.

Memory 120 may hold, for example, environment files, trained world models, and trained policy models. The environment file is one type of environment data that includes information regarding the environment in which the agent acts. A policy model is a learning model that has learned a policy, and is a model for an agent to act. One or more environment files, one or more learned world models, or one or more learned policy models may be prepared and held in the memory 120.

The communication device 130 communicates various data or information. The communication device 130 communicates according to a wired or wireless communication method. The communication method is WAN (Wide Area Network), LAN (Local Area Network), cellular communication for mobile phones (e.g. LTE, 5G), or short-range communication (e.g. infrared communication or Bluetooth (registered trademark) communication), or Power line communication or the like may also be used.

The operating device 140 may include a mouse, keyboard, touch pad, touch panel, microphone, or other input device. The operating device 140 receives input of various data and information.

Display device 150 may include a liquid crystal display device, an organic EL device, or other display device. The display device 150 displays various data and information. The display device 150 displays, for example, a screen or image that will be described later.

The processor 110 performs at least the processing necessary to present the learning state of the world model. As mentioned above, the world model is a reinforcement learning method that learns a predictive model of the state transition of the environment. The state transition of the environment indicates what state the environment transitions to when what action is performed in what state. By incorporating the world model, agents can potentially predict the outcome of a series of actions, reducing trial and error in task learning.

The processor 110 has at least an information specifying section 111, a latent state processing section 112, and a restored image processing section 113 as functional configurations.

The information specifying unit 111 specifies various types of information when determining the learning state of the world model. For example, the information specifying unit 111 specifies information that defines the virtual environment (for example, an environment file or a learned world model) or information that defines the behavior of the agent (for example, a learned policy model). Furthermore, the information specifying unit 111 specifies information that defines observation standards for the virtual environment (for example, information regarding a virtual camera that images the virtual environment (camera information)). The camera information includes, for example, information on the position (camera coordinates) and orientation of the virtual camera with respect to the virtual environment. The information specifying unit 111 may manually specify and set various model and camera information, for example, based on information held in the memory 120 or information on operations input to the operating device 140.

The latent state processing unit 112 maps latent states on the latent space based on the learned world model and the observed image acquired from the simulator unit 200. The latent state processing unit 112 may calculate the latent state by compressing the observed image. The latent state may be vector data. The latent state processing unit 112 inputs the observed image to the world model and derives the latent state as an output of the world model. This latent state is, for example, multidimensional information. The latent state processing unit 112 may dimensionally compress the obtained multidimensional latent state to derive (eg, calculate) a three-dimensional latent state. Note that the latent state processing unit 112 may perform dimension compression into a two-dimensional latent state instead of a three-dimensional latent state. The latent state processing unit 112 maps a three-dimensional latent state on a latent state mapping screen showing a latent space, and causes the display device 150 to display the map. Note that the latent state processing unit 112 may map onto a two-dimensional latent state on a two-dimensional plane instead of three-dimensionally.

Furthermore, if the user is able to handle a space of four dimensions or more through special training, the latent state processing unit 112 performs dimension compression on the four or more dimensional latent state, and displays the four-dimensional space on the latent state mapping screen. Information may also be displayed. However, the number of dimensions that a general user can intuitively handle is one to three dimensions, and in one dimension, all latent states are mapped to one point, making it difficult to recognize multiple latent states. Can not. Therefore, it is desirable that the latent state processing unit 112 handles two-dimensional or three-dimensional latent states. Further, in order to respond to various user requests, the latent state processing unit 112 may be configured to be able to switch the number of dimensions.

Furthermore, the latent state expressed by latent state mapping changes depending on the method used for dimension compression and the parameters left in dimension compression. Therefore, the latent state processing unit 112 may be able to change the method of dimension compression, the parameters to be left, and the like. By doing so, the GUI unit 100 can provide an interface that is easy to use for experienced users who are aware of the parameters and the like that they want to evaluate with priority.

The restored image processing unit 113 generates a restored image based on the learned world model and the latent state derived by the latent state processing unit 112. For example, the restored image processing unit 113 inputs the latent state to the world model and derives the restored image as an output of the world model. The restored image processing unit 113 may generate a restored image based on a multidimensional latent state or a dimensionally compressed two-dimensional or three-dimensional latent state. For example, the restored image processing unit 113 may generate a restored image by restoring the latent state. The restored image processing unit 113 displays the generated restored image on the display device 150.

Since the latent state is obtained by compressing the observed image, the restored image corresponds to the observed image. However, depending on the learning state of the world model, the observed image and the restored image may match, or may not match. The restored image processing unit 113 may determine the degree of correlation (eg, degree of coincidence) between the observed image and the restored image. The restored image processing unit 113 can determine the degree of correlation, for example, by summing up the differences in each pixel value between the observed image and the restored image. Further, the restored image processing unit 113 may determine the degree of correlation more precisely using a known technique used in image searches and the like. The restored image processing unit 113 determines that the learning state of the world model is good when this degree of correlation satisfies a predetermined standard, and that the learning state of the world model is poor when this degree of correlation does not meet the predetermined standard. You can judge. Note that the restored image processing unit 113 may display the information on the latent state mapping on the display device 150 without determining the learning state of the world model. Then, the user may judge whether the learning state of the world model is good or bad from the state of latent state mapping.

FIG. 3 is a block diagram showing an example of the configuration of the GUI section 100. The simulator unit 200 includes, for example, a processor 210, a memory 220, and a communication device 230.

Processor 210 may include an MPU, CPU, DSP, GPU, or the like. The processor may be configured with various integrated circuits (eg, LSI, FPGA). The processor 210 implements various functions by executing programs held in the memory 220. The processor 210 centrally controls each section of the simulator section 200 and performs various processes. Note that the processor 210 of the simulator section 200 may have higher performance than the processor 110 of the GUI section 100.

Memory 220 includes primary storage (eg, RAM or ROM). Memory 220 may include a secondary storage device (eg, HDD or SSD), a tertiary storage device (eg, optical disk or SD card), or the like. Furthermore, memory 220 may be an external storage medium. The memory 220 stores various data, information, programs, etc.

The communication device 230 communicates various data or information. The communication device 230 communicates according to a wired or wireless communication method. The communication method may be WAN, LAN, cellular communication for mobile phones (eg, LTE, 5G), short-range communication (eg, infrared communication or Bluetooth (registered trademark) communication), power line communication, or the like.

The processor 210 has at least an information setting section 211, a policy execution section 212, and an observation image processing section 213 as functional configurations.

The information setting section 211 sets various information. For example, the environment file, camera information, and policy model are acquired from the GUI unit 100 and set. The environment file, camera information, and policy model that are set are the same as those specified in the GUI section 100. This configuration information may be held in memory 220.

The policy execution unit 212 executes the policy according to the set trained policy model. Specifically, the policy execution unit 212 inputs information indicating the state of the environment (for example, a set environment file, camera information) into the policy model, and outputs the action that the agent should take (for example, a robot device) as the output of the policy model. (control information for the player to grasp the ball). The state of the environment can change depending on the derived actions of the agent.

The observation image processing unit 213 controls the virtual camera and acquires observation images from the virtual camera. The virtual camera images the dynamic environment according to the configured camera information. The acquired observation image is transmitted to the GUI section 100 by the communication device 230. The virtual camera may sequentially acquire observation images while the strategy is being executed, or may acquire observation images before the strategy is executed or observation images after the strategy is executed.

FIG. 4 is a diagram showing the list screen GL displayed on the display device 150.

The list screen GL shows the process from the environment defined by the environment file to obtaining the restored image, and includes screens and images obtained at each timing in this process. Specifically, the list screen GL includes an environment screen G1, a model setting screen G2, a camera setting screen G3, an observation screen G4, a latent state mapping screen G5, and a restoration screen G6.

The environment screen G1 displays an environment (virtual environment) defined by an environment file. In the environment screen G1, the environment is shown as a three-dimensional model, for example. In the environment shown by the environment screen G1, a robot device 10 as an agent, a ball 20, a shield 30, and the like are arranged. In this environment, for example, the robot device 10 is trying to grab the ball 20 while moving the robot arm 15 and the like. Therefore, the task is, for example, to bring the robot arm 15 closer to the ball 20.

The model setting screen G2 is a setting screen for setting information regarding various models and the like. The model setting screen G2 supports, for example, setting of an environment file, a world model, and a policy model regarding the environment. The camera setting screen G3 is a screen for setting camera information regarding the virtual camera.

The observation screen G4 is a screen that displays an observation image obtained by capturing an image of the environment from a virtual camera. Observation images are sequentially obtained each time a strategy is executed, so they are updated each time a strategy is executed. Furthermore, since the agent operates each time a policy is executed, the state of the environment may change sequentially. The observation screen G4 in FIG. 4 shows an observation image in which a shielding object 30 has entered the imaging range of the virtual camera, and when viewed from the virtual camera, a part of the robot device 10 is hidden behind the shielding object 30 and is difficult to see. Displayed.

The latent state mapping screen G5 is a screen on which the latent state based on the observed image is mapped onto two-dimensional coordinates or three-dimensional coordinates. On the latent state mapping screen G5, a plurality of latent states corresponding to a plurality of observation images are shown as dots and mapped. Since the latent state is a compressed image of the observed image, the latent state mapping screen G5 shows the relationship between multiple environmental states due to the execution of the policy by displaying multiple feature points of the environment indicated by the observed image. There is.

The restoration screen G6 is a screen that displays a restored image based on the latent state. In the restored screen G6 of FIG. 4, the restored image shows that the shielding object 30 has entered the imaging range of the virtual camera, and when viewed from the virtual camera, a part of the robot device 10 is hidden behind the shielding object 30 and is difficult to see. Displayed. Note that in the example of FIG. 4, the observed image and the restored image do not match and are slightly different.

Note that the arrangement of each screen on the list screen GL shown in FIG. 4 is an example, and is not limited to this. Further, a part of each screen included in the list screen GL shown in FIG. 4 may be hidden, or other screens not shown in FIG. 4 may be included and displayed in the list screen GL. Good too.

FIG. 5 is a diagram showing an example of the model setting screen G21. FIG. 6 is a diagram showing an example of the model setting screen G22 on which path names are displayed. Model setting screen G21 is displayed on display device 150. The model setting screens G21 and G22 are examples of the model setting screen G2.

As shown in FIG. 5, the information specifying unit 111 uses the model setting screen G21 via the operating device 140 to specify an environment file, a world model, and a policy model. Each of the specified information (environment file, world model, and policy model) may be input via the operation device 140, or one of the plural types held in the memory 120 may be selected. good. The information specifying unit 111 transmits (for example, uploads) each piece of specified information to the simulator unit 200 and instructs the simulator unit 200 to set each piece of information.

That is, the information specifying unit 111 uploads the environment file to the simulator unit 200 via the communication device 130, thereby setting (reflecting) the input execution environment in the simulator unit 200. The information specifying unit 111 uploads the world model to the simulator unit 200 via the communication device 130 to set a model (world model) in the simulator unit 200 for the agent to use to predict changes in the environment and restore images ( To reflect. The information specifying unit 111 uploads the policy model to the simulator unit 200 via the communication device 130, thereby setting (reflecting) a model for the agent to act (policy model) in the simulator unit 200. Note that when the uploading of the setting information is completed, the respective path names may be displayed (see FIG. 6). Note that it is not essential for the information specifying unit 111 to upload and set the world model and policy model to the simulator unit 200, and the processing based on the world model and policy model may be completed on the GUI unit 100 side.

Note that the information specifying unit 111 may specify the environment file, the world model, and the policy model according to predetermined specification criteria without any operation using the operating device 140.

FIG. 7 is a diagram showing an example of the camera setting screen G31. FIG. 8 is a diagram showing an example of the camera setting screen G32 when a plurality of virtual cameras are provided. Camera setting screens G31 and G32 are displayed on display device 150. Camera setting screens G31 and G32 are examples of camera setting screen G3.

The information specifying unit 111 uses the camera setting screen G3 via the operating device 140 to set details of camera information. The camera information is information regarding a virtual camera that images the environment. The camera information includes, for example, information on the position (camera coordinates) and orientation of the virtual camera with respect to the environment. In this case, the information specifying unit 111 may specify, via the operating device 140, the X coordinate, Y coordinate, and Z coordinate in the environment as the position where the virtual camera is placed. Further, the information specifying unit 111 may specify the values of roll, pitch, and yaw as the orientation of the virtual camera via the operating device 140. Further, the information specifying unit 111 may store default information of camera information in the memory 120, for example, and specify this default information as the camera information without operating via the operating device 140.

Further, the information specifying unit 111 may set camera information based on the operation of the operating device 140 on the environment screen G1. For example, the information specifying unit 111 may adjust the position and orientation of the virtual camera by dragging the virtual camera using a mouse operation on the environment screen G1.

Further, when setting camera information, a list screen GL may be displayed on the display device 150. In this case, when setting camera information, the user can set camera information installed in the environment (obtained by the agent) while viewing the environment screen G1 included in the list screen GL.

Further, when setting camera information, the information specifying unit 111 may interactively reflect the set camera information on the environment screen G1 and the observation screen G4. That is, the information specifying unit 111 may update the displayed environment screen G1 and observation screen G4 based on the image captured by the virtual camera according to the set position (camera coordinates) and direction (camera angle). In this case, the user can intuitively understand in what position or direction the virtual camera should be placed in order to observe the environment while adjusting the specification of camera information (for example, while moving the viewpoint). Note that the observation image on the observation screen G4 can be acquired via the communication device 130 by operating in cooperation with the simulator section 200.

Further, the number of virtual cameras that capture images of the environment may not be one, but a plurality of them. For example, the information specifying unit 111 may set the camera information of the second and subsequent virtual cameras by accepting a press of the camera addition button B1 (see FIG. 7) via the operation device 140 (see FIG. 8). reference). By having different camera information for multiple virtual cameras, the environment can be observed from various viewpoints.

Next, execution of the policy model will be explained.

The policy model is a model that outputs the next action according to the observed image. When the policy model is changed, the behavior of the agent due to policy execution changes.

In the GUI unit 100 , when the information specifying unit 111 receives an instruction to execute the policy via the operating device 140 , the information specifying unit 111 transmits the instruction to execute the policy to the simulator unit 200 via the communication device 130 . For example, the instruction to execute the policy via the operating device 140 may be a press of the play button B2 (see FIG. 6) on the model setting screen G2 (G22). In the simulator unit 200, upon receiving the policy execution instruction, the policy execution unit 212 executes the policy according to the set policy model in accordance with the execution instruction. Agents operate within the environment according to the execution of the policy.

Furthermore, in the GUI unit 100 , when receiving an instruction to stop the execution of the policy via the operating device 140 , the information specifying unit 111 transmits the instruction to stop the execution of the policy to the simulator unit 200 via the communication device 130 . For example, the instruction to stop execution of the policy via the operating device 140 may be, for example, pressing the stop button B3 (see FIG. 6) on the model setting screen G2 (G22). In the simulator unit 200, when the processor 210 receives the instruction to stop execution of the policy, it stops the execution of the set policy in accordance with this stop instruction. The agent stops working in the environment as the policy model stops running.

The GUI section 100 and the simulator section 200 cooperate with each other to interactively display an environment screen G1 and an observation screen G4 that reflect changes in the environment due to execution of the policy. Specifically, in the simulator unit 200, the communication device 230 transmits the changed environmental image and observation image to the GUI unit 100 when executing the policy model. In the GUI unit 100, the policy execution unit 212 receives the environmental image and observation image from the simulator unit 200 via the communication device 130, and displays the environment screen G1 including the environment image and the observation image including the observation image via the display device 150. Screen G4 is displayed. Note that during the period in which the implementation of the policy continues, changes between the environmental image and the observed image may occur sequentially. Therefore, the GUI unit 100 may acquire sequentially changing environmental images and observed images from the simulator unit 200 and display them. Therefore, the information processing system 5 can visualize and provide the user with the environment and observation images that change one after another as a result of executing the policy. Therefore, the user can intuitively understand changes in the environment due to execution of the policy in real time.

Furthermore, the policy execution unit 212 may obtain operation information input by the operation device 140 of the GUI unit 100 via the communication device 230, and move the virtual camera when executing the policy model according to this movement instruction. For example, during execution of the policy model, the policy execution unit 212 obtains operation information for dragging the camera icon CI of the virtual camera by a mouse operation on the environment screen G1, and determines the position and orientation of the virtual camera based on this operation information. It may be adjusted (for example, by moving the viewpoint). That is, the policy execution unit 212 may change the camera information of the virtual camera that was set before executing the policy model while executing the policy model. For example, the policy execution unit 212 may change the position (viewpoint) of the virtual camera by moving the position of the camera icon CI by a drag operation. For example, the policy execution unit 212 may change the orientation (line-of-sight direction) of the virtual camera by moving the position of the lens (not shown) of the camera icon CI. Note that the display example of the camera icon CI shown in FIG. 4 is just an example, and the camera icon CI may be displayed in other positions or display modes. The display device 150 can display information (for example, an environmental image, an observed image, a latent state, a restored image) based on an image captured by a virtual camera whose camera information has been changed. Therefore, the user can interactively search for suitable camera information (for example, camera position and orientation) while checking the agent's operational behavior, latent state, restored image, etc. by executing the policy model, and use the virtual camera to capture images. The image based on it is adjustable.

FIG. 9 is a diagram showing an example of the latent state mapping screen G5. In FIG. 9, the state of the latent state drawn on the latent state mapping screen G5 changes over time. Further, FIG. 9 illustrates that a restored image is generated based on one latent state mapped on the latent state mapping screen G5.

In the simulator unit 200, an observation image processing unit 213 controls a virtual camera. The virtual camera images the environment and generates an observation image according to the set camera information. The action of the agent according to the execution of the policy affects the environment, and the environment can change. The virtual camera may sequentially capture images of a changing environment in chronological order and may sequentially generate observation images. The communication device 230 sequentially transmits the generated observation images to the GUI unit 100.

In the GUI section 100, the communication device 130 receives observed images from the simulator section 200. The latent state processing unit 112 compresses the received observed image and generates a latent state corresponding to the observed image. In this case, the latent state processing unit 112 may generate a latent state based on the observed image using the specified world model. For example, the latent state processing unit 112 uses the observed image as input to a world model, and obtains a multidimensional latent state as an output of the world model. The latent state processing unit 112 performs dimension compression on the acquired multidimensional latent state to generate a two-dimensional or three-dimensional latent state. In this case, the latent state processing unit 112 may generate a dimensionally compressed latent state according to a dimensionally compressed technique (for example, PCA (principal component analysis)). A two-dimensional or three-dimensional latent state can be visually mapped onto a two-dimensional plane or three-dimensional space. The latent state processing unit 112 maps and draws, for example, a three-dimensional latent state on a latent state mapping screen G5 showing a latent space.

Further, the latent state processing unit 112 sequentially receives observation images from the simulator unit 200. Therefore, the latent state processing unit 112 generates a plurality of latent states based on a plurality of sequentially received observation images. Therefore, the latent state processing unit 112 draws the plurality of generated latent states on the latent state mapping screen G5. Since the observed images are obtained sequentially in chronological order, the number of latent states drawn also increases in chronological order. The latent state processing unit 112 maps each generated latent state onto the latent state mapping screen G5 and displays it on the display device 150.

In FIG. 9, first, the current latent state p1 corresponding to the current observation image is mapped and displayed on the latent state mapping screen G51. The current observed image here is, for example, an observed image of the environment before execution of the policy model.

When the simulator unit 200 starts executing the policy, the latent state processing unit 112 draws each latent state corresponding to each observation image obtained at each step of the policy execution on the latent state mapping screen G52. Therefore, the number of mapped latent states increases as time passes while the policy is being executed. In the latent state mapping screen G52, the latent state p11 is the current latent state (the latest latent state obtained during execution of this strategy). The latent state p12 is a latent state (sequence of latent states) sequentially obtained by executing the policy, and indicates a latent state other than the latent state p11.

The closeness between latent states on the latent state mapping screen G5 indicates the semantic similarity of the latent states. Therefore, by checking the display on the latent state mapping screen G5, the user can understand which observations that the agent recognizes as being close to each other, corresponding to which latent state. Changes in the environment due to the implementation of policies are expected to change little by little over time, and are unlikely to change significantly over time. Therefore, it is assumed that a plurality of latent states corresponding to a plurality of observation images that are close in time series are drawn at positions close to each other on the latent state mapping screen G5. Therefore, if multiple latent states obtained at close points in time are placed at far positions on the latent state mapping screen G5, the user may be unable to learn the world model that derives the latent states from the observed images. Be able to recognize that there is a possibility that the learning is insufficient or that it is being learned in the wrong direction.

Note that when the latent state is dimensionally compressed into three dimensions, the screen displayed on the latent state mapping screen G5 is information obtained by projecting three-dimensional information onto the two-dimensional projection surface of the display device 150. . In this case, if the user wishes to observe the projected result from another viewpoint, the latent state processing unit 112, via the operation device 140, displays a latent state indicating the three-dimensional space to which the latent state is mapped. The mapping screen G5 may be rotated in any direction and displayed via the display device 150. For example, the latent state processing unit 112 may accept a drag operation using a mouse, and may rotate the directions of three mutually orthogonal axes defining a three-dimensional space in accordance with the drag operation. This allows the user to check the positional relationships of a plurality of latent states from various viewpoints and line-of-sight directions.

Further, the latent state mapping screen G53 is a state that is later in time series than the latent state mapping screen G52, and the number of latent states is increasing. In the latent state mapping screen G53, the latent state p21 is the current latent state (the latest latent state obtained during execution of this strategy). The latent state p22 is a latent state (sequence of latent states) sequentially obtained by executing the policy, and indicates a latent state other than the latent state p21.

Here, the restored image processing unit 113 may specify one latent state p23 from among the plurality of latent states drawn on the latent state mapping screen G53 via the operating device 140. Regarding the latent state 23, the restored image processing unit 113 may display the specified latent state p23 on the display device 150 in a display mode different from the other latent states p21 and p22. The restored image processing unit 113 generates a restored image corresponding to the latent state p23 based on the specified latent state p23. For example, the restored image processing unit 113 performs restoration processing on the specified latent state p23 to generate a restored image. In this case, the restored image processing unit 113 may use the designated latent state p23 as an input to the world model, and may obtain the restored image as an output of the world model. The restored image processing unit 113 displays the generated restored image on the display device 150. By checking the display of the restored screen G61 including this restored image, the user can check how the agent understands the environment, the degree of learning of the agent's world model, etc.

The latent state is a compressed observation image. Furthermore, since the restored image is a latent state restored, the observed image is compressed and restored. Therefore, if the learning accuracy of the world model that derives the latent state from the observed image is high, the observed screen G4 showing the observed image and the restored screen G6 (G61) showing the restored image should match. In the list screen GL of FIG. 4, the observed image on the observation screen G and the restored image on the restored screen G6 are slightly different. Therefore, in this case, the user can understand that the learning state of the world model is not perfect.

Next, an example of the operation of the information processing system 5 will be described.
FIG. 10 is a sequence diagram showing an example of the operation of the information processing system 5. As shown in FIG.

First, in the GUI unit 100, the information specifying unit 111 inputs and specifies a learned world model and a learned policy model via the operating device 140 (S11). The trained models here include a world model and a policy model. For example, one or more learned world models and policy models may be stored in the memory 120, and which world model and policy model to use may be specified.

The information specifying unit 111 inputs environmental conditions via the operating device 140 and specifies the environmental conditions (S12). The environmental conditions here include environment files related to the environment, camera information of the virtual camera (for example, the position and angle (orientation) of the virtual camera), information on the behavior of the agent that affects the environment (for example, information on the behavior of the robot device), Including etc. For example, one or more environment files may be maintained in memory 120 and which environment file to use may be specified.

The communication device 130 transmits the determined policy model and environmental conditions to the simulator unit 200, and instructs the simulator unit 200 to set (reflect) the policy model and environmental conditions (S13).

In the simulator unit 200, the information setting unit 211 receives the policy model and environmental conditions from the GUI unit 100 via the communication device 230, sets the policy model and environmental conditions, and causes the memory 220 to retain the setting information. (S21).

The policy execution unit 212 executes the policy in the environment where the agent is placed based on the set environmental conditions and policy model (S22). Execution of the policy may cause the environment to change as the agent operates in the environment. The observation image processing unit 213 controls the virtual camera to capture an image of the environment and generate an observation image according to the set camera information (S23). That is, the observed image processing unit 213 acquires an observed image via the virtual camera as environmental information after reflecting the environmental conditions. The communication device 230 transmits the acquired observation image to the GUI unit 100 (S24).

Note that the policy is executed continuously in chronological order according to the policy model. That is, the policy execution unit 212 sequentially executes the policy model, and the observed image processing unit 213 sequentially acquires observed images via the virtual camera. The communication device 230 may sequentially transmit the sequentially obtained observation images one by one, or may repeat sequentially transmitting several observation images at once.

In the GUI section 100, the communication device 130 receives the observation image from the GUI section 100 (S14). In this case, the communication device 130 may sequentially receive the observed images one by one, or may sequentially and repeatedly receive several observed images at once.

The latent state processing unit 112 compresses the observed image and derives a latent state corresponding to the observed image based on the observed image and the learned world model (designated world model) (S15). Since the latent state is generated for each observed image, a plurality of latent states are derived. Multiple latent states are mapped into a latent space, which is a vector space. In this case, the original multidimensional latent state may be dimensionally compressed into a two-dimensional or three-dimensional latent state.

The restored image processing unit 113 restores the latent state and generates a restored image based on the derived latent state and the learned world model (designated world model) (S16). The restored image processing unit 113 may generate a restored image corresponding to a latent state specified via the operating device 140, for example. Further, the restored image processing unit 113 may generate one or more restored images corresponding to any one or more restored images without particularly specifying a latent state.

The display device 150 displays the derived observed image, latent state, and restored image (S17). For example, the observation image is displayed on the observation screen G4, the latent state is displayed on the latent state mapping screen G5, and the restored image is displayed on the restored screen G6. Further, the display device 150 may display a list screen GL including these images. In this way, the GUI unit 100 calculates and visualizes the restored image and latent space.

In this way, the information processing system 5 can present to the user the distribution of a plurality of latent states on the latent state mapping screen G5 showing the latent space and a restored image in which the latent states are restored. Therefore, the information processing system 5 can visualize whether the learning state of the world model is good or bad.

In the world model, the observed image and the restored image are not completely reversible, and the amount of information may gradually decrease as compression and restoration are repeated. The world model learns to make the restored image closer to the original observed image by acquiring latent states that are characteristically captured by the observed information of the task. Furthermore, since the world model inherently performs time-series prediction, feature points (latent states) with similar properties are designed to be close on the vector space.

By checking the distribution of multiple latent states, the user can recognize which observations the agent has determined to be close (semantically similar). In addition, by checking the restored image, the user can check how the agent understands and the learning progress of the world model. If the restored image is correctly restored, the user can confirm that both the latent state and the restoration are successfully learned. Therefore, the information processing system 5 can improve the interpretability of the world model and facilitate experiments by users (for example, engineers).

Note that in this embodiment, there is mainly one virtual camera and a single viewpoint, but there may be a plurality of virtual cameras and a multi-viewpoint.

Furthermore, in this embodiment, the task is illustrated as the agent's robot device 10 grabbing the ball 20, but the task is not limited to this. In other words, the task may be an operation other than the robot device 10 grasping an object. Furthermore, the agent may be other than a robot device, such as an automatic driving simulator or a game simulator. This embodiment is applicable to all technologies using reinforcement learning of a world model.

Further, in this embodiment, the processing to be shared between the GUI unit 100 and the simulator unit 200 may be shared by another method. For example, the GUI unit 100 may perform processing other than input via the operating device 140 and display by the display device 150 on the simulator unit 200 side as much as possible. For example, the simulator unit 200 may include the latent state processing unit 112 and the restored image processing unit 113.

Furthermore, in this embodiment, the GUI section 100 and the simulator section 200 may be configured as an integrated information processing device. In other words, the simulator section 200 may be omitted and the processing may be completed only on the GUI section 100 side. That is, the GUI section 100 may include the policy execution section 212 and the observed image processing section 213.

Further, in this embodiment, an example has been described assuming that the virtual environment is observed as an image, but the idea of this embodiment can also be applied to the case of observing other information such as audio. For example, when observing audio, the virtual camera may be used as a virtual microphone, and the audio measured by the virtual microphone or the restored audio may be output instead of the observed image or restored image.

Furthermore, in the present embodiment, the display device 150 displays the plurality of latent states as points arranged at two-dimensional or three-dimensional coordinates in the dimensionally compressed latent space; Other displays may be displayed in association with the above. For example, the display device 150 may display a table listing each coordinate as a numerical value. Further, when displaying as two-dimensional or three-dimensional points, the display device 150 may display lines or the like between adjacent points. In this case, the display device 150 may change the form (color and thickness) of the line to reflect the degree of similarity in a higher dimension. As a result of the loss of higher-order information in the process of dimensional compression, each latent state may appear close in the latent state mapping, but may be far apart in the higher-order dimension. For example, in the case of two-dimensional compression, even if the plane coordinates remaining as two-dimensional information appear to be close, the coordinates in the depth direction lost during the compression process may be extremely far away. A similar problem also exists when information of four or more dimensions is compressed into three dimensions. Therefore, the display device 150 can intuitively express the degree of similarity based on the lost information by displaying the information lost in the process of dimensional compression as other information such as the shape of the line. .

[Overview of embodiment]
As described above, the information processing system 5 of the above embodiment includes a learned world model, environmental data (for example, an environment file) regarding the environment in which the agent operates, and a learned policy model that defines the behavior of the agent. , an information specifying section 111 (an example of a specifying section) is provided. The information processing system 5 includes a policy execution unit 212 that sequentially executes policies in the environment based on the specified environmental data and the policy model, and a policy execution unit 212 that images the environment with a virtual camera each time a policy is executed, and and an observed image processing unit 213 that generates an observed image. The information processing system 5 includes a latent state processing unit 112 that compresses each of a plurality of observed images and derives each of a plurality of latent states based on a world model, and a display device 150 that displays the plurality of latent states. (an example of the section) and.

In other words, the information processing system 5 can visualize the latent state mapping by sampling the results of execution according to the specified environment and policy. Latent states are obtained from observed images based on a world model. The latent state corresponds to the observed image and indicates a characteristic part of the environment. Therefore, when multiple features indicated by multiple latent states are similar, the distance between the mapped multiple latent states becomes close, and when multiple features are dissimilar, the distance between the multiple mapped latent states becomes long. Become. Therefore, the user can easily check the display of the latent state mapping and intuitively understand the learning state of the world model based on, for example, the difference between the distance between the plurality of latent states and the assumed distance.

Further, the latent state processing unit 112 compresses each of the plurality of observed images based on the world model, derives each of the plurality of multidimensional latent states, and dimensionally compresses each of the plurality of multidimensional latent states. Each of a plurality of two-dimensional or three-dimensional latent states may be derived in this way. The display device 150 may display a plurality of latent states in association with two-dimensional or three-dimensional coordinates in the dimensionally compressed latent space.

Thereby, the information processing system 5 can adjust the latent state so that it can be displayed even if the latent state is multidimensional at the beginning of deriving the latent state based on the world model.

Furthermore, the information processing system 5 may further include a restored image processing section 113. The restored image processing unit 113 may designate one of the plurality of latent states, restore the specified one latent state based on the world model, and generate a restored image. Display device 150 may display the restored image.

Thereby, by checking the display of the restored image, the user can check how the agent understands the environment, the degree of learning of the agent's world model, etc.

Further, the information specifying unit 111 may specify camera information including the position and orientation of the virtual camera with respect to the environment. The observation image processing unit 213 may generate a plurality of observation images by capturing an image of the environment with a virtual camera based on the camera information.

Thereby, the information processing system 5 can acquire observation images taken from various positions and in various directions, and can grasp the learning state by the world model in detail.

The information processing system 5 may further include an operation device 140 (an example of a first operation section). The changed camera information may be changed, and the environment may be imaged by a virtual camera based on the changed camera information to generate a changed observation image.The display device 150 may display the changed observation image. good.

Thereby, the information processing system 5 can specify the camera information of the virtual camera while checking the observation image generated according to the change of the camera information. Therefore, camera information can be adjusted so that the observation image desired by the user can be obtained.

The information processing system 5 may further include an operation device 140 (an example of a second operation unit). The display device 150 rotates the latent space in which the latent state is drawn based on the operation on the operation device 140. It may be displayed as

As a result, the information processing system 5 can check the latent space by looking in various directions from various viewpoints, compared to a case where the viewpoint (camera position) and line of sight direction (camera orientation) for checking the latent space are fixed. It can be visually confirmed, and the details of the positional relationship between multiple latent states arranged in the latent space can be easily confirmed.

Further, the information processing system 5 may include a GUI section 100 (an example of an operation display device) and a simulator section 200 (an example of a simulator device). The GUI section 100 and the simulator section 200 may be communicably connected. The GUI section 100 may include an information specifying section 111, a latent state processing section 112, and a display device 150. The simulator unit 200 may include a policy execution unit 212 and an observed image processing unit 213.

Thereby, the information processing system 5 can execute, for example, a strategy with a high processing load on the simulator unit 200 side having a high processing capacity, and can perform distributed processing according to the processing capacity of each device.

Further, the information processing system 5 may be configured by a single information processing device (for example, the GUI section 100). Thereby, the information processing system 5 can complete the process for checking the learning state of the world model using a single information processing device.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that those skilled in the art can come up with various changes or modifications within the scope of the claims, and these naturally fall within the technical scope of the present disclosure. Understood. Further, each of the constituent elements in the above embodiments may be arbitrarily combined without departing from the spirit of the disclosure.

The execution order of each process such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, specification, and drawings is specifically defined as "before" or "before". ” etc., and unless the output of the previous process is used in the subsequent process, it can be implemented in any order. Even if the claims, specifications, and operational flows in the drawings are explained using "first," "next," etc. for convenience, this does not mean that it is essential to implement them in this order. isn't it.

The present disclosure is useful for information processing devices, information processing methods, and the like that make it easy to intuitively understand the learning state of a world model.

5 Information processing system 100 GUI section 110 Processor 111 Information specification section 112 Latent state processing section 113 Restored image processing section 120 Memory 130 Communication device 140 Operation device 150 Display device 200 Simulator section 210 Processor 211 Information setting section 212 Policy execution section 213 Observed image Processing unit 220 Memory 230 Communication device GL List screen G1 Environment screen G2 Model setting screen G3 Camera setting screen G4 Observation screen G5 Latent state mapping screen G6 Restoration screen

Claims (9)

a specification unit that specifies a learned world model, environmental data regarding the environment in which the agent operates, and a learned policy model that defines the behavior of the agent;
a policy execution unit that sequentially executes policies in the environment based on the environment data and the policy model;
an observation image processing unit that generates a plurality of observation images by capturing an image of the environment with a virtual camera each time the policy is executed;
a latent state processing unit that compresses each of the plurality of observed images and derives each of the plurality of latent states based on the world model;
a display unit that displays the plurality of latent states;
An information processing system equipped with. The latent state processing unit is
compressing each of the plurality of observed images based on the world model to derive each of the plurality of multidimensional latent states;
Dimensionally compressing each of the plurality of multidimensional latent states to derive each of the plurality of two-dimensional or three-dimensional latent states,
The display unit displays the plurality of latent states in association with two-dimensional or three-dimensional coordinates in the dimensionally compressed latent space.
The information processing system according to claim 1. further comprising a restored image processing unit;
The restored image processing unit includes:
specifying one latent state among the plurality of latent states;
generating a restored image by restoring the specified one latent state based on the world model;
the display unit displays the restored image;
The information processing system according to claim 1 or 2. The specifying unit specifies camera information including the position and orientation of the virtual camera with respect to the environment,
The observation image processing unit generates the plurality of observation images by capturing an image of the environment with the virtual camera based on the camera information.
The information processing system according to any one of claims 1 to 3. further comprising a first operation section,
The observation image processing unit includes:
changing the camera information specified by the specifying unit based on the operation on the first operating unit;
capturing an image of the environment with the virtual camera based on the changed camera information to generate the changed observation image;
the display unit displays the changed observation image;
The information processing system according to claim 4. further comprising a second operation section,
The display unit rotates and displays the latent space in which the latent state is drawn based on the operation on the second operation unit.
The information processing system according to any one of claims 1 to 5. The information processing system includes an operation display device and a simulator device,
The operation display device and the simulator device are communicably connected,
The operation display device includes the designation section, the observation image processing section, the latent state processing section, and the display section,
The simulator device includes the policy execution unit.
The information processing system according to any one of claims 1 to 6. The information processing system is configured by a single information processing device,
The information processing system according to any one of claims 1 to 6. specifying a learned world model, environmental data regarding the environment in which the agent operates, and a learned policy model that defines the behavior of the agent;
sequentially executing strategies in the environment based on the environment data and the strategy model;
capturing an image of the environment with a virtual camera each time the policy is executed to generate a plurality of observation images;
compressing each of the plurality of observation images to derive each of the plurality of latent states based on the world model;
Displaying the plurality of latent states on a display unit;
An information processing method having

Images