JP2020038631A - Information processing apparatus, information processing method, program, and system - Google Patents

Abstract

To stably control movement of a moving body which can mount objects of plural kinds to be carried.SOLUTION: The present invention is directed to an information processing apparatus for achieving the above object and for determining a control value for use in controlling a position of a moving body for carrying a carriage object. The apparatus is characterized by having acquiring means for first information which allows a three-dimensional shape of a carriage object to be specified on the basis of a first image obtained by capturing the carriage object and second information which, on the basis of a second image obtained by capturing an environment where the moving body moves, allows distance between an object and the moving body in the environment to be specified, and determining means for determining the control value for preventing the carriage object and the object from getting closer to each other as compared with a predetermined distance.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a technique for controlling movement of a moving object.

  For example, there is a mobile object called an automated guided vehicle (AGV). Conventionally, when this moving body is run in an environment such as a factory or a distribution warehouse, a tape is attached to the floor and the tape is mounted on the moving body as in Patent Document 1 in order to stably control the movement of the moving body. It is known to travel while detecting with a sensor.

JP 2010-33434 A

  However, in order to stably move a moving object carrying unspecified luggage, it is necessary to change the moving route according to the amount and shape of the luggage carried on the moving object. That is, it is necessary to manually set the traveling route so that the vehicle travels with an appropriate distance between the obstacle in the environment and the moving object or the luggage.

  The present invention has been made in view of the above problems, and has as its object to stably control the movement of a moving body on which a plurality of types of goods can be mounted.

  An information processing apparatus according to the present invention that achieves the above object is an information processing apparatus according to the present invention that achieves the above object, and an information processing apparatus that determines a control value for controlling a position of a moving object that carries a conveyed object. And based on first information capable of specifying a three-dimensional shape of the transported object based on a first image of the transported object and a second image of an environment in which the moving object moves. Acquiring means for acquiring second information capable of specifying the distance between the object and the moving object in the environment, and the transported object and the object based on the first information and the second information. And determining means for determining the control value for preventing the control value from approaching a predetermined distance.

  ADVANTAGE OF THE INVENTION According to this invention, the movement control of the moving body which can carry several types of goods can be performed stably.

The figure explaining an example of the environment in which the mobile body system runs. FIG. 2 illustrates a system configuration example of an information processing system. FIG. 2 is a block diagram illustrating a functional configuration example of a mobile system. FIG. 2 is a diagram illustrating a hardware configuration example of an information processing apparatus. 9 is a flowchart illustrating processing executed by the information processing device. The figure explaining an example of the course which a mobile system runs. FIG. 2 is a block diagram illustrating a functional configuration example of a mobile system. 9 is a flowchart illustrating processing executed by the information processing device. The figure which shows an example of the GUI which presents display information. FIG. 2 is a block diagram illustrating a functional configuration example of a mobile system. 9 is a flowchart illustrating processing executed by the information processing device. FIG. 2 is a block diagram illustrating a functional configuration example of a mobile system. The figure which shows an example of the GUI which presents display information. 9 is a flowchart illustrating processing executed by the information processing device.

  Hereinafter, embodiments will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(Embodiment 1)
In the present embodiment, movement control of an automatic guided vehicle (AGV) carrying a load will be described. The automatic guided vehicle is hereinafter referred to as a moving object. This moving body can carry a plurality of types of transported goods, and the path through which the moving body can pass changes depending on the shape of the transported goods mounted on the moving body. Further, the mobile body system 12 can recognize the three-dimensional structure of the space from the image of the camera mounted on the vehicle body. In FIG. 1, when the moving object moves in an environment where there are beams on the ceiling and bumps on the walls and shelves, places where the moving object can pass vary depending on the presence or absence and height of the conveyed object. Specifically, when the height of the conveyed object is lower than the broken line A, the mobile system 12 can go straight. However, since the height of the transported object 15 reaches the broken line B, in this case, it is necessary to control the movement so that the transported object 15 does not go straight but advances in a detour. In addition, even when the transported object has a size that protrudes in the width direction of the mobile system, the mobile system needs to travel along a route that takes into consideration the shape of the transported object. In addition, mobile objects used in factories and warehouses, such as unmanned transport vehicles, carry various types of goods according to tasks and unload at designated locations. In the process of performing a certain task, the amount and size of the load carried by the moving object are not always constant. Even if the information about the shape of the goods is given at the beginning, the information about the shapes of the goods has to be updated before and after loading / unloading the luggage. Therefore, based on information on the height and the three-dimensional shape of the transported object obtained from the image, a control value for controlling the position of the mobile object such that the mobile object system travels on a route where the transported object does not contact an obstacle. (Direction or speed) will be described. FIG. 2 shows a configuration example of the information processing system. Based on the three-dimensional shape of the transported object estimated from the image captured by the transportable object sensor 110 and the position and orientation information of the mobile system 12 estimated from the image captured by the environment sensor 120 mounted on the mobile system 12, The mobile system 12 is controlled.

(Description of configuration)
FIG. 2 illustrates a configuration example of an information processing system that controls the position of a moving object called an automated guided vehicle (AGV) in the present embodiment. The information processing system 1 according to the present embodiment includes at least one or more mobile systems 12, a mobile management system 13, and a process management system 14. The information processing system 1 is, for example, a distribution system or a production system.

  The mobile system 12 is an automated guided vehicle (AGV) that transports the transported object 15 according to a schedule of a process required for performing a task in a use environment such as a factory or a distribution warehouse. A plurality of mobile systems 12 are moving (running) in the environment. Also, the mobile system 12 bidirectionally transmits and receives various information to and from the mobile management system 13 and other mobile systems 12 by wireless communication such as Wi-Fi communication. The moving body system 12 includes an information processing device 10 that determines a control value for controlling the position of the moving body, an actuator unit 130 that controls movement of the moving body according to the control value, a conveyed object sensor 110 that images the conveyed object 15, and a movement. An environment sensor 120 for observing the surrounding environment of the body is mounted.

  The mobile management system 13 is a system that manages the mobile system 12. For example, a computer server, a PC, an embedded system, or a PLC (Programmable Logic Controller). The mobile management system 13 bidirectionally transmits and receives various information for controlling the movement of the mobile system 12 with the mobile system 12 by wireless communication such as Wi-Fi communication. Further, the mobile object management system 13 communicates with the process management system 14.

  The process management system 14 manages a schedule of a process executed in the information processing system 1 in a factory or a distribution warehouse. For example, a production management system, a physical distribution management system, and a MES (Manufacturing Execution System). Further, the process management system 14 communicates with the mobile object management system 13.

  The goods 15 are the goods carried by the mobile system 12. For example, it is an object to be transported, such as a container storing a plurality of industrial parts or a cargo formed by stacking a plurality of containers or cardboard. The conveyed article 15 is conveyed to the mobile system 12 by loading or towing.

  FIG. 3 is a diagram illustrating an example of a functional configuration of a mobile system 12 including the information processing device 10 according to the present embodiment. The information processing apparatus 10 includes a load image input unit 1110, a load information acquisition unit 1120, an environment image input unit 1130, a position and orientation information acquisition unit 1140, a holding unit 1150, and a determination unit 1160. The load image input unit 1110 is connected to the load sensor 110 mounted on the mobile system 12. The environment image input unit 1130 is connected to the environment sensor 120 mounted on the mobile system 12. The determination unit 1160 is connected to the actuator unit 130. In addition, a communication device (not shown) bidirectionally communicates information with the mobile management system 13, and inputs and outputs various functions of the information processing device 10. However, FIG. 3 is an example of a device configuration, and does not limit the applicable range of the present invention.

  The conveyed object sensor (first sensor) 110 is a measuring device that measures the conveyed object 15 conveyed by the mobile system 12. In the present embodiment, the conveyed object sensor 110 is a depth camera based on the ToF method that captures a depth map in which each pixel stores depth information, and outputs a depth map. It is assumed that the positional relationship between the load sensor 110 and the mobile system 12 and the positional relationship between the load sensor 110 and the environment sensor 120 have been calibrated in advance.

  For example, a gray scale camera or a color camera that acquires a grayscale image or a color image may be used as the transported object sensor 110. In this case, for example, the three-dimensional bounding box representing the shape of the conveyed object is directly estimated from the color image by a learning-based method using CNN (Convolutional Neural Network). Specifically, the three-dimensional shape of the transported object is estimated by using a learned model (CNN) that outputs the absolute value of the size of the transported object by using the color image and the distance value as inputs. The trained model is trained in advance by teacher data to which information of the distance value and the size of the object has been added to images of various objects. Alternatively, the processing described in the present embodiment may be performed by learning a network for estimating a depth map from a monocular color image using a CNN, and using the depth map estimated from the color image. It is also possible to match the CAD data held in a storage device (not shown) in advance with the image acquired by the conveyed object sensor 110, and to estimate the approximate shape of the conveyed object 15 from the dimension of the CAD data having the highest matching degree. Further, the load sensor 110 may measure the shape of the load 15 by using a line sensor that linearly senses the distance from the sensor to the target, for example. In this case, the shape of the transported object 15 is estimated, for example, by scanning a line according to the movement of the transported object at the timing when the transported object is loaded on the moving body, and obtaining a three-dimensional point group of the transported object. I do. As described above, as long as information on the shape of the transported object 15 can be obtained, the sensor used as the transported object sensor is not limited, and any sensor may be used.

  The environment sensor 120 is a measuring device that measures the surrounding environment in which the moving body is traveling. In the present embodiment, the environment sensor 120 is a depth camera. It is assumed that the environment sensor 120 outputs a depth map observing the surroundings of the moving object. Here, the depth map refers to an image that holds a value correlated with the distance (depth) to the measurement target for each pixel of the image acquired by the environment sensor 120. Usually, the value correlated with the distance to the measurement target is an integer value that can be configured as a normal image, and is multiplied by a predetermined coefficient determined from the focal length to obtain the physical distance to the target (for example, millimeters). ) Can be converted. The method of measuring the distance for generating the depth map is not particularly limited. For example, there is an active stereo system in which a multi-slit line irradiated on an object is imaged by a camera and distance measurement is performed by triangulation. A Time-of-flight (ToF) system using the flight time of light, such as Lidar, may be used. Further, a passive type in which the depth of each pixel is calculated by triangulation from an image captured by a stereo camera may be used. In the present embodiment, the environment sensor 120 is a depth camera based on the ToF method. The environment sensor 120 sends the measured depth map to the environment image input unit 1130.

  The actuator unit 130 drives the wheels based on the control values determined by the determining unit 1160 (for example, the rotational torque of the wheels and the movement control information on the traveling direction). The movement control in the present embodiment is to control a motor that is an actuator included in the moving body and a steering that changes the direction of wheels. By controlling these, the moving body is moved to a predetermined destination. The control value is a command value for controlling the moving body. Specifically, it is the traveling direction of the moving body, the acceleration, the rotation speed of the wheels, and the like. Here, it refers to a control value such that a mobile object carrying a load travels on an optimal route (for example, the shortest route) for traveling to a destination.

  The position and orientation of the environment sensor according to the present embodiment are six parameters including three parameters representing the position of the environment sensor 120 in the world coordinate system defined in the real space and three parameters representing the orientation of the environment sensor 120. It is. Here, the world coordinate system is defined as a coordinate system composed of two axes that define the floor surface and one axis that is orthogonal to them and represents the height direction. At the design stage of a moving body such as an AGV, the mounting position of the environment sensor 120 with respect to the position of the center of gravity of the moving body is measured, and a matrix representing the mounting position and posture described above is stored in the external memory H14. The position of the center of gravity of the moving object can be obtained by multiplying the position and orientation of the environment sensor by the matrix representing the mounting position and orientation described above. That is, by acquiring the position and orientation of the environment sensor in the world coordinate system, the position and orientation of the mobile system 12 in the world coordinate system can be easily acquired. Further, a three-dimensional coordinate system defined on the imaging device with the optical axis of the transported object sensor 110 being the Z axis, the horizontal direction of the image being the X axis, and the vertical direction being the Y axis is referred to as a first coordinate system. Similarly, a three-dimensional coordinate system defined on the imaging device with the optical axis of the environment sensor 120 as the Z axis, the horizontal direction of the image as the X axis, and the vertical direction as the Y axis is referred to as a second coordinate system. Further, a three-dimensional coordinate system defined on the mobile body system 12 having the traveling direction of the mobile body system 12 as the X axis, the height direction as the Z axis, and the axis orthogonal thereto as the Y axis is referred to as a third coordinate system. Call.

  The transported object image input unit (first input unit) 1110 performs a time series (depth map) storing depth values for each pixel of the image of the scene in which the transported object is observed, as visual information acquired by the transported object sensor 110. (For example, 60 frames per second). Then, the information is output to the conveyed object information acquisition unit 1120. Here, the depth value is a distance between the conveyed object sensor 110 and the conveyed object conveyed by the AGV.

  The cargo information acquisition unit (first acquisition unit) 1120 acquires information (first information) on the three-dimensional shape of the cargo from the depth map input by the cargo image input unit 1110. In the present embodiment, a rough shape represented by a three-dimensional bounding box (three parameters representing the position of the center of gravity and three parameters representing the size of the bounding box) of the carried object is estimated as information on the shape of the carried object. Specifically, it indicates the height of the moving object on which the load is placed, and the size of the portion of the load that protrudes from the moving object in the width direction. The conveyed object information acquiring unit 1120 further outputs the acquired conveyed object shape to the determining unit 1160.

  The environment image input unit (second input unit) 1130 outputs, from the environment sensor 120, a depth map in which depth values are stored for each pixel of an image of a scene in which the environment in which the moving object moves is stored in a time series (for example, 60 pixels per second). Frame). Then, it outputs the depth map to the position and orientation information acquisition unit 1140. The depth value is a distance between the environment sensor 120 and an object (for example, a wall or an obstacle) in the surrounding environment.

  The position and orientation information acquisition unit (second acquisition unit) 1140 acquires information that can specify the distance between the object and the moving object in the environment. That is, a position / posture estimation map serving as an index of position / posture acquisition is held, and the information (second information) of the environment sensor is used using the depth map and the position / posture estimation map input by the environment image input unit 1130. ) To get. Here, the position and orientation estimation map is three-dimensional point cloud data indicating the shape of the environment. In the present embodiment, it is assumed that the position and orientation information acquisition unit 1140 holds the point cloud as a data list storing three values of three-dimensional coordinates (X, Y, Z) in an arbitrary world coordinate system. It is assumed that the position / posture estimation map is generated by converting the CAD model of the environment into a three-dimensional point group indicating the depth of the object surface in advance. Note that control map information described later may be used as the position and orientation estimation map. The position and orientation information acquisition unit 1140 further outputs the acquired position and orientation to the determination unit 1160. Since the positional relationship between the environment sensor and the moving body has been calibrated in advance, if the position and orientation of the environment sensor are known, the position and the traveling direction of the moving body can be known.

  The holding unit 1150 holds an occupancy map as control map information. The occupancy grid map is a map that divides a scene into grids and holds the probability that an obstacle exists on each grid. The occupancy grid map is stored as a three-dimensional voxel space (X, Y, Z) in the world coordinate system. Alternatively, the information is held as a two-dimensional grid space (X, Y) excluding height information. In the present embodiment, the occupancy grid map is held as information of a three-dimensional voxel space. The occupancy grid map is generated by converting the CAD model of the environment into an occupancy grid map in advance, similarly to the position and orientation estimation map. Further, in addition to these, the target position / posture indicating the three-dimensional coordinates and the posture as the destination of the AGV is held. Although there may be one or more target position / posture, an example in which the target position / posture is one point will be described here for simplicity. Further, the holding unit 1150 holds information on the approximate shape of the moving object including the height, width, and depth of the moving object. Since the outline shape of the load may change depending on the loading / unloading of the load, the information of the state where the load is not loaded is held here as an initial value. The information on the conveyed object is acquired by a method described later at the timing when the moving object starts the task of conveying the conveyed object. The holding unit 1150 outputs map information obtained from the position / posture estimation map or the occupancy grid map to the determination unit 1160 as necessary. Furthermore, the target position and orientation are output to the determination unit 1160.

  The deciding unit 1160 sets the control value based on the three-dimensional shape of the conveyed object acquired by the conveyed object information acquiring unit 1110 so that the moving object and the conveyed object are separated from the environment (obstacle) by a certain distance or more. decide. If necessary, the position and orientation of the environment sensor 120 acquired by the position and orientation information acquisition unit 1140, the control map information held by the holding unit 1150, and the information on the target position and orientation are used. The determining unit 1160 outputs the determined control value to the actuator unit 130.

  FIG. 4 is a diagram illustrating a hardware configuration of the information processing apparatus 10. H11 is a CPU for controlling various devices connected to the system bus H21. H12 is a ROM, which stores a BIOS program and a boot program. H13 is a RAM, which is used as a main storage device of the CPU H11. H14 is an external memory, which stores a program to be processed by the information processing device 10. The input unit H15 is a keyboard, a mouse, and a robot controller, and performs a process related to input of information and the like. The display unit H16 outputs the calculation result of the information processing device 10 to the display device according to the instruction from H11. The type of display device is not limited, such as a liquid crystal display device, a projector, and an LED indicator. Further, the display unit H16 included in the information processing device may serve as a display device. H17 is a communication interface for performing information communication via a network, and the communication interface may be Ethernet (registered trademark), and any type such as USB, serial communication, wireless communication, etc. may be used. Note that information is exchanged with the above-mentioned mobile management system 13 via the communication interface H17. H18 is an I / O for inputting visual information from the imaging device H19. Note that the image pickup device H19 is the above-described transported object sensor 110 or the environment sensor 120. H20 is the actuator unit 130 described above.

  Next, a processing procedure in the present embodiment will be described. FIG. 5 is a flowchart illustrating a procedure of a process executed by the information processing apparatus 10 according to the present embodiment. Hereinafter, the flowchart is realized by the CPU executing the control program. In the following description, each step (step) is described with a leading S, so that the notation of the step (step) is omitted. The processing steps include S110, S120, S130, S140, S150, S160, and S170.

  In S110, the system is initialized. That is, the program is read from the external memory H14, and the information processing apparatus 10 is set in an operable state. In addition, the parameters of the devices connected to the information processing apparatus 10 (camera internal parameters and external parameters of the transport sensor 110 and the environment sensor 120) and the initial position and orientation of the transport sensor 110 are read into the RAM H13. In addition, each device of the AGV is activated and brought into a controllable state. In addition to these, the three-dimensional coordinates of the destination to which the AGV should travel are received from the mobile management system via the communication I / F (H17), and stored in the storage unit 1150.

  In S120, the transported object image input unit 1110 inputs a depth map which is an image of the transported object captured by the transported object sensor 120. Specifically, a depth map of a scene in which the transported object 15 is imaged by the depth camera is input. In the present embodiment, the depth map is a two-dimensional array list that stores the depth value of each pixel.

  In S130, the cargo information acquisition unit 1120 uses the depth map (first image) input by the cargo image input unit 1110 to specify the three-dimensional shape of the cargo 15 (first information). To get. In the present embodiment, the information that can specify the three-dimensional shape of the conveyed article 15 is a rough shape, and is six parameters (a position three parameter and a size three parameter) representing a three-dimensional bounding box. Specifically, an Axis-aligned bounding box (AABB) configured by the height, the width, the depth, and the position of the center of gravity in the moving object coordinate system is acquired.

  To acquire the schematic shape of the transported object 15, first, a three-dimensional point group defined in the first coordinate system is acquired from the depth map. The three-dimensional point group is obtained by multiplying the image coordinates in the normalized image coordinate system by the depth value of each pixel of the depth map. Next, the coordinate system of the acquired three-dimensional point group is converted from the coordinate system of the image pickup of the transported object to the coordinate system of the moving body. Further, by removing planes from the three-dimensional point group using a RANSAC (RANdom Sample Consensus) algorithm, three-dimensional points that are candidates for a transport object are separated from the three-dimensional point group that is a background such as a floor surface. I do. Finally, by obtaining the minimum value and the maximum value in each axis of the three-dimensional point group from the three-dimensional point group of the transportable candidate, the height, width, depth, and center of gravity of the transported object in the moving body coordinate system ( AABB) is obtained as a schematic shape of the conveyed object. Finally, the acquired value of the schematic shape is output to the determination unit 1160.

  In S140, the environment image input unit 1130 acquires a depth map observing the surrounding environment of the moving object. This depth map is acquired by the environment sensor 120. In the present embodiment, the depth map is a two-dimensional array list that stores the depth value of each pixel.

  In S150, the position and orientation information acquisition unit 1140 acquires the position and orientation (second information) of the environment sensor 120 using the depth map input by the environment image input unit 1130 and the position and orientation estimation map. Specifically, first, a three-dimensional point group defined in the second coordinate system is obtained from the depth map. The three-dimensional point group is obtained by multiplying the image coordinates in the normalized image coordinate system by the depth value of each pixel of the depth map. Next, using the position and orientation of the environment sensor 120 at the previous time, the three-dimensional point group is coordinate-transformed to the position and orientation coordinate system at the previous time. That is, the three-dimensional point group is multiplied by the matrix of the position and orientation at the previous time. The position and orientation are acquired such that the sum of the distances between the nearest points of the acquired three-dimensional points and the three-dimensional points in the point cloud of the map information held by the holding unit 1130 is reduced. Specifically, the position and orientation of the environment sensor 120 with respect to the position and orientation at the previous time are acquired using an Iterative Closest Point (ICP) algorithm. Finally, the position and orientation in the world coordinate system are converted to the world coordinate system and output to the determination unit 1160. By obtaining the position and orientation of the moving body, the distance between the surrounding environment and the moving body can be obtained.

  In S160, the determination unit 1160 determines whether or not the transported object is based on the three-dimensional information (second information) of the environment in which the moving object moves and information (first information) that can specify the height of the transported object. A control value for suppressing the obstacle from approaching a predetermined distance is determined. Specifically, the control value is determined so as to move along a route having a height at which the moving object carrying the load can pass. An example is shown in FIG. In FIG. 6A, the mobile unit E12a performs a task of transporting a conveyed object to a destination E16a. First, from the three-dimensional occupancy grid map E11, an occupancy grid map E13a, which is a three-dimensional map obtained by extracting the occupancy grid map E11 at the height of the moving object E12a including the transported object, is obtained. Then, a two-dimensional occupancy grid map E14a is obtained by projecting the projection on the floor plane. On the two-dimensional occupancy grid map shown in E14a, an optimal route E17a for moving from the position and orientation E15a of the moving object to the position and orientation E16a of the destination is calculated. For example, when the moving body E12a goes straight as it is, the transported object of E12a comes into contact with the environment (for example, X in the figure), and the transported object may collapse. In order not to select the route, a control value is determined that passes through a trajectory E17a that does not pass through an area where an obstacle exists. FIG. 6B illustrates a case where the heights of the goods are different. The three-dimensional occupancy grid map extracted according to the height of E12b is E13b. X is not displayed on the occupancy grid map R14b projected on the two-dimensional map at the height of E13b. Since the maps are different in this way, the calculated route and control value are also different. Specifically, when the moving object E12b carries a smaller amount of goods than the E12a, the passage (the road where the obstacle X was located) in front of the E12b is allowed to pass. Head for.

  The processing contents in S160 will be described. First, based on the approximate shape (AABB) of the conveyed object in the moving object coordinate system estimated by the position and orientation information acquisition unit 1140 and the approximate shape (AABB) of the AGV in the moving object coordinate system, the entire moving object including them is included. Get the approximate shape of height, width and depth. The general shape of the entire moving body is obtained by taking the minimum and maximum values of height, width, and depth. Next, from the occupancy grid map of the three-dimensional grid held by the holding unit 1150, a partial map of a range that can contact the moving object is extracted from the occupancy grid map. Specifically, in the occupancy grid map, if the position of the center of gravity of the entire moving object is (Xw, Yw, Zw) and the size of the 3D bounding box is (Sx, Sy, Sz), the height direction Zw ± with respect to the floor surface Slice the occupancy grid map in the range of Sz / 2. The map extracted in this way is obtained as a partial occupancy grid map. Furthermore, an occupancy grid map of a two-dimensional grid is obtained by projecting the partial occupancy grid map in the height direction (z direction) with respect to the floor surface. Here, the projection in the height direction refers to an operation of scanning the occupied grid (x, y, z) in the z direction and obtaining the maximum value of the probability that an obstacle exists at each (x, y). Similarly, the target position and orientation and the current position and orientation of the mobile body system 12 expressed as a three-dimensional position and orientation in the world coordinate system are represented by parameters of a total of three degrees of freedom of position two degrees of freedom and posture one degree of freedom. Is projected onto a two-dimensional plane such that Here, the position 2 degree of freedom indicates positions X and Y on a plane horizontal to the floor of the environment. In addition, the attitude 1 degree of freedom indicates a rotation direction on a plane horizontal to the floor of the environment. Finally, based on the occupancy grid map of the two-dimensional grid and the information on the position and orientation of the target and the current position and orientation, it is necessary to minimize the positions and orientations of both and avoid grids with a high probability of the presence of obstacles. Determine the control value. Specifically, first, based on the current position and posture of the moving object and the input target position and posture, a control value (forward speed and turning direction / speed) that reduces the Euclidean distance between both is set. All possible variations are calculated as control value candidates. Then, for each control value candidate, the position and orientation after control are calculated as the predicted position and orientation. Then, referring to the occupancy grid map of the two-dimensional grid, the probability that an obstacle corresponding to each predicted position and orientation exists is obtained. By calculating a candidate for a control value having a probability that an obstacle exists of 0, a candidate for a control value that does not cause a collision is extracted. Among the extracted control value candidates, the control value that minimizes the Euclidean distance between the target position and orientation and the predicted position and orientation is determined as the final control value. Then, the determined control value is output to the actuator unit 130. Then, the actuator unit 130 controls the AGV using the control value determined by the determining unit 1160.

  Specifically, the rotation amount of the motor serving as the actuator and the steering angle value for changing the direction of the wheels are adjusted so that the control values (forward speed and turning direction / speed) determined by the determination unit 1160 are obtained.

  In S170, it is determined whether to end the system. Specifically, if the Euclidean distance between the destination coordinates held by the holding unit 1150 and the position and orientation of the environment sensor 120 acquired by the position and orientation information acquisition unit 1140 is equal to or less than a predetermined threshold (for example, 1 m), the destination is determined. Finish as it arrives. If not, the process returns to S140 to continue the process.

  In the first embodiment, the moving object sensor acquires the three-dimensional schematic shape of the moving object, obtains the general shape of the entire moving object including the moving object, and moves the moving object including the moving object and the obstacle in the environment without contact. The route is acquired and the movement of the moving object is controlled. In this way, stable and safe control that prevents the cargo from colliding with the obstacles in the environment according to the size of the cargo to be transported (height of the cargo, the portion of the cargo protruding from the moving body, etc.) Becomes possible. Further, even in the case where the size of the cargo changes each time the cargo is transported, the movement control of the mobile body system 12 can be performed without setting the size of the cargo in advance and reducing the labor.

(Modification 1-1)
In the present embodiment, the transported object 15 is imaged by the transported object sensor 110, but the present invention is not limited to this. For example, the transported object 15 may be imaged using the environment sensor 120 instead of the transported object sensor 110. In this case, the environment sensor 120 and the conveyed object sensor 110 may be the same device. In this case, for example, a wide-area camera is mounted on a moving body that also serves as the load sensor 100 and the environment sensor 120. In addition, the surveillance camera installed in the environment so that the moving object can be imaged may be a wide-angle lens camera or a panoramic camera, and the environment and the moving object on which the load is mounted may be simultaneously imaged. Since the transported object sensor 110 images the environment and also the transported object, it is possible to control the movement of the moving object according to the shape, position, and type of the transported object with a smaller number of imaging devices. In addition, a plurality of sensors may be linked to acquire a transported object image. For example, images are acquired from a depth camera mounted on a moving object and one or more wide-area cameras installed in the environment. By using a plurality of sensors, the blind spot is reduced, and the information on the cargo can be more accurately estimated.

(Modification 1-2)
In the present embodiment, it is assumed that the transported goods sensor 110 is mounted on the mobile body system 12, but the present invention is not limited to this. For example, a monitoring camera that is installed at a place where the moving object and the conveyed object can be observed and that can communicate with the process management system 14 or the moving object system 12 may be used as the conveyed object sensor 110. Here, the surveillance camera is an RGB color camera that monitors the status of a use environment such as in a factory or a distribution warehouse. The holding unit 150 in the process management system 14 or the mobile system 12 holds the position and orientation of the monitoring camera on the world coordinate system of the environment. A monitoring camera as the load sensor 110 observes information on the shape and position and orientation of the load 15. The observed information is converted into information on the shape and position of the transported object 15 in the moving object coordinate system using the position and orientation of the monitoring camera in the world coordinate system and the position and orientation of the environment sensor 120, and transmitted to the determination unit 1160. Is entered. Further, the external camera used as the transported object sensor 110 is not limited to a monitoring camera. For example, a camera installed on the side of an automatic machine for loading a transported object on a moving body is used. Instead of a camera, for example, a two-dimensional distance sensor such as a line sensor may be used. In addition to the camera installed in the environment, a camera mounted on another AGV or a camera held by the user may be used. There is no particular limitation on the sensor used as the cargo sensor 110 as long as the cargo 15 can be measured and the information on the cargo 15 on the mobile system 12 can be acquired.

(Modification 1-3)
In the present embodiment, the cargo information acquisition unit 1120 acquires the value of the three-dimensional bounding box as the three-dimensional schematic shape of the cargo, but is not limited thereto. For example, the three-dimensional point group itself of the cargo candidate in the depth map may be held as a shape, or a mesh may be generated from the three-dimensional point group and handled as a three-dimensional polygon model. Alternatively, the shape of the conveyed object is held by a method in which a signed distance to the nearest three-dimensional surface is stored for each voxel, such as a TSDF volume (Truncated Signed Distance Function volume). Alternatively, the shape is represented in the form of an occupancy grid map in which the existence probabilities of the objects are stored in voxels. There is no particular limitation on the means for estimating the shape of the conveyed object and the method of expression. Similarly, the holding method / expression of the shape of the moving object held by the determining unit 1160 is not limited to a schematic shape such as a bounding box, and the detailed shape may be held using the method described above. Good. As the shape information of the conveyed object acquired by the position and orientation information acquiring unit 1140, any information may be used as long as the shape and size of the conveyed object 15 can be expressed, and there is no limitation.

(Modification 1-4)
In the present embodiment, the cargo information acquisition unit 1120 determines the three-dimensional point group of the cargo candidate by segmentation of the plane portion and the cargo candidate portion, but the present invention is not particularly limited to this. For example, consider the case where the approximate positional relationship between a load 15 loaded or towed by the mobile system 12 and a load sensor 110 is known. At this time, the processing may be performed assuming that the range in which the conveyed object exists is given as a 3D ROI (Region of Interest) in the depth map. In this case, instead of performing the segmentation of the plane portion, the cargo information acquisition unit 1120 uses the 3D ROI representing the range in which the above-described cargo exists, and determines a three-dimensional point group within the 3D ROI as a cargo candidate. Are extracted as a three-dimensional point group. Alternatively, when the shape of a part of a transported object is known, such as a transported object constituted by stacked boxes, the transporting is performed using three-dimensional object detection based on template matching or feature point matching. The position of the shape of the whole object may be obtained.

(Modification 1-5)
In the present embodiment, the control map is not limited to a method using an occupancy grid map set in advance. For example, an occupancy grid map is dynamically updated by calculating an area where an obstacle is present and an area where an obstacle is not present based on the depth map acquired from the environment sensor 120 and the position and orientation information estimated by the position and orientation information acquisition unit 1140. May be. The occupancy grid map is generated from 0 without prior information. Alternatively, the occupancy grid map stored in advance is obtained by updating. There is no particular limitation on the method of generating and updating the occupancy grid map as long as information on obstacles in the surrounding environment can be represented. By dynamically updating the occupancy grid map, the moving object can be controlled in consideration of obstacles not included in the occupancy grid map generated in advance, such as luggage newly placed in the environment.

(Modification 1-6)
In the present embodiment, the method of using the three-dimensional occupancy grid map as the control map and the method of estimating the position and orientation parameters having six degrees of freedom as the position and orientation information of the environment sensor 120 have been described, but are not limited thereto. Not something. For example, a two-dimensional occupancy grid map having no height information is used. Alternatively, two-dimensional point cloud data (in combination with color information) on a plane horizontal to the floor of the environment or a two-dimensional model on a plane horizontal to the floor of the environment are used as control maps. The position and orientation information includes two degrees of freedom (positions X and Y on the plane horizontal to the floor of the environment) of the mobile system 12 on the world coordinate system of the environment and one degree of freedom of posture (on the floor of the environment). (A rotation direction on a horizontal plane) with respect to a total of three degrees of freedom. The control map is not limited to a storage / representation method as long as the information of the obstacle in the surrounding environment can be stored in a form that can be used by the determination unit 1160, and any method may be used.

(Modification 1-7)
In the present embodiment, the method has been described in which the position and orientation information acquisition unit 1140 acquires the position and orientation of the environment sensor 120 based on an ICP algorithm of a three-dimensional point group representing a scene in the environment and a depth map. The method of obtaining is not limited to the above. For example, the map is generated by moving the mobile system 12 in the environment without generating a position / posture estimation map in advance. Specifically, by integrating the depth map obtained from the environment sensor mounted on the mobile body system 12 from a plurality of viewpoints while operating the mobile body system 12 with a remote controller or a hand push, the three-dimensional point cloud data is obtained. Generate. Further, as the position / posture estimation map, a CAD drawing or map image of the environment may be used as it is or may be converted into a data format and used for position / posture acquisition. Further, instead of the depth sensor, an imaging device that captures a scene in the environment as a color image or a grayscale image may be used as the environment sensor 120. In this case, the captured image may be a color image or a grayscale image. In this case, a feature point map composed of three-dimensional feature points is generated from the input color image or grayscale image, and the feature points in the image input to the environment image input unit and the feature points in the feature point map are generated. And the position and orientation are acquired. Many of these processes are proposed as SLAM (Simultaneous Localization and Mapping) technology for estimating position and orientation while creating map information, and these may be used. In addition, the position and orientation information may be estimated by the position and orientation information acquisition unit 1140 using the communication status of Wi-Fi communication and a beacon installed in the environment. As long as the position and orientation information acquisition unit 1140 can acquire the position and orientation of the moving body 12 in the world coordinate system, there is no particular limitation on the acquisition method, and any method may be used.

(Modification 1-8)
In the present embodiment, the method of acquiring the shape of the object has been described. However, the information of the conveyed object estimated by the conveyed object information obtaining unit 1120 is not limited to the shape. For example, semantic information such as the type and state of the object may be used. The determination unit 1160 may acquire the control value and determine the control value based on the acquired control value. Specifically, for example, a material having a fragile property such as glass or liquid or a type of a conveyed object that is inconvenient to shake is defined in advance. From the color image including the transported object 15 captured by the transportable object sensor 110, by semantic segmentation for determining what kind of object each pixel is, the semantic class of the transportable object candidate area is estimated, and the Estimate the type. Then, if the estimated type is included in the type of the fragile transportation object determined in advance, the maximum speed of the control value of the determination unit 1160 is limited. For example, when it is estimated that the conveyed object is made of glass, movement control with as little vibration as possible is performed. Thereby, by controlling the speed according to the type of the conveyed object, the movement of the moving object is controlled so that the conveyed object is hardly shaken or the conveyed object is hardly collapsed in the conveyance of the conveyed object. It becomes possible.

(Modification 1-9)
In the present embodiment, the shape of the conveyed object information is measured from the depth map of the conveyed object input to the conveyed object image input unit 1110, but the present invention is not limited to this. For example, information on a transported item may be obtained by using an ID tag attached to the transported item. Specifically, for example, an ID tag in which information on the shape and type of the conveyed article or information associated with the information is stored is added to the conveyed article. Alternatively, the information on the shape and type of the conveyed object is obtained by recognizing the ID tag from the image captured by the conveyed object sensor 110 or the environment sensor 120, by recognizing the ID tag. The means for obtaining information on the conveyed goods is not limited to the method using images, and for example, the process management system 14 or the moving body management system 13 manages the shape, type, and position of each conveyed goods. In S110, at the timing of receiving the target position and orientation, the load image input unit 1110 receives information on the load to be loaded. In this manner, information on the shape and type of the conveyed object may be obtained. In this case, there is no need to connect the load sensor 110 to the load image input unit 1120, and there is no particular problem even without the load sensor 110.

(Modification 1-10)
In the present embodiment, the mobile system 12 is applicable to various forms of an automatic guided vehicle (AGV). For example, it may be a moving object that is transported by squeezing under a conveyed object and pushing. Alternatively, an AGV in which wheels move freely in all directions may be used. In this case, the control value is calculated not only for the control value in the front-back direction but also for the movement in the horizontal direction, and the optimum control value is determined. Further, a tow type AGV may be used. In this case, the control value is obtained in consideration of the movement of the to-be-carried object or the joint with the bogie. Since the towing type rolls sideways, the way of determining the control value may be changed depending on the form of the moving body, for example, by passing the path with a sufficiently wide path. The mobile system 12 may be an autonomous vehicle or an autonomous mobile robot. The information processing device described in the present embodiment may be applied to them. Further, the present embodiment may be applied to an automatic transport forklift having a movable fork. This embodiment may be applied to a flying object such as a drone.

(Modification 1-11)
In the present embodiment, the determining unit 1160 determines a control value in consideration of the contact between the moving object including the transported object and the obstacle based on the schematic shape of the transported object 15 estimated by the transported information obtaining unit 1110. But it is not limited to this. For example, the determining unit 1160 communicates with the mobile object management system 13 and the process management system 14 to set the system so that a mobile object having a shape that does not cause contact between the conveyed object and the surrounding environment does not occur. May be operated. Specifically, the mobile management system 13 holds information on the schematic shapes of a plurality of mobile systems. Then, the determining unit 1160 acquires a plurality of two-dimensional occupied grids corresponding to the schematic shapes of the plurality of mobile systems. As a result, a mobile system having a shape capable of being transported on the shortest route is determined. Then, the determining unit 1160 communicates with the mobile object management system 13 so that the determined mobile object carries the object. Depending on the shape of the conveyed goods, instead of controlling only one moving object, the operation is controlled so that the entire system is optimized by selectively operating a plurality of moving objects of different shapes. It becomes possible.

(Modification 1-12)
A method in which the determination unit 1160 determines a control value of a moving object using a partial three-dimensional control map extracted from a range that comes into contact with the moving object without projecting the three-dimensional control map in two dimensions. Is described. In S160 of FIG. 5, determination unit 1160 determines a control value for controlling AGV. Specifically, based on the destination coordinates defined on the occupied grid map and the position and orientation of the moving object on the occupied grid map, avoid grids with a high probability of the presence of obstacles on the occupied grid map, A control value that reduces the Euclidean distance between the two is determined.

  First, the outline shape of the entire moving object is acquired based on the outline shape (AABB) of the object in the moving object coordinate system estimated by the object information acquiring unit 1120. Next, as in the first embodiment, from the three-dimensional occupancy grid map held by the holding unit 1150, an occupancy grid map of a portion that can be in contact with the moving object is extracted as a partial occupancy grid map. Next, based on the current position and orientation of the moving object and the input target position and orientation, all variations that can be taken as control values (forward speed and turning direction / speed) that reduce the Euclidean distance between them are controlled. Get as a value candidate. For each control value candidate, a predicted position and orientation after control is obtained. Then, a collision between the moving body and the partial occupancy grid map at the predicted position and orientation is determined. Specifically, the collision determination scans the partial occupancy grid map within the range of the approximate shape of the moving body at the predicted position and orientation. If there is an occupied grid having a probability of becoming an obstacle that is higher than a predetermined value, the risk of collision is determined, and the control value candidate that has obtained the predicted position and orientation is rejected. Among the control value candidates that have not been rejected, the control value that minimizes the Euclidean distance between the target position and orientation and the predicted position and orientation is determined as the final control value. Then, the determined control value is output to the actuator unit 130. Then, the actuator unit 130 controls the AGV using the control value determined by the determining unit 1160. Thereby, it is possible to perform stable and safe control such that the cargo does not collide with an obstacle in the environment according to the size of the cargo to be carried. Further, even in the case where the size of the cargo changes each time the cargo is transported, the movement control of the mobile body system 12 can be performed without setting the size of the cargo in advance and reducing the labor.

(Modification 1-13)
The occupancy grid map is held as the control map held by the real holding unit 1150, and the optimum route is determined based on the occupancy grid map. The determination unit 1160 determines the control value of the moving object. It is not limited. In addition, the determination unit 1160 may use any control value as long as it determines a control value for reaching the destination so that the moving object shape does not contact the obstacle in the environment. You may. For example, a plurality of depth maps may be held as key frames, and a control map may be held as a set of key frames. In this case, for example, while performing movement control to minimize the Euclidean distance between the target position and orientation and the current position and orientation, for example, a depth map in a key frame that is the closest to the current position and orientation, and a schematic shape of the entire moving object Calculates the closest distance to. When the distance is equal to or shorter than a predetermined value, control such as stopping the movement or turning is performed. As a threshold value for determining an obstacle, for example, if it is higher than 0, it is determined that there is an obstacle. Further, the present invention is not limited to holding the depth map as a key frame, but calculates the closest distance between the depth map input to the environment image input unit 1120 and the schematic shape of the entire moving object, and May be controlled.

  Further, based on an ESDF (Euclidean Signed Distance Fields) storing a signed distance to the nearest obstacle as a control map, a control value for avoiding an obstacle is calculated, and movement control is performed. Is also good. In addition, as the control map, a cost map storing a value that becomes smaller as the position is closer to the destination is held. The control unit 1160 may determine the control value using a deep reinforcement learning device, which is a neural network trained to determine the control value by using this and the input depth map as an input. In this way, the AGV is operated stably and safely by calculating the contact between the shape of the entire moving object including the conveyed object and obstacles such as walls, and determining the control value for moving while avoiding them. can do.

(Modification 1-14)
The determination unit 1160 used Dynamic Window Approach. This is based on the current position and orientation of the moving object and the input target position and orientation, and all possible variations of the control value such that the Euclidean distance between the two at a certain time is reduced to the control value. Calculate as a candidate. However, the method of calculating the control value is not limited to the above, as long as the method can set an AGV route while avoiding an area having a high possibility of collision with an obstacle. For example, Graph is a method of calculating a variation of a control value at a plurality of future times, not at one time, calculating a position and orientation (prediction trajectory) at a plurality of times, and selecting a control value that follows the prediction trajectory. A Search Approach may be used. At this time, the determination unit 1160 selects a predicted trajectory that passes through only the area where the probability of the presence of the obstacle in the occupied grid map of the two-dimensional grid is 0, and calculates a control value at a plurality of times that follows the predicted trajectory. . The method of selecting a predicted trajectory may select a predicted trajectory that minimizes the movement route, or a predicted trajectory that maximizes the distance to an area where the probability of the presence of an obstacle in the occupancy grid map is not zero. Alternatively, a predicted trajectory that minimizes a change in steering may be selected. In addition, a certain area without obstacles is randomly sampled from the two-dimensional space from the current position. Then, an arbitrary method such as a randomized approach that selects a predicted trajectory generated by repeatedly sampling the surrounding space at random with the point as a node can be used.

  In the present embodiment, the occupancy grid map in the range of Zw ± Sz / 2 is sliced and acquired as a partial occupancy grid map. However, the range of Zw−Sz / 2 <Z <Zw + Sz / 2 + Δm (Δm is a margin) May be sliced. By doing so, the AGV can move away from the obstacle by a predetermined margin or more, and the AGV can be operated safely.

  The predetermined value such as Δm may be set in advance by the user, or may be changed according to the traveling speed of the moving object. For example, when the setting is made by the user, in consideration of the regulations of the factory, the occupancy grid map is set so as to travel by Δm = 1 m from a position where the probability that an object exists is greater than 0. In this way, by providing a margin between the obstacle and the surrounding environment and the transported object, the transported object can be stably transported even if slightly shifted. When setting Δm according to the traveling speed of the moving body, for example, the traveling speed is determined so as to be proportional to a predetermined value. Specifically, when the moving body travels at 1 m / s, Δm = 1 m. When the moving body runs at 2 m / s, Δm = 2 m. By setting the distance between the obstacle and the transported object in this manner, the possibility of collision with the obstacle can be suppressed.

  Further, the position and orientation information acquisition unit 1140 acquires and calculates a variance value as the value of the degree of variation in the position and orientation when calculating the position and orientation, and the determination unit 1160 can set Δm based on the acquired variance value. For example, the variance and the predetermined value Δm are determined so as to be proportional. Alternatively, a variation in control of the moving object may be measured in advance, and the predetermined value Δm may be determined in proportion to the variation. The variation of the control is a variation degree of a movement locus actually measured when the moving body is set to move along the same route and moved. Similarly, the determination unit 1160 may determine the predetermined distance based on the height of the step in the traveling environment of the moving body measured by the position and orientation information acquisition unit 1140. The predetermined value Δm may be set so as to be proportional to the vertical shake of the moving object and the inclination value of the luggage measured when the moving object is moved. For example, if there is a step or a slope on the traveling route, there is a risk that the moving object shakes up and down and the conveyed object collides with a ceiling, a monitoring camera, or the like. Therefore, Δm may be set in accordance with the height of the step, for the entire traveling route or for a region having a step or irregularities of a predetermined height. The data of the unevenness of the running surface may be obtained by, for example, running on the running route of the moving body in advance and measuring a change in the three-dimensional position in the height direction. By setting a certain distance or more between the obstacle and the transported object, the possibility that the transported object collides with the obstacle can be further suppressed.

(Embodiment 2)
In the first embodiment, the occupancy grid map is held as the control map held by the holding unit 1150, the optimal route is acquired based on the occupancy grid map, and the determination unit 1160 acquires the control value of the moving object. . In the second embodiment, a method of calculating the position of a moving object including a transported object based on an image captured by an environment sensor and controlling the moving object without holding a control map will be described.

  FIG. 7 illustrates an example of a functional configuration of the information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 according to the present embodiment is different from the information processing apparatus 10 according to the first embodiment in that the holding unit 1150 is omitted. In the present embodiment, only the configuration added or different in function from the first embodiment will be described, and the description of the other configurations will be omitted because they have the same functions as those in FIGS.

  The determination unit 1160 is based on the environment image, the position and orientation information of the environment sensor 120 acquired by the position and orientation information acquisition unit 1140, the schematic shape of the conveyed object acquired by the conveyed object information acquisition unit 1120, and the position and orientation of the target. A control value for controlling the movement of the AGV is determined. Unlike the first embodiment, the target position / posture of the moving object is directly input from the moving object management system 13 to the determination unit 1160 by a communication device (not shown). The determining unit 1160 outputs the determined control value to the actuator unit 130.

  The procedure of the entire process in the present embodiment is the same as that in FIG. In the present embodiment, only the procedure that is different from that of the first embodiment will be described, and the other procedures are the same as those in FIG.

  In S160 of FIG. 5, the determining unit 1160 determines the obstacle based on the schematic shape of the conveyed object 15 estimated by the conveyed object information obtaining unit 1120, the depth map input to the environment image input unit 1130, and the target position of the moving object. The control value of the moving object is determined so as to go to the target position while avoiding an object. More specifically, all variations that can be taken as control values (forward speed and turning direction / speed) that reduce the Euclidean distance between the two based on the current position and orientation of the moving object and the input target position and orientation. Is obtained as a control value candidate. For each control value candidate, a predicted position and orientation after control is obtained. Then, the closest distance between the approximate shape and position of the moving object including the transported object at the predicted position and orientation and the three-dimensional point group obtained from the depth map input from the environment image input unit 1130 is calculated. If the distance is equal to or less than a predetermined value, it is determined that there is a risk of collision, and the control value candidate that has obtained the predicted position and orientation is rejected. The control value that minimizes the Euclidean distance between the target position and orientation and the predicted position and orientation is selected as the final control value from among the control value candidates that have not been rejected. The determined control value is output to the actuator unit 130.

  As described above, in the second embodiment, a method of acquiring the position of the schematic shape of the moving object including the transported object based on the image input from the environment sensor 120 without holding the control map and controlling the moving object Was mentioned. Even in the case where the control map is not held, by acquiring the distance between the moving object and the obstacle in the environment from the image input in time series from the environment sensor 120 in the range observed each time, Even when the control map is not held, safe moving body control is possible.

(Embodiment 3)
In the first embodiment, the approximate shape of the transported object 15 is measured only once from the depth map acquired by the transported object sensor 110, and the contact between the moving object including the measured shape of the transported object and the obstacle in the scene is considered. The trajectory was acquired to control the moving object. In the third embodiment, a method of controlling a moving object based on a change in the position or shape of a conveyed object by constantly tracking the shape and position of the conveyed object 15 will be described.

  The configuration of the apparatus according to the present embodiment is the same as that of FIG. 3 illustrating the configuration of the information processing apparatus 10 described in the first embodiment, and a description thereof will not be repeated. In addition, in addition to the schematic shape information of the conveyed article 15 described in the first embodiment, the conveyed article information acquiring unit 1120 further calculates the shape and position of the conveyed article 15 at the previous time so that its time-series change can be calculated. It also holds information. Then, the time-series change information of the transported object 15 is also output to the determination unit 1160. The determining unit 1160 determines a control value for controlling the AGV based on the time change information of the transported object 15 acquired by the transported object information acquiring unit 1120 in addition to the information described in the first embodiment. The above points are different from the first embodiment.

  FIG. 8 shows a processing procedure executed by the information processing apparatus applied to a moving object called an automatic guided vehicle (AGV) in the present embodiment. In the present embodiment, only the procedure that is different from that of the first embodiment will be described, and the other procedures are the same as those in FIG.

  In S230, as in S130 of the first embodiment, the cargo information acquisition unit 1120 acquires the information of the three-dimensional bounding box as the schematic shape of the cargo 15 from the depth map. Next, the difference between the held three-dimensional bounding box at the previous time and the value at the current time is obtained, thereby obtaining the amount of change in the approximate shape and position of the transported object 15 from the previous time to the current time. . It is assumed that the amount of change holds a total of six parameters including three parameters of the size (height / width / depth) of the bounding box and three parameters of the position. Finally, the acquired outline shape of the conveyed article 15 at the current time and the value of the amount of change are output to the determination unit 1160.

  In S260, based on the schematic shape of the cargo 15 estimated by the cargo information acquisition unit 1120 and the occupancy grid map held by the holding unit 1150, the determination unit 1160 moves toward the target position and posture while avoiding obstacles. Determine the control value of the moving object. In the present embodiment, in addition to this, the control value is determined based on the amount of change in the shape and position of the conveyed article 15 acquired by the conveyed article information acquiring unit 1120. After the control value of the moving object is determined by the processing described in the first embodiment, final control of the moving object is further performed based on the amount of change in the shape and position of the conveyed object 15 acquired by the conveyed object information acquiring unit 1120. Determine the value.

  More specifically, the amount of change in the general shape and position of the conveyed goods is checked, and if both or either of them has been changed by a predetermined amount or more, the conveyed goods have collapsed or shifted. To obtain a control value so as to reduce or stop the speed of the moving object. On the other hand, when the change amount of the conveyed object is 0 or small, it is assumed that the conveyed object is stably conveyed, and the control value acquired by the method described in S160 in the first embodiment is used as it is. If the amount of change in the conveyed object is not large enough to cause collapse or displacement of the conveyed object, but not so small that the amount of change can be regarded as 0, it is determined that the conveyed object is shaking. Determine the range and obtain the control value. Specifically, the three-dimensional bounding box representing the schematic shape of the transported object 15 and the amount of change in the shape and the position of the transported object are added to obtain a schematic shape of the transported object that covers the maximum swing width. Based on this, the general shape of the entire moving body described in S160 is obtained, and the control value is determined. The determined control value is output to the actuator unit 130.

  In S470, it is determined whether to end the system. Specifically, similarly to S170 of the first embodiment, the Euclidean distance between the destination coordinates held by the holding unit 1150 and the position and orientation of the environment sensor 120 acquired by the position and orientation information acquisition unit 1140 is a predetermined threshold (for example, 1 m). If it is below, it is determined that the vehicle has arrived at the destination, and the processing ends. Otherwise, the process returns to S120 to continue the processing.

  As described above, in the third embodiment, the shape and position of the conveyed object are measured and tracked for each frame, the amount of change is obtained, and a state change such as displacement or collapse of the conveyed object is determined according to the amount of change. How to change the control of the said. In the case of a change in the state of the transported object, it is based on control such as stopping the moving object, limiting the maximum speed, and calculating the contact between the moving object and the environment taking into account the swing width of the transferred object. Control. As a result, it is possible to safely control the moving body in consideration of temporal changes such as collapse, displacement, and shaking of the transported object.

(Modification 3-1)
In the present embodiment, the determination unit 1160 determines the control value of the moving object according to the amount of change in the shape and the position of the transported object 15, but is not limited thereto. For example, when the occurrence of collapse or displacement of the conveyed object is determined in accordance with the amount of change, instead of acquiring a control value, an alarm (not shown) is sounded to indicate the abnormality to the outside. Alternatively, an abnormality is indicated by lighting a lamp (not shown). Alternatively, it communicates with the mobile management system 13 to notify the system that an abnormality has occurred.

(Embodiment 4)
A UI that can be commonly applied to the first to third embodiments will be described. A description will be given of how a user checks visual information acquired by an imaging device, position and orientation acquired by an acquisition unit, object detection results, map information, and the like. In addition, it will be described that the AGV is controlled by a user input because the AGV is operated by automatic control. For example, a GUI is displayed on a display as a display device so that the user can control the AGV so that the user can check the status of the AGV, and inputs an operation from the user using an input device such as a mouse or a touch panel. In the present embodiment, the display is mounted on the AGV, but is not limited to such a configuration. That is, a liquid crystal display connected to the mobile object management system, which uses the display of the mobile terminal of the user as a display device, is used as the display device via the communication I / F (H17). The display information can be generated by the information processing device regardless of whether a display device mounted on the AGV is used or a display device not mounted on the AGV. When a display device not mounted on the AGV is used, a computer attached to the display device may acquire information necessary for generating display information from the information processing device and generate the display information.

  The configuration of the apparatus according to the fourth embodiment includes a generation unit 1170 as shown in FIG. 10 in addition to the functional configuration of FIG. The generation unit 1140 includes a conveyed object image acquired by the conveyed object sensor 110, a position and orientation of the moving object acquired by the position and orientation information acquisition unit 1140, an image acquired by the environment sensor 120, and information on the object measured by the first acquisition unit. , And generates display information based on the control value acquired by the determining unit 1160. Further, display unit 140 displays the generated display information on a touch panel display or the like mounted on the AGV. The details of the display information will be described later.

  FIG. 9 illustrates a GUI 100 that is an example of display information presented by the display device according to the present embodiment. G110 is a window for presenting a two-dimensional occupancy grid map. G120 is a window for presenting the three-dimensional occupancy grid map held by the holding unit 1150. G130 is a window for presenting the image acquired by the environment sensor 120. G140 is a window for presenting the image acquired by the conveyed object sensor 110. G150 is for presenting the position and orientation acquired by the position and orientation information acquisition unit 1140, the information on the conveyed object 15 measured by the conveyed object information acquisition unit 1120, and the display information on the control value determined by the determination unit 1160. It is a window.

  G110 shows a presentation example of a two-dimensional occupancy grid map obtained by projecting the three-dimensional occupancy grid map held by the holding unit 1150 onto a floor plane. G111 is an AGV equipped with an environment sensor 120 and a carried object sensor 110. G111 indicates the outline shape of the AGV calculated based on the outline shape of the article 15 measured by the article sensor 110. The position and orientation information acquisition unit 1140 composes on the 2D map based on the position and orientation of the environment sensor (the position and orientation of the AGV). G112 is an example in which a balloon is presented as a warning alert. For example, according to the method described in the second embodiment, an alert is displayed when the determination unit 1160 detects collapse or displacement of a conveyed object based on the change amount of the conveyed object 15 obtained by the conveyed object information obtaining unit 1120. . G113 is an example in which the scheduled progress route of the AGV is presented as an arrow based on the control value determined by the determination unit 1160. In FIG. 9, the AGV is heading for the destination presented at G114. As described above, the user can easily grasp the operation status of the AGV by presenting the position of the AGV, the alert of the abnormality, and the route of the abnormality. G111 to G114 may be configured so that the user can more easily grasp the operation status by changing the color, the thickness of the line, and the shape.

  G120 indicates a presentation example of a three-dimensional occupancy grid map held by the holding unit 1150. G121 is a state in which the schematic shape of the conveyed article 15 acquired by the conveyed article information acquiring unit 1120 is displayed on an AGV model. G122 is a display example showing a partial map that extracts a range that can contact the schematic shape of the AGV including the transported object 15. G110 is a projection of the three-dimensional occupancy grid map in this range onto the floor plane. As described above, by presenting the three-dimensional occupancy grid map, the user can grasp the operation status while observing the three-dimensional contact state between the obstacles having different heights existing in the environment and the AGV and the transported object. can do.

  G130 indicates a presentation example of an image acquired by the environment sensor 120. Like G130, the traveling direction of another AGV may be displayed by an arrow. The display of the arrow is generated based on information acquired from the mobile management system 13 or the like.

  G140 shows a presentation example of an image of the conveyed object 15 acquired by the conveyed object sensor 110. G141 is a presentation example of information indicating the schematic shape and position of the conveyed article 15 acquired by the conveyed article information acquiring unit 1120 and the type of the conveyed article described in the first embodiment. G142 is a presentation example showing the schematic shape of the conveyed object, including the swing width of the conveyed object, calculated from the amount of change in the shape and position of the conveyed object 15 described in the second embodiment. Here, the arrows regarding the three axes are displayed, but the range in which the conveyed object exists may be displayed as a wire frame. This allows the user to intuitively grasp the status of the transported item, including the size and position of the transported item.

  G150 indicates a GUI for manually operating the AGV, a position and orientation of the AGV acquired by the position and orientation information acquisition unit 1140, a control value determined by the determination unit 1160, and a presentation example of a target position and orientation of the AGV. I have. G151 is an emergency stop button, and the user can stop the movement of the AGV by touching this button with a finger. G152 is a mouse cursor, which can be moved according to a user's touch operation through a mouse, controller, or touch panel (not shown), and a button or radio button in the GUI can be operated by pressing a button. . G153 is an example in which an AGV controller is presented. By moving the circle inside the controller up, down, left, and right, the user can perform the front, rear, left, and right operations of the AGV according to those inputs. G154 is an example in which the internal state of the AGV is presented. The state where the AGV is running automatically and operating at a speed of 0.5 m / s per second is shown as an example. In addition, the target position and attitude, such as the time since the AGV started running, the remaining time required to reach the destination, and the difference between the expected arrival time and the expected time, were also presented. G156 is a GUI for setting the operation of the AGV and the display information. The user can perform operations such as whether to generate map information and whether to present a detected object. G157 is an example in which the target position and orientation of the AGV are presented. This is an example in which the position and orientation acquired by the second acquisition unit 1140, the destination coordinates received from the moving object management system 13, and the name of the article carried by the AGV are presented. As described above, by presenting the GUI related to the input from the user, which indicates the target position and orientation, the AGV can be operated more intuitively.

  The processing executed by the information processing apparatus will be described with reference to FIG. In step S565, the generation unit 1170 controls the captured image, the shape of the conveyed object obtained by the conveyed object information obtaining unit 1120, the position and orientation of the moving object obtained by the position and orientation information obtaining unit 1140, and the control value determined by the determining unit 1160. Alternatively, display information is generated based on any of the occupancy grid maps. Then, the display information is output to the display unit 140.

  In the fourth embodiment, the captured image, the shape of the conveyed object obtained by the conveyed object information obtaining unit 1120, the position and orientation of the moving object obtained by the position and orientation information obtaining unit 1140, the control value or the occupation grid determined by the determining unit 1160 Display information is generated based on any of the maps. Thereby, the user can easily confirm the state of the information processing apparatus. Further, the user inputs the control value of the AGV, various parameters, the display mode, and the like. Thus, various settings of the AGV can be easily changed or moved. As described above, by presenting the GUI, the AGV can be easily operated.

(Embodiment 5)
In the present embodiment, an information processing apparatus that recommends a user with a transport amount or a travel route that can efficiently execute a task based on a destination of a moving object and a type of a transported object will be described. Normally, for example, when the same luggage is transported several times (or several times), the total required time of the task can be reduced by loading as many luggage as possible per task. However, depending on the environment, the route that can be traveled when the number of items to be conveyed increases is limited, and as a result, the vehicle may be detoured and the required time may increase. In other words, the optimal amount and method of loading the cargo to minimize the required time can vary depending on the environment. Therefore, in the present embodiment, the optimal combination of the transport amount and the transport route is presented to the user in consideration of the size of the package and the structure of the environment. Thereby, the moving object can be controlled efficiently.

  An example of a functional configuration of the information processing apparatus 10 will be described with reference to FIG. The description of the same functional configuration as in the first embodiment will be omitted.

  The task input unit (third input unit) 1200 inputs the coordinates of the destination or task information indicating the total amount and size of the goods to be carried by the task. The coordinates of the destination included in the task information are linked to the control map held by the holding unit 1150. The total amount of goods carried by the task is also included in the task information. Also, it is assumed that the cargo information indicating the three-dimensional shape data indicating the size per unit of the cargo is known. For example, it indicates that the conveyed object is a cube or a horizontally long rectangular parallelepiped, and that it is conveyed by 10 units. In addition, the user may input a transport amount (for example, for four units) to be transported to the moving body in one process using the UI or the like. In addition, the user may specify how to load the goods. For example, when four cubic goods are mounted on a moving object, the user inputs a method of stacking, such as flat stacking, 2 × 2 stacking, or all stacking.

  The candidate acquiring unit (third acquiring unit) 1220 acquires at least one candidate of a combination of the transport amount and the moving route based on the task information and the control map. The candidate of the combination of the transport amount and the movement route is, for example, the following information. The combination of route 1 with a transport amount of 4 requires 50 seconds, and the combination of route 1 with a transport amount of 5 requires 1 minute. These pieces of information acquire the combination of the transport amount and the travel route in consideration of the size of the transported object and the structure of the traveling environment of the moving object. When a plurality of goods are carried, the shape changes depending on the method of stacking, and thus a plurality of variations of the method of stacking are also acquired.

  The generating unit 1270 generates display information for displaying the candidates acquired by the candidate acquiring unit 1220 on a display or the like. When recommending the transport amount to the user, for example, the transport amount with the shortest required time is recommended. In addition, the transport amount may be fixed, and a method of loading or a route candidate may be recommended. When recommending the stacking method, the route and the required time are associated with each stacking method candidate (for example, flat stacking, stacking, etc.). When recommending a route, an optimal carrying amount or loading method is linked to each route candidate. For example, for a short product or a flat product, a route with a partially lower ceiling can be recommended. In a route where the ceiling is partially low, the movable bodies that can travel are limited, and congestion among a plurality of movable bodies may be avoided. For example, such information is presented to the user in a GUI as shown in FIG. G160 is an image showing the estimated value of the shape of the conveyed object according to the carried amount (conveyed amount) of the conveyed object. For example, when the transport amount is four, it is assumed that it is mounted in a shape as indicated by the hatched portion in the figure. A route candidate is obtained using the shape of the conveyed article. G171 is a map indicating a destination (G172) in the control map and a route corresponding to the candidate of the transport amount. The map is, for example, the above-described control map (occupancy grid map). As an example of the route, R1 is a route having a sufficient width and height of the passage. R2 is a route that is sufficiently high but narrower than R1. R3 is the route with the smallest width and height of the passage. G181 is a list indicating combinations of transport amount candidates and routes. Furthermore, the predicted value of the required time corresponding to the transport amount and the route is displayed.

  The determination unit 1260 determines a control value based on a predetermined criterion from a candidate combination of the transport amount and the movement route. The predetermined criterion is, for example, a criterion for determining a combination that requires the shortest time. The control value may be determined according to an instruction from the user. Alternatively, the user may determine the control value by selecting a preferable combination of the transport amount and the route from a list such as G181 in FIG. If necessary, the control value is determined based on the consistency with the information (first information) on the actually loaded cargo from the cargo information acquisition unit 1120.

  The processing executed by the information processing system 1 will be described with reference to FIG. In S1410, the information processing system 1 initializes the current position of the moving object, the information on the transported goods, the control map, and the task information. In S1420, the task input unit 1200 inputs the coordinates of the destination or the task information indicating the total amount and size of the goods to be carried by the task. Here, it is assumed that the coordinates of the destination are input to the task input unit 1200 by the user. The destination information may be acquired from the process management system 14 by a communication unit (not shown). Here, the user inputs task information indicating the total amount of the goods. In S1430, the candidate acquiring unit 1220 acquires at least one or more combinations of the transport amount and the movement route based on the task information and the control map. First, the amount of the load that can be mounted on the moving object is estimated based on the load information acquired in S1410. The cargo information is three-dimensional shape data per unit of the cargo. It is assumed that the three-dimensional shape of the bed of the moving body is known. The candidate acquiring unit 1220 estimates the maximum transport amount based on the three-dimensional shape of the carrier of the moving object and the three-dimensional shape per unit of the transported object. Further, according to the amount of the conveyed goods (from one unit of the conveyed goods to each maximum conveyed amount), the method of stacking the conveyed goods and its shape are estimated. From the estimated three-dimensional shape, a route on which a moving object that carries the conveyed object by X is estimated based on the task information and the control map. Then, the required time for traveling on the route is acquired. In S1440, the generation unit 1270 generates display information for displaying the combination acquired by the candidate acquisition unit 1220, specifically, the movement route on a display or the like. For example, an image that estimates the appearance of a loaded object or a moving route from the current position of the moving object to the destination is displayed. In S1450, determination unit 1260 determines a control value based on a predetermined criterion from candidates for the combination of the transport amount and the movement route. When there are a plurality of candidates for a certain transport amount, a transport amount satisfying a predetermined condition is determined from the candidates. For example, if the user wants the mobile to travel on a route that can be traveled in the shortest time, the condition is set to time, and a combination of the transport amount and the route that minimizes the total required time of the task is determined. A control value is obtained based on the determined transport amount and the moving route. It should be noted that the transport amount or the route may be determined from the combination candidates indicated by G181 according to the input by the user. In S1460, the load sensor 110 captures an image of the load mounted on the moving object. In S1470, the transported article information acquisition unit acquires transported article information (first information) based on the image of the transported article captured in S1460. In S1480, the cargo information acquisition unit compares the first information with the determined combination of the cargo and the control value, and checks whether the cargo determined in S1450 is loaded. If it can be confirmed that the designated transported article is correctly placed, the process proceeds to S1490. If there is a difference between the determined amount of the load and the amount of the load obtained from the first information that is larger than a predetermined value (for example, more or less than one unit of the load), the process returns to S1450. In S1490, actuator section 130 causes the moving body to travel according to the control value corresponding to the determined route. By the processing described above, the amount and the route of the goods to be loaded on the moving body can be appropriately combined, so that the moving body can be efficiently controlled.

(Other modifications)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  The environment image input unit according to the present invention may be any unit that inputs information on the surrounding environment acquired by a sensor mounted on a moving object. The input surrounding environment information is at least one of a normal image, a parallax image, a depth map in which a depth value is stored in each pixel of the image, and a point cloud that holds three-dimensional coordinates of scene points with respect to the imaging device. One or more. In addition to this, three-dimensional information measured by a three-dimensional measuring device such as a LiDAR or a TOF camera, a target position and a posture of the moving object received from the moving object management system via the communication I / F, and a position and posture of the moving object May be entered. In addition, parameters of the imaging device such as the focal length, the lens center, and the lens distortion can be input to the information processing device.

  The position and orientation information acquisition unit in the present invention may be any unit as long as it acquires the position and orientation of the moving body using the information on the surrounding environment input by the environment image input unit. It may be obtained based on an ICP algorithm using a three-dimensional point group representing a scene in an environment and a depth map input by an environment image input unit, or may be obtained from a grayscale image using a SLAM technique. . Further, the position and orientation may be obtained by using the communication status of Wi-Fi communication and the beacon installed in the environment.

  The holding unit in the present invention may be any holding unit as long as it holds a control map to be referred to for controlling the moving object. The control map may be any map that represents the structure and space of the environment. For example, an occupancy grid map that divides a scene into grids and holds the probability that an obstacle exists in each grid. The occupancy grid map may be held as a three-dimensional voxel space (X, Y, Z) in the world coordinate system, or may be held as a two-dimensional grid space (X, Y) excluding height information. You may. Alternatively, similarly to the position and orientation estimation map, for example, a set of three-dimensional point group data (combination with color information) and key frame data (combination of depth map and color information linked to the position and orientation in the environment) Alternatively, a three-dimensional model of the environment may be used.

  The transported object image input unit according to the present invention may be of any type as long as the shape and / or position and / or object type information of the transported object transported by the moving object is input. The shape of the conveyed item is, for example, general shape information expressed by a three-dimensional bounding box (three parameters indicating the position of the center of gravity and three parameters indicating the size of the bounding box) of the conveyed item. Further, it may be represented by three-dimensional point cloud data (combination of color information) and three-dimensional mesh data. Moreover, you may represent as an occupancy grid map like a control map.

  The determining unit in the present invention may be any unit as long as it determines a control value for controlling the movement of the moving object. For example, based on the occupancy grid map held by the holding unit and the shape of the load input by the first input unit, a control value is determined so that the load does not contact the surrounding environment. In determining the control value, the control value is determined by determining and extracting a control map in a range where there is a possibility that the conveyed object may come into contact with the surrounding environment. Alternatively, the control value is determined using a two-dimensional control map obtained by projecting the three-dimensional control map onto the floor plane.

  The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or an apparatus via a data communication network or various storage media. Then, the computer (or CPU, MPU, or the like) of the system or apparatus reads out and executes the program. Further, the program may be provided by being recorded on a computer-readable recording medium.

12 Mobile system 15 Cargo

Claims (20)

An information processing device that determines a control value for controlling a position of a moving object that conveys a load,
The first information that can specify the three-dimensional shape of the transported object based on the first image of the transported object, and the second image that captures the environment in which the moving object moves, Acquiring means for acquiring second information capable of specifying a distance between an object and the moving body;
Information processing means for determining, based on the first information and the second information, the control value for preventing the conveyed object and the object from approaching more than a predetermined distance. apparatus.   2. The information processing apparatus according to claim 1, wherein the obtaining unit obtains the first information based on a learned model that outputs the size of the conveyed object using the image as an input. 3. The obtaining means obtains a position of the moving body based on the second image,
The information processing apparatus according to claim 1, wherein the determining unit determines the control value for reaching a predetermined destination position from the position of the moving body.   The information processing apparatus according to claim 1, wherein the first information is information indicating a size of the conveyed object in a height direction.   The information processing apparatus according to claim 1, wherein the determining unit determines the predetermined distance based on a height of a step existing on a traveling route in the environment.   The information processing apparatus according to claim 1, wherein the first information is information indicating a size of the conveyed material protruding from the moving body in a width direction.   The information processing apparatus according to claim 1, wherein the first image is an image captured by an imaging device mounted on the moving body.   7. The image according to claim 1, wherein the first image is an image captured by an imaging device installed at a position where the moving object and the transported object can be observed. 8. Information processing device.   The information processing apparatus according to claim 1, wherein the second image is an image captured by an imaging device mounted on the moving body. Further, holding means for holding a map indicating the position of the object in the environment,
The information processing apparatus according to claim 1, wherein the determining unit determines the control value based on the map such that the control value travels on a route where the object does not exist. The map is a map expressing the environment in three dimensions,
The said determination means determines the said control value based on the said map extracted according to the height of the said conveyed object so that it may drive | work on the path | route where the said object does not exist. Information processing device. The acquiring means acquires information indicating a type of the conveyed article based on the first image,
The information processing apparatus according to claim 1, wherein the determining unit determines the control value according to the type of the conveyed article based on information indicating the type of the conveyed article. apparatus. The acquiring means updates the first information according to a change in the first image,
The said determination means determines the said control value according to the shape of the said conveyed article, when the shape of the said conveyed article changes based on the updated 1st information. The information processing device according to any one of claims 12 to 12.   14. The display device according to claim 1, further comprising a display unit configured to display display information indicating a moving route of the moving object based on the control value determined by the determining unit. Information processing device. The acquiring means further acquires a three-dimensional map of the environment extracted according to the height of the conveyed article, based on the first information,
The information processing apparatus according to claim 1, wherein the determining unit determines the control value based on the three-dimensional map. An information processing device that determines a control value for controlling a position of a moving object that conveys a load,
Based on information that can specify a three-dimensional shape of the article estimated from an image captured by an imaging device mounted on the moving object and a map of an environment in which the moving object moves, Acquisition means for acquiring a combination of the amount and the travel route;
An information processing apparatus, comprising: a determination unit configured to determine, based on the combination, the control value that travels on the movement route on which the transported object does not collide with an object in the environment. An information processing device that determines a control value for controlling a position of a moving object that conveys a load,
The first information that can specify the three-dimensional shape of the transported object based on the first image of the transported object, and the second image that captures the environment in which the moving object moves, Acquiring means for acquiring second information capable of specifying a distance between an object and the moving body;
An information processing apparatus, comprising: a determination unit configured to determine the control value for traveling on a route where the conveyed object does not collide with the object based on the first information and the second information.   A program for causing a computer to function as each unit included in the information processing apparatus according to claim 1. An image pickup apparatus and an information processing system having an information processing apparatus that determines a control value for controlling a position of a moving object that carries a conveyed object,
The imaging device is installed at a position where the moving body and the transported object can be observed,
The information processing device,
A second image that captures an environment in which the moving object moves by acquiring first information that can specify a three-dimensional shape of the load based on a first image obtained by capturing the load by the imaging device; Acquiring means for acquiring second information capable of specifying a distance between the object and the moving object in the environment, based on
An information processing apparatus comprising: a determination unit that determines, based on the first information and the second information, a control value that suppresses the transported object and the object from approaching a predetermined distance. system. An information processing method for determining a control value for controlling a position of a moving object that transports a load,
Obtaining first information capable of specifying the three-dimensional shape of the conveyed object based on the first image of the conveyed object, based on a second image of an environment in which the moving object moves, An obtaining step of obtaining second information capable of specifying a distance between the object and the moving body in the environment,
A step of determining, based on the first information and the second information, the control value for preventing the transported object and the object from approaching a predetermined distance. Processing method.

Images