JP2021017324A - Robot system, information processing device, control method of the same, and program - Google Patents

Abstract

To quickly correspond to a display state that causes sales opportunity loss.SOLUTION: A carrier robot, comprising a commodity loading part for loading commodities and an automatic traveling function, and an information processing device are communicably connected, in a robot system. The information processing device acquires a position where there is a problem in preparation of displayed commodities, and according to the position, transmits it as a carrying destination of the commodities to be taken out. The carrier robot carries a loading carrier part, which has received the carrying destination and has loaded the commodities, to the received carrying destination by the automatic traveling function.SELECTED DRAWING: Figure 1

Description

Related to robot systems, information processing equipment, their control methods and program technology.

In the retail industry, in order to solve the recent labor shortage problem, a mechanism has been demonstrated in the market that allows products to be automatically settled and purchased without human intervention, such as unmanned stores using network cameras.

Patent Document 1 discloses a technique of detecting the position of a product based on a photographed image of a display area and evaluating the current display state using a plurality of evaluation indexes related to the disorder of the product from the detection result.

JP-A-2016-207164

With the technique of Patent Document 1, even if the display state can be presented to the store staff, if it is not possible to deal with it immediately without taking a hand for another job, a loss of sales opportunity will result.

An object of the present invention is to provide a mechanism capable of quickly responding to a display state that causes a loss of sales opportunity.

The present invention is a robot system in which a product loading unit on which a product is placed, a transfer robot having an autonomous traveling function, and an information processing device are communicably connected to each other, and the information processing device is an assortment of displayed products. The transport robot includes a receiving means for acquiring a position having a problem in the above position and a transmitting means for transmitting the product as a transport destination of the product to be delivered according to the position. It is characterized by comprising a transport control means for transporting the loaded product loading unit to the received transport destination by the autonomous traveling function.

The present invention communicates with a transport robot including an autonomous travel function, a receiving means for receiving a transport destination, and a transport control means for transporting a product loading unit on which a product is placed to the received transport destination by the autonomous travel function. An information processing device that is connectable and has a problematic position in the assortment of displayed products, and a transmission means that transmits the product as a transport destination according to the position. It is characterized by being prepared.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a mechanism for promptly responding to a display state that causes a loss of sales opportunity.

It is a front view which the circulation robot, the transfer robot, and the product loading part of the present invention are viewed from the direction facing the product shelf. It is a side view which shows the relative positional relationship between a circulation robot and a product shelf in the case of confirming an assortment of products by circulating in a store of this invention. It is a side view which shows the relative positional relationship between a transport robot and a product shelf when a product to be put out is loaded on a product loading section and arrives at a transport destination, according to the present invention. It is a side view which shows the relative positional relationship with the product shelf when the product loading part is separated from a transfer robot of this invention. The circulation robot of the present invention circulates in the store to check the product assortment, and the transfer robot places the product having a problem in the product assortment from the backyard onto the product loading section, and the product shelf having a problem in the product assortment ( It is a figure which showed the map in the store which transports a product loading part to the vicinity of a transport destination). In the images taken by the circulation robot of the present invention to circulate in the store and determine the assortment, there are missing items on the product shelves, or the number of products displayed is less than when the store was opened. It is a figure which showed the image example which is judged that there is a problem in the current product lineup. In the images taken by the circulation robot of the present invention to circulate in the store and determine the assortment, the number of products displayed on the product shelves is more than that at the time of opening. It is a figure which showed the image example which it is judged that there is no problem in the present product lineup because it has not decreased. It is a figure which shows the flowchart which shows the order of processing performed when the circulation robot of this invention circulates in a store. It is a figure which shows the flowchart which shows the control order of the photographing process of the product shelf executed by the circulation robot of this invention. It is a figure which shows the flowchart which shows the control order of the assortment determination process of the display product executed by the circulation robot of this invention. It is a figure which shows the flowchart which shows the control order of the transmission process of the transport destination which transports the goods to be taken out executed by the circulation robot of this invention. It is a figure which shows the flowchart which shows the control order of the process performed when the transfer robot of this invention conveys a product to be taken out from a backyard to a transfer destination. It is a figure which shows the flowchart which shows the order of the receiving process of the transport destination which transports the goods to be taken out performed by the transport robot of this invention. It is a figure which shows the flowchart which shows the order of the product take-out traveling process performed by the transfer robot of this invention. It is a figure which shows the flowchart which shows the order of the return traveling process performed by the transfer robot of this invention. It is a figure which shows the system block diagram of the robot system of this invention. It is a figure which shows the content which is displayed on the monitor screen of the transfer robot of this invention. It is a figure which shows the hardware block diagram of the information processing apparatus incorporated in the circulation robot, the transfer robot, and the in-store controller used in the robot system of the present invention. It is a figure which shows the functional block diagram of the information processing apparatus incorporated in the circulation robot, the transfer robot, and the controller in a store used in the robot system of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 will be described.

FIG. 1 is a front view of the circulation robot 300, the transfer robot 310, and the product loading unit 320 of the present invention as viewed from the direction facing the product shelf 411.

The circulation robot 300 periodically circulates in the store during business hours by using the autonomous traveling function, and when a problem is found in the product assortment on the product shelf 411 in the store, there is a problem in the product assortment. The position of the product shelf and the like can be notified to the transfer robot 310.

The circulation robot 300 can freely change the shooting direction of the camera 301 for photographing the product shelf 411, the pillar 302 for installing the camera 301 at a fixed position, the loading platform 303 for autonomously moving the set of cameras 301, and the camera 301. It is provided with wheels 304 for autonomously traveling the pan / tilt unit 315 and the loading platform 303 in the store.

The shooting range (viewing angle) of the camera 301 that shoots the product shelf 411 with one image can be freely shot in the left-right direction of the product shelf 411 by changing the pan angle of the pan / tilt unit 315 by several steps. By changing the tilt angle of the pan / tilt unit 315, it is possible to take a picture in the vertical direction of the product shelf 411.

In addition, the shooting range (viewing angle) of the camera 301 that shoots the product shelf 411 with one image is the average viewing angle (about 90 ° to about 120 °) of the human eye so as to appropriately reproduce the customer's line of sight. , It is set in almost the same way. In addition, the zoom value (angle of view) of the camera 301 that captures the product shelf 411 with one image is also the average of the human eye so as to appropriately reproduce the customer's line of sight from the relative distance between the camera 301 and the product shelf 411. The zoom value is set to be almost the same as the sense of distance.

Further, when the circulation robot 300 pauses at a certain point in front of the product shelf 411 on the route, the pan angle is changed to take an image in multiple directions, and the tilt angle is changed to take a camera image 600 in multiple directions. To do.

From the camera image 600 of the obtained assortment of products in one direction, it is determined whether or not there is a problem with the assortment of the photographed products, and the result is (1) whether or not there is a problem with the assortment of products. In addition to the flag information indicating whether or not, (3) information indicating the exact position of the location in front of the product shelf 411 on the route using the GPS (position information) of the circulation robot 300 itself, (4) pan angle. Information indicating the orientation of the part where the product lineup was confirmed from the camera 301 using the camera 301 and the height for calculating the number of product shelves, (5) the date and time of shooting, and the product lineup problem. The four times when it is confirmed whether or not there is a product are uniquely associated with the camera image ID of one captured image, added to the EXIF information and the like, and recorded and saved.

The transport robot 310 was delivered by the store staff in the backyard to the vicinity of the product shelf (transport destination) where a problem was found in the product lineup on the product shelf using the autonomous traveling function in the store during business hours. The product loading unit 320 on which the product is placed can be towed and transported.

The transport robot 310 provides product delivery information such as the type of product to be delivered, the position of the product shelf to which the store staff is to deliver, the quantity to be delivered, and the current position of the back yard. A display monitor 305 that can be attached to and detached from the transfer robot 310 for displaying 1701, a pillar 306 for installing the display monitor 305 in a fixed position, a loading platform 307 that can move a set of the display monitor 305 autonomously, and a loading platform 307. The wheel 308 that autonomously travels in the store, and the connecting portion 309 that turns on / off the connecting state with the product loading portion 320 to be pulled are provided.

The connecting portion 309 is connected to the product loading unit 320 in the backyard, and is for disconnecting the product loading unit 320 from the transport robot 310 when the product loading unit 320 can be transported to the transport destination.

The product loading unit 320 is a trolley that carries a distribution crate 312 or the like loaded with supplementary products 311 that the store staff has put out in the backyard.

The product loading unit 320 includes a loading platform 313, wheels 314, and a connected unit (not shown) capable of turning ON / OFF the connection by the connecting unit 309.

A distribution crate 312 or the like may be placed on the loading platform 313, and any form may be used as long as it is a trolley capable of transporting the replenished product 311 that has been put out.

FIG. 2 will be described.

FIG. 2 is a diagram showing a situation in which the circulation robot 300 circulates in the store and confirms the assortment of products.

The circulation robot 300 automatically travels in a direction that does not intersect with the front of the product shelf 411 while the camera 301 that photographs the product shelf 411 faces the front of the product shelf 411.

A view of the circulation robot 300 viewed from the direction side where it does not intersect is shown, and a view seen from the side surface side is shown.

The position where the camera 301 of the circulation robot 300 is fixed is 170 cm, which is the same as the height of an average Japanese man, to 150 cm, which is the same as the height of a smaller Japanese woman, so that it matches the height of the eyes of customers visiting the store. In this embodiment, it is installed at a height of about 160 cm in the middle. This is the fixed height (160 cm) of the camera 301 and the distance between the camera 301 and the product shelf 411 (in order to realistically reproduce how the product lineup can be seen from the customer's line of sight. An autonomous driving route is set in advance so that the circulation robot automatically travels on a route (50 cm to 100 cm) that is the same as the position of the customer's line of sight looking around the store during shopping.

Further, the position where the camera 301 is fixed can be appropriately set to an optimum height such as 100 cm to 120 cm according to the average height according to the age group and nationality of the customers who visit the store.

For example, if the number of product shelves 411 is four and there is no problem with the product lineup, product 412 (manufactured by manufacturer A), supplementary product 311 (manufactured by manufacturer B), and product 413 (manufactured by manufacturer C) are ordered from the top shelf. ) Products 414 (manufactured by D manufacturer) are displayed separately by manufacturer, but in Fig. 2, the second-tier replenishment product 311 (manufactured by B manufacturer) from the upper shelf is out of stock, and there is a problem with the product lineup. Represents a situation.

FIG. 3 will be described.

FIG. 3 is a diagram showing a situation in which the transport robot 310 loads the product on the product loading section, pulls it, and arrives at the transport destination.

The transfer robot 310 autonomously travels in a direction that does not intersect the front of the product shelf 411.

A view of the transfer robot 310 as viewed from the non-intersecting direction side is shown, and a view of the transfer robot 310 as seen from the side surface side is shown.

In response to the shortage of the replenishment product 311 (manufactured by B manufacturer) in the second stage from the upper shelf discovered by the circulation robot 300 shown in FIG. 2, the replenishment product 311 (B) was put out by the store staff in the backyard. This is the situation after the product loading unit (manufactured by the manufacturer) has been transported to the front of the product shelf 411 on which the replenishment product 311 (manufactured by the manufacturer B), which is an appropriate destination, is displayed.

FIG. 4 will be described.

FIG. 4 is a diagram showing a situation in which the transport robot 310 disconnects the product loading section on which the product is loaded and only the product loading section on which the product is loaded is left at the transport destination. At this time, the transport robot 310 is in a situation where the product loading unit 320 is separated from the transport destination and returned to the transport source (backyard) by the autonomous traveling function.

After that, the store staff manually displays the replenishment product 311 placed on the product loading unit 320 on the product shelf, and the problem of product assortment is solved. Further, the empty product loading unit 320 from which the replenishment product 311 has disappeared is manually returned to the backyard by the store staff.

FIG. 5 will be described.

FIG. 5 is an in-store map in which the circulation robot 300 circulates in the store to check the product lineup, and the transfer robot 310 puts the product on the product loading section from the backyard and transports it to the vicinity of the product shelf (transport destination). It is a figure which showed 400.

In this example, the in-store layout of a convenience store is shown, but the layout is not limited to the convenience store, and any in-store layout for retailing and selling products such as food and clothes may be used.

In the figure, → is a traveling route in which the circulation robot 300 periodically circulates in the store (for example, in the morning and afternoon). The circulation of the circulation robot 300 is performed during business hours, the product on the product shelf having a problem in the assortment is found, and the position information of the product shelf having the problem in the assortment is immediately transferred to the transfer destination by the transfer robot 310. It is sent as (destination).

By automatically delivering the transfer destination (destination) transported by the transfer robot 310 to the transfer robot 310 in this way, it is not necessary to manually set the transfer destination (destination) of the transfer robot 310 waiting in the backyard. It can be omitted and business productivity is improved.

Further, by linking the automatic distribution of one circulation robot 300 and a plurality of transfer robots 310, it is possible to quickly and automatically set an accurate transfer destination to each transfer robot 310.

The transfer robot 310 is connected to the product loading unit 320 in the backyard, transports the replenishment product 311 to the front or the vicinity of the product shelf having a problem in the product lineup, and transports the product loading unit 320 as shown in FIG. Leave first and return to the backyard.

FIG. 6 will be described.

In FIG. 6, in the image taken by the circulation robot 300 to circulate in the store and determine the product lineup, the number of products displayed on the product shelves is less than that at the time of opening. Therefore, it is a figure which showed an example of the camera image 600 of the case which it is determined that there is a problem in an assortment of goods.

In the evening and after the bargain sale, the product lineup is in a situation where the product shelves are out of stock or the number of products on display is less than when the store was opened. If there is a problem with the current product lineup, there is a greater possibility that the customer will have less chance to pick it up and consider purchasing the product, so return to the product lineup at the time of opening or immediately after the product is released as much as possible and see from the customer's perspective. It is important to restore the alignment.

FIG. 7 will be described.

In FIG. 7, in the image taken by the circulation robot 300 to circulate in the store and determine the assortment, the number of products displayed on the product shelves is the same as when the store is opened. It is a figure which showed an example of the camera image 500 of the case which it is judged that there is no problem in an assortment because it is properly replenished. It is also a situation in which the replenishment product 311 transported to the front of the product shelf shown in FIG. 4 is manually replenished to the product shelf by the store staff.

Next, the processing executed by the robot system in the present embodiment will be described with reference to the flowcharts of FIGS. 8 to 15.

FIG. 8 will be described.

FIG. 8 is a flowchart showing the order of each process performed when the circulation robot circulates in the store. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S101, the autonomous movement route that circulates automatically in the store is compared with the current position of the circulation robot 300 using GPS, and it is determined whether or not the autonomous driving of the autonomous movement route is completed. If it is finished (YES), the whole process is finished. If it does not end (NO), the process proceeds to S102 and a process of taking a picture for confirming the assortment of products is performed in front of the product shelf.

In step S102, a process of photographing the product shelves is performed in front of the product shelves.

In step S103, a process of determining whether or not there is a problem in the product assortment is performed from the camera image.

In step S104, a process of transmitting the position of the product shelf determined to have a problem in the product assortment as the transport destination of the product to be delivered is performed.

FIG. 9 will be described.

FIG. 9 is a sub-flow chart showing the order of shooting processing of the product shelves performed by the circulation robot 300. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S201, the vehicle autonomously travels on an autonomous movement route that automatically circulates in the store registered in advance, and starts moving to the front of the product shelf 411 in the store (passage shown by → in FIG. 3). ..

In step S202, it is determined whether or not the current traveling position is in front of the product shelf 411 for which the preset product assortment should be confirmed. If (YES) is in front of the preset product shelf 411, the process proceeds to step S203. If it is not in front of the preset product shelf 411 (NO), the process returns to step S201.

In step S203, the movement is temporarily stopped in front of the preset product shelf 411. It is desirable that this stop position is the center position with respect to the width direction of each product shelf 411 to be confirmed, but the stop position where the product lineup of the product shelf 411 to be confirmed in the store can be photographed from the customer's line of sight. It should be.

In step S204, the shooting range of the camera 301 that shoots the product shelf 411 is changed, and camera images 600 in a plurality of directions (for example, 3 directions in the left middle right × 9 directions in the upper middle and lower directions) are taken and taken. The camera image 600 is recorded in the image recording unit 211 of the circulation robot 300.

At this time, an image ID is assigned to the camera image 600, and (3) GPS is used to obtain information indicating the exact position of the product shelf 411 on the route, and (4) pan angle and tilt angle. Information that indicates the orientation of the location and the height for calculating the number of product shelves used, (5) the date and time of shooting, and the time when it was confirmed whether there was a problem with the product lineup are used in EXIF information, etc. It is added and recorded and saved.

The position in front of the product shelf 411 in the store (the destination of the product to be delivered) can be specified from the (3) GPS information saved here, and the product from the camera 301 can be specified from the information in (4). The orientation information up to the shelf 411 and the height information of the product shelf 411 can be specified, and the time when it is confirmed whether or not there is a problem in the product assortment can be specified from the information in (5).

In step S205, it is determined whether or not all the determined shooting directions have been shot. If the shooting is finished (YES), the shooting process is finished. If the shooting is not completed (NO), the process returns to step S204.

FIG. 10 will be described.

FIG. 10 is a sub-flow chart showing the order of the assortment determination processing of the displayed products performed by the circulation robot. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S301, one camera image 600 in a certain direction taken in step S204 is read out at this stop position.

In step S302, the read camera image 600 is analyzed, and as the input information, each image feature amount of each of the displayed products 421 to 414 groups (detection of product rectangle, identification of product type, identification of background type, By extracting the relative distance between products, the relative color between products, and the difference between product types), there are no shortages on the product shelves, or the number of products on display has not decreased. Using a trained model that pre-learned the state, etc., where there is no problem with the product assortment as at the time of opening, as output information, from the image feature quantity of the extracted product 421-414 group, the current product Determine if there is a problem with the assortment.

If there is a problem (YES) in the current product lineup as output information, the process proceeds to S303. If there is no problem with the current product lineup (NO), proceed to S304.

In step S303, the read camera image is recorded and saved by adding (1) flag information indicating that there is a problem in the current product assortment to the EXIF information or the like in the image ID.

In step S304, the read camera image is recorded and saved by adding (1) flag information indicating that there is no problem in the current product lineup to the EXIF information or the like to the image ID.

Next, returning to the circle 1, at this stop position, one camera image 500 in another direction taken in step S204 is read out. This loop is repeated with 600 groups of camera images in multiple directions taken at this stop position. The process ends when there is no more.

FIG. 11 will be described.

FIG. 11 is a sub-flow chart showing the order of transmission processing of the destinations of the goods to be delivered by the circulation robot. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S501, at this stop position, in step S303, (1) a flag information indicating that there is a problem with the current product assortment is added to the EXIF information and the like, and one camera image 600 recorded and saved is read out. ..

In step S502, the positions of the detailed parts of the product shelf having a problem in the current product assortment are acquired by using the information of (3) and (4) of the image ID written in step S204.

When acquiring the position of the detailed part of the product shelf, the GPS of (3) is used to use the information indicating the exact position of the part in front of the product shelf 411 on the route.

As another modification, the orientation information of the surveillance camera installed so as to face a fixed direction on the ceiling in the store without using the camera 301 of the circulation robot, and the linear distance from the surveillance camera to each product shelf. You may use the position of the detail part for each product shelf previously associated with the camera image 600 taken by the surveillance camera, which is derived from.

In step S503, the position of the detailed portion acquired in step S502 is sequentially transmitted to the transfer robot 310 via the in-store controller 100 or directly. At this time, at the same time, the type of product having a problem in the product assortment and the number of missing products are also transmitted.

If the communication mode is received by the in-store controller 100, the in-store controller 100 has the position of the detailed portion acquired in step S502, the type of product having a problem in the product assortment, and the number of missing products. May be relayed to the transfer robot 310.

Next, returning to the circle 3, at this stop position, in step S303, (1) another flag information indicating that there is a problem with the current product assortment is added to the EXIF information and the like, and is recorded and saved. One camera image 600 is read out. This loop is repeated until there are no more camera image 600 groups to which the flag (1) is attached. The process ends when there is no more.

FIG. 12 will be described.

FIG. 12 is a flowchart showing the order of each process performed when the transport robot transports the product to the transport destination. Each processing step is executed by the CPU 201 of the transfer robot 310.

In step S601, the product delivery to be performed now is based on the information on whether or not the transportation of all the products to be delivered has been completed (corresponding) in the product list to be delivered displayed on the screen of the display monitor 305. Determine if the transport is complete. If it is finished (YES), the whole process is finished. If it does not end (NO), the process proceeds to S602 to perform the process of receiving the transport destination.

In step S602, a process of receiving the position of the product shelf determined to have a problem in the product assortment as the transport destination of the product to be delivered is performed.

In step S603, a process of autonomously traveling to the transport destination is performed in order to transport the product to be delivered.

In step S604, an autonomous traveling process for returning to the transport source is performed.

FIG. 13 will be described.

FIG. 13 is a diagram showing a flowchart showing the order of receiving processing of the delivery destination of the product to be delivered by the transfer robot 310. Each processing step is executed by the CPU 201 of the transfer robot 310.

In step S701, the position of the detailed part of the product shelf having a problem in the current product assortment, which was transmitted in step S503, is relayed through the in-store controller 100 or directly and sequentially received. At the same time, the type of product having a problem in the product assortment and the number of missing products are also received.

In step S702, the position of the detailed part of the product shelf having a problem with the current product assortment, the type of the product having a problem with the product assortment, the number of missing items, the reception time, and the product delivery status (FIG. 17). The display is controlled so that the content shown in 1701) is displayed on the display monitor 305.

In step S703, it is determined whether or not the connection with the carriage 320 is completed based on whether or not the connection lock button 1703 is pressed. If it is not completed (NO), the process returns to step S702. If completed (YES), the process proceeds to S704.

In step S704, the connecting portion 309 is turned on, and the connecting portion 309 connects with the carriage 320.

In step S705, it is determined whether or not to start autonomous traveling depending on whether or not the transfer start button 1702 is pressed. If it does not start (NO), the process returns to step S704. If it starts (YES), the process proceeds to step S706.

The transport start button 1702 is a receiving means for receiving a transport instruction from the backyard to the transport destination.

In step S706, the place where the transport start button 1702 is pressed is acquired by GPS and registered in the storage unit as the backyard product delivery start location (transport source).

FIG. 14 will be described.

FIG. 14 is a diagram showing a flowchart showing the order of product delivery travel processing performed by the transfer robot. Each processing step is executed by the CPU 201 of the transfer robot 310.

In step S801, the vehicle autonomously moves from the transport source to the transport destination displayed in step S702 and designated by the store staff as the destination (transport control).

In step S802, it is determined whether the current traveling position (GPS) is in front of or near the product shelf having a problem with the current product assortment displayed in step S702. If it is forward (YES), the process proceeds to step S803. If it is not forward (NO), the process returns to step S801.

In step S803, the movement is stopped in front of the product shelf 411 having a problem in the current product assortment. This stop position is preferably the center position with respect to the width direction of the product shelf 411 having a problem in assortment, but in the vicinity of the product shelf 411 having a problem in assortment (for example, around within 5 m). Any safe stop position with no obstacles will do.

In step S804, it is determined whether or not the disconnection with the carriage 320 is completed based on whether or not the connection unlock button 1704 is pressed. If it is not completed (NO), the process returns to step S803. If it is completed (YES), the process proceeds to S805.

In step S805, the connecting portion 309 is turned off, and the connecting portion 309 releases the connection with the carriage 320. As a result, the carriage 320 is separated from the transfer robot 310 (disengagement control).

In step S806, it is determined whether or not to start autonomous driving depending on whether or not the return start button 1705 is pressed. If it does not start (NO), the process returns to step S805. If it starts (YES), the process ends.

FIG. 15 will be described.

FIG. 15 is a diagram showing a flowchart showing the order of return traveling processes performed by the transfer robot.

In step S901, the return from the place where the return start button 1705 is pressed is autonomously returned to the destination registered in step S706 (return control).

In step S902, it is determined whether the current traveling position (GPS) is the transport source registered in step S706. If it is the transport source (YES), the process proceeds to step S903. If it is not the transport source (NO), the process returns to step S902.

In step S903, the movement is stopped at the transport source to end the process.

FIG. 16 will be described.

FIG. 16 is a system configuration diagram showing an example of the configuration of the robot system of the present invention.

The robot system is configured such that a circulation robot 300, a transfer robot 310, an in-store controller 100, a data management cloud 200, and a wireless communication router 101 are connected via an Internet line or a LAN.

Further, the circulation robot 300, the transfer robot 310, the in-store controller 100, and the wireless communication router 101 are arranged in the store.

The data management cloud 200 also stores the camera image 600 taken by the circulation robot 300, the EXIF information in the camera image 600, and the learned model installed in the circulation robot 300 for backup. As appropriate, the latest trained model additionally trained with the new teacher image is installed in the circulation robot 300. In addition, the data management cloud 200 collects new teacher images from a large number of stores and improves the determination accuracy of the out-of-stock determination on a daily basis.

The data management cloud 200 has a large assortment of products from a large number of teacher images (learning datasets), image feature quantities (detection of product rectangles, specification of product types, specification of types such as shelves and backgrounds, relative between products Distance, relative color between products, difference between product types, relative distance between shelves and backgrounds, relative color between shelves and backgrounds, difference between shelves and background types, etc.) To create a trained model that has learned in advance the product lineup without any shortages (such as when the product is placed as it was when the store opened), and periodically apply the latest trained model to the circulation robot 300. Send to.

The circulation robot 300 is constantly connected to the in-store controller 100 and the data management cloud 200 by wireless communication (Wi-Fi).

The in-store controller 100 is also constantly connected to the transfer robot 310 by wireless communication (Wi-Fi).

FIG. 17 will be described.

The EXIF information for each image ID is overwritten with the following flags, position information, and date and time.

(1) Flag information indicating either that there is a problem in the product lineup or that there is no problem.

(3) Position information indicating the exact position of the place in front of the product shelf 411 on the route using GPS.

(4) Information that uses the pan angle and tilt angle to indicate the orientation information of the location and the height for calculating the number of product shelves.

(5) The date and time when the product was photographed and the time when the circulation robot confirmed whether there was a problem.

FIG. 18 is a block diagram showing a hardware configuration of an information processing device applicable to the circulation robot 300, the transfer robot 310, the in-store controller 100, the data management cloud 200, and the wireless communication router 101 according to the embodiment of the present invention. Since each device has the same configuration, the same reference numerals will be used for description.

The circulation robot 300, the transfer robot 310, the in-store controller 100, the data management cloud 200, and the wireless communication router 101 are via the system bus 204, the CPU (Central Processing Unit) 201, the ROM (Read Only Memory) 202, and the RAM (Random Access). Memory) 203, input controller 205, video controller 206, memory controller 207, communication I / F controller 208, and the like are connected. The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

Further, in the ROM 202 or the external memory 211 (external memory), in order to realize a function executed by each server or each PC, the BIOS (Basic Input / Output System) or OS (Operating System) which is a control program of the CPU 201. Various programs and the like, which will be described later, are stored. In addition, information necessary for carrying out the present invention is stored. The external memory may be a database.

The RAM 203 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing the process from the ROM 202 or the external memory 211 into the RAM 203 and executing the loaded program.

The input controller 205 controls input from a pointing device such as a keyboard (KB) 209 or a mouse (not shown). The input controller 205 also controls the camera image input from the camera device 213.

The video controller 206 controls the display on a display such as the display 210. The display may be a display such as a liquid crystal display. These are used by the administrator as needed.

The memory controller 207 is an external storage device (hard disk (HD)) for storing boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA (Personal Computer). Controls access to external memory 211 such as compact flash® memory connected via an adapter to the Memory Card International Association card slot.

The communication I / F controller 208 connects and communicates with an external device via the network, and executes communication control processing on the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

The CPU 201 can display the outline font on the display 210 by executing the outline font expansion (rasterization) process on the display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction by a mouse cursor (not shown) or the like on the display 210.

Various programs described later for realizing the present invention are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as needed.

Although one embodiment has been described above, the present invention can take an embodiment as a system, an apparatus, a method, a program, a recording medium, or the like, and specifically, is composed of a plurality of devices. It may be applied to a system or a device consisting of one device.

Further, the program in the present invention is a program in which a computer can execute the processing method of the flowchart shown in each figure, and the storage medium of the present invention stores a program in which the computer can execute the processing method in each figure. The program in the present invention may be a program for each processing method of each device in each figure.

As described above, the recording medium on which the program that realizes the functions of the above-described embodiment is recorded is supplied to the system or the device, and the computer (or CPU or MPU) of the system or the device stores the program in the recording medium. Needless to say, the object of the present invention can be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon. Disks, solid state drives, etc. can be used.

Further, by executing the program read by the computer, not only the function of the above-described embodiment is realized, but also the OS (operating system) or the like running on the computer is actually operated based on the instruction of the program. Needless to say, there are cases where a part or all of the processing is performed and the processing realizes the functions of the above-described embodiment.

Further, the program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, and then the function expansion board is based on the instruction of the program code. It goes without saying that there are cases where the CPU or the like provided in the function expansion unit performs a part or all of the actual processing, and the processing realizes the functions of the above-described embodiment.

Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. It goes without saying that the present invention can also be applied when it is achieved by supplying a program to a system or device. In this case, by reading the recording medium in which the program for achieving the present invention is stored into the system or device, the system or device can enjoy the effect of the present invention.

Further, by downloading and reading a program for achieving the present invention from a server, database, or the like on the network by a communication program, the system or device can enjoy the effect of the present invention.

It should be noted that all the configurations in which the above-described embodiments and modifications thereof are combined are also included in the present invention.

100 In-store controller 200 Data management cloud 300 Circulation robot 310 Transfer robot 320 Cart (product loading section)
411 Product shelf 421-414 Product 301 Camera 400 In-store map 500 Camera image 600 Camera image

Claims (9)

It is a robot system in which a product loading unit for loading products, a transfer robot having an autonomous traveling function, and an information processing device are connected so as to be able to communicate with each other.
The information processing device
An acquisition method to acquire the position where there is a problem with the assortment of displayed products,
A transmission means for transmitting the goods to be delivered according to the above position, and
With
The transfer robot
A receiving means for receiving the destination and
A transport control means for transporting the product loading unit on which the product is placed to the received transport destination by the autonomous traveling function.
A robot system characterized by being equipped with. The transfer robot
With a monitor
A receiving means for receiving the product type to be put out, and
A display control means for displaying the received product type to be delivered and the destination on the monitor.
With more
The robot system according to claim 1, wherein the transport control means starts transporting to the transport destination when the loading of the displayed product type to be delivered is completed. The transfer robot
A means for determining whether or not the vehicle has arrived at the destination by the autonomous traveling function,
A detachment control means for detaching the product loading unit at the transfer destination,
The robot system according to claim 1 or 2, wherein the robot system is provided with. The transfer robot
Reception means for receiving transportation instructions and
When the storage means for storing the place where the transportation instruction is received and the storage destination arrive at the transportation destination and the product loading unit is withdrawn.
Return control means for returning to the place where the transport instruction was received, and
3. The robot system according to claim 3. It is communicably connected to a transport robot having an autonomous travel function, a receiving means for receiving a transport destination, and a transport control means for transporting a product loading unit on which a product is placed to the received transport destination by the autonomous travel function. Information processing device
An acquisition method to acquire the position where there is a problem with the assortment of displayed products,
A transmission means for transmitting the goods to be delivered according to the above position, and
An information processing device characterized by being equipped with. It is a control method of a robot system in which a product loading unit for loading products, a transfer robot having an autonomous traveling function, and an information processing device are connected so as to be able to communicate with each other.
The acquisition process in which the acquisition means of the information processing device acquires a position where there is a problem in the assortment of displayed products, and
A transmission step in which the transmission means of the information processing device transmits as a transport destination of the product to be delivered according to the position.
Including
A receiving process in which the receiving means of the transfer robot receives the transfer destination,
A transfer control step in which the transfer control means of the transfer robot conveys the product loading unit on which the product is placed to the received transfer destination by the autonomous traveling function.
A control method characterized by including. It is communicably connected to a transport robot having an autonomous travel function, a receiving means for receiving a transport destination, and a transport control means for transporting a product loading unit on which a product is placed to the received transport destination by the autonomous travel function. It is a control method for information processing equipment.
The acquisition process in which the acquisition means of the information processing device acquires a position where there is a problem in the assortment of displayed products, and
A transmission step in which the transmission means of the information processing device transmits as a transport destination of the product to be delivered according to the position.
A control method characterized by including. It is a program that can be read and executed by a robot system in which a transport robot equipped with a product loading unit for loading products, an autonomous traveling function, and an information processing device are communicatively connected.
The information processing device
An acquisition method to acquire the position where there is a problem with the assortment of displayed products,
A transmission means for transmitting the goods to be delivered according to the above position, and
To make it work
The transfer robot
A receiving means for receiving the destination and
A transport control means for transporting the product loading unit on which the product is placed to the received transport destination by the autonomous traveling function, and
A program to make it work. It is communicably connected to a transport robot having an autonomous travel function, a receiving means for receiving a transport destination, and a transport control means for transporting a product loading unit on which a product is placed to the received transport destination by the autonomous travel function. A program that can be read and executed by an information processing device
The information processing device
An acquisition method to acquire the position where there is a problem with the assortment of displayed products,
A transmission means for transmitting the goods to be delivered according to the above position, and
A program to function as.

Images