JP2021000687A - Robot system, control method for the same, and program - Google Patents

Abstract

To instantaneously cope with such a display state as to cause a sales opportunity loss.SOLUTION: A robot system comprising a robot arm capable of holding a product photographs the displayed product, determines whether or not a display state of the photographed product is disordered, and re-arranges the product in the disordered display state so as to correct the disordered display state by using the robot arm.SELECTED DRAWING: Figure 1

Description

Regarding robot systems, their control methods and programming technology.

In the retail industry, in order to solve the recent labor shortage problem, a mechanism has been demonstrated in the market that allows products to be automatically settled and purchased without human intervention, such as unmanned stores using network cameras.

Patent Document 1 discloses a technique of detecting the position of a product based on a photographed image of a display area and evaluating the current display state using a plurality of evaluation indexes related to the disorder of the product from the detection result.

JP-A-2016-207164

With the technique of Patent Document 1, even if the display state can be presented to the store staff, if it is not possible to deal with it immediately without taking a hand for another job, a loss of sales opportunity will result.

An object of the present invention is to provide a mechanism capable of quickly responding to a display state that causes a loss of sales opportunity.

The present invention is a robot system including a robot arm capable of holding a product, and is a photographing means for photographing the displayed product and a disorder determining means for determining whether or not the displayed state of the photographed product is disturbed. It is characterized by providing a sorting means for rearranging products having a disorder in the display state by using the robot arm so that the disorder in the display state is eliminated.

The present invention is a robot system including a robot arm capable of holding a product, which is a photographing means for photographing a displayed product and a blind spot determining means for determining whether or not there is a blind spot in the displayed state of the photographed product. The product is characterized in that the product having a blind spot in the display state is rearranged by using the robot arm so that the blind spot in the display state disappears.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a mechanism for promptly responding to a display state that causes a loss of sales opportunity.

It is a figure which shows the relative positional relationship between a circulation robot and a product shelf in the case of confirming the disorder of a display state of this invention. It is a front view seen from the direction which the circulation robot of this invention faces a product shelf. It is a figure which showed the map in the store which the circulation robot of this invention circulates in a store, and sorts a disordered display. It is a figure which showed a beautiful example in the image which the circulation robot of this invention circulates in a store and took for making a turbulence determination. It is a figure which showed an example which it is determined that the circulation robot 300 of this invention circulates in a store, and is determined to be disordered in the image taken for performing the disorder determination. FIG. 5 is a diagram showing an example of an electronic shelf label that notifies store staff of support when it is determined that the disordered display cannot be rearranged by the robot arm in the present invention. It is a figure which shows the flowchart which shows the order of each process performed when the circulation robot of this invention circulates in a store. It is a figure which shows the flowchart which shows the order of the processing which performs the photographing performed by the circulation robot of this invention. It is a figure which shows the flowchart which shows the order of the turbulence determination processing performed by the circulation robot of this invention. It is a figure which shows the flowchart which shows the order of the sort determination process performed by the circulation robot of this invention. It is a figure which shows the flowchart which shows the order of the notification processing of the display support performed by the robot system of this invention. It is a figure which shows the flowchart which shows the order of the notification processing at the time of completion of correspondence performed by the robot system of this invention. It is a figure which shows the system block diagram of the robot system of this invention. It is a figure which shows the data table structure used in the robot system of this invention. It is a figure which shows the hardware block diagram of the information processing apparatus used in the robot system of this invention. It is a figure which shows the functional block diagram of the information processing apparatus used in the robot system of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 will be described.

FIG. 1 shows, in the present invention, a robot arm is used to display products so that when a robot arm is used to reduce the disorder of the display state when the product is circulated in a store by automatic running and a disorder of the display state is found on a certain product shelf. It is a circulation robot that rearranges 300.

In this circulation robot, 300 is a camera 302 that shoots a product shelf 311, a robot arm 305 that rearranges the display of products, an end effector 307 that grabs, holds, and lifts the displayed products, and a position in the three-dimensional space of the displayed products. A stereo camera 306 that recognizes the robot arm 306, a pedestal 308 that fixes the robot arm 305 set, a pillar 303 for installing the camera 302 in a fixed position, a loading platform 309 that can automatically move the camera 302 set and the robot arm 305 set, and the camera 302. The pan part 301 (horizontal direction) and tilt part 301 (vertical direction) that freely change the shooting direction, and the rotating part that rotates each axis of the wheel 310 that automatically runs the loading platform 309 in the store and the robot arm 305. It is equipped with 304.

Furthermore, the position where the camera 302 is fixed is between 170 cm, which is the same as the height of an average Japanese man, and 150 cm, which is the same as the height of a smaller Japanese woman, so that it matches the height of the eyes of customers visiting the store. Preferably, in this embodiment, it is installed at a height of about 160 cm in the middle. This is a fixed height (160 cm) of the camera 302 and a camera 302 when the circulation robot 300 runs, in order to realistically reproduce how the display state can be seen from the line of sight of the customer who visited the store. The automatic driving route is set in advance so that the circulation robot automatically travels on the route where the distance (50 cm to 100 cm) from the product shelf 311 is the same as the position of the customer's line of sight looking around the store during shopping. Has been done.

Further, the position where the camera 302 is fixed can be appropriately set to an optimum height such as 100 cm to 120 cm according to the average height according to the age group and nationality of the customers who visit the store.

The circulation robot 300 automatically travels in a direction in which the camera 302 that photographs the product shelf 311 faces the front of the product shelf 311 and does not intersect with the front of the product shelf 311.

FIG. 1 shows a view of the circulation robot 300 viewed from a direction side that does not intersect, and shows a view of the product shelf 311 viewed from the side surface side thereof.

In FIG. 1, the number of product shelves 311 is four, and product 312 (manufactured by manufacturer A), product 313 (manufactured by manufacturer B), and product 314 (manufactured by manufacturer C) are divided by manufacturer in order from the top shelf. It shows the state of being displayed.

Furthermore, since the product 314 (manufactured by D manufacturer) placed in the lower row is placed on the rear side of the shelf, it is difficult to see from the height of the camera 302 that shoots the product shelf 311. It shows the display state where it is placed.

FIG. 1 is a side view of the circulation robot 300 as viewed from the direction of automatic traveling, which is substantially parallel to the arrangement of the product shelves 311.

FIG. 2 will be described.

FIG. 1 is a front view of the circulation robot 300 of the present invention as viewed from the direction facing the product shelf 311.

The shooting range (viewing angle) of the camera 302 that shoots the product shelf 311 with one sheet can be freely shot in the left-right direction of the product shelf 311 by changing the pan angle of the pan portion 301 by several steps, and tilts. By changing the tilt angle of the unit 301, it is possible to take a picture of the product shelf 311 in the vertical direction as well.

In addition, the shooting range (viewing angle) of the camera 302 that shoots the product shelf 311 with one image is the average viewing angle (about 90 ° to about 120 °) of the human eye so as to appropriately reproduce the customer's line of sight. , It is set in almost the same way. In addition, the zoom value (angle of view) of the camera 302 that captures the product shelf 311 with one image is also the average of the human eye so as to appropriately reproduce the customer's line of sight from the relative distance between the camera 302 and the product shelf 311. The zoom value is set to be almost the same as the sense of distance.

Further, when the circulation robot 300 stops at a certain point in front of the product shelf 311 on the route, the pan angle is changed to take an image in a plurality of directions, and the tilt angle is changed to take a camera image 500 in a plurality of directions.

From the camera image of the obtained product display state in one direction, it is determined whether or not the photographed product display state is disturbed, and the result is (1) a flag indicating whether or not there is disorder. Information, (2) flag information indicating whether or not there is a blind spot, and (3) information indicating the exact position of the location in front of the product shelf 311 on the route using GPS, (4) pan angle. Information indicating the orientation of the location and the height for calculating the number of product shelves using the tilt angle, (5) the date and time of shooting, and the time of occurrence when it was confirmed whether there was any disturbance. It is uniquely associated with the camera image ID of one captured image, added to EXIF information, etc., and recorded and saved.

The shooting range of the stereo camera 306 can be taken in the left-right direction of the product shelf 311 by changing the Θ angle of the rotating unit 304 by several steps, and by changing the Θ angle of the rotating unit 304, the product shelf It is also possible to shoot the 311 in the vertical direction. Further, by freely changing the XYZ coordinates of the robot arm 305, it is possible to freely change the shooting position for shooting.

In this way, when the circulation robot 300 stops at a certain point in front of the product shelf 311 on the route, different directions can be obtained by freely changing the Θ angle of the rotating portion 304 and the XYZ coordinates of the robot arm 305. Since a stereo image (not shown) is taken from the robot, it is possible to more accurately recognize the position of the displayed product in the three-dimensional space.

In a three-dimensional space that can be rearranged with the robot arm 305 so that there is no disturbance determined by the camera image of the display state from the obtained stereo image that can recognize the position of the product in the three-dimensional space from multiple directions. It is determined whether or not there is, and as a result, (6) flag information indicating whether or not the robot arm 305 can be rearranged, and (1) one image having flag information indicating whether or not there is disturbance. It is uniquely associated with the camera image ID of the above, added to EXIF information, etc., and recorded and saved.

The drive range of the robot arm 305 is to move the products distributed in the shelves to the front of the shelves so as to improve the disordered display state or the blind spot display state on each shelf. .. Properly stand the fallen items on the shelves. Orient products on the shelves that are not facing the customer's line of sight. The products 314 (manufactured by D manufacturer) placed on the rear side of the shelves are moved to the front of the shelves so that the products are arranged neatly, and the display state is rearranged.

The products that are already on display are close to the expiration date and expiration date, so it is necessary to bring them to the front of the shelf as much as possible so that they can be noticed by the customer and be found from the customer's perspective.

Furthermore, the products that have already been displayed are close to the expiration date and expiration date, and by moving them as close to the front of the shelf as possible, space has been secured for additional product replenishment. The work of additional replenishment of products by the store staff or the circulation robot 300 is also facilitated. When the robot arm 305 moves each product to rearrange the display state, the expiration date and expiration date written on each product are read, and the expiration date and expiration date are read in order of the front of the shelf or one place. You may move it to (the center or the corner of the shelf).

FIG. 3 will be described.

FIG. 3 is a diagram showing an in-store map 400 in which the circulation robot 300 circulates in the store and rearranges the disordered displays. In this example, the in-store layout of a convenience store is shown. Further, even when the in-store layout is displayed on the in-store terminal 102, such an in-store map 400 is displayed by UI, and the position where the current circulation robot 300 is running in the store and the display on the product shelf are displayed. The position of the electronic shelf label that lights up that it is disturbed and the UI of the in-store map 400 are displayed.

→ in FIG. 3 is a traveling route in which the circulation robot circulates in the store. This circulation robot circulates during business hours, finds products on the product shelves whose display is disturbed, and returns them to the correct state as soon as possible with the robot arm.

FIG. 4 will be described.

FIG. 4 is a diagram showing an example of a camera image 500 in a case where the circulation robot 300 circulates in the store and is determined to be beautiful among the images taken for determining the disorder.

One electronic shelf tag is installed for each product on each product shelf. For example, as for milk product 313 (manufactured by C manufacturer), products with close (old) expiration dates are brought to the front of the shelf and arranged neatly to the back of the shelf in order of expiration date. The display state at the time of opening or immediately after the product is put out is thus in a beautiful state by the store staff.

FIG. 5 will be described.

FIG. 5 shows an image taken by the circulation robot 300 circulating in the store to determine the disorder, and the display state is disturbed compared to when the store is opened or immediately after the product is put out, or the position of the blind spot from the customer's point of view. It is a figure which showed an example of the camera image 600 of the case which it is determined that there is a product placed in.

In the evening or after the bargain sale, the product 314 is biased toward the back of the shelf, the product 314 is oriented incorrectly, or the product 314 collapses. As a result, only the products with a near (old) expiration date in the front and the products 314 in poor display condition will be left unsold. Such disrupted display conditions increase the likelihood that customers will have less chance of picking up and considering purchasing products, such as bargain sale items and items with a limited expiration date of only a few days, such as milk and cake. It is important to restore the display status from the customer's point of view by returning to the display status at the time of opening the store or immediately after the product is put out as much as possible.

FIG. 6 will be described.

FIG. 6 shows an electronic shelf label 700. On 701, the product name, features, production area, price, design, etc. placed on the product shelf 311 are switched and displayed every day. Reference numeral 702 is a support notification lamp that lights up when a support notification display instruction is received. Also, when the support notification lamp 702 is pushed by the store staff, it goes out. 703 is the correct answer button and can be pressed by the store staff. 704 is an incorrect answer button and can be pressed by the store staff.

Next, FIG. 7 will be described which describes the process executed by the robot system in the present embodiment by using the flowcharts of FIGS. 7 to 12.

FIG. 7 is a flowchart showing an example of the overall processing executed by the robot system. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S101, the autonomous movement route that circulates automatically in the store is compared with the current position of the circulation robot 300 using GPS, and it is determined whether or not the automatic travel of the autonomous movement route is completed. If it is finished (YES), the whole process is finished. If it does not end (NO), the process proceeds to S102 to take a picture in front of the product shelf to check the display status of the product.

In step S103, a process of determining whether or not there is a disorder or a blind spot in the display state is performed from the camera image for confirming the display state of the product in front of the product shelf.

In step S104, it is determined from the stereo image that can recognize the position of the product having the disorder or the blind spot in the display state in the three-dimensional space, whether the product with the disorder or the blind spot can be rearranged by the robot arm. I do.

In step S105, a process of notifying the store staff of the support for sorting the products having disorder or blind spots is performed.

FIG. 8 will be described.

FIG. 8 is a sub-flow chart showing an example of a shooting process executed by the robot system. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S201, the product autonomously travels on the autonomous movement route that automatically circulates in the store registered in advance, and starts moving to the front of the product shelf 311 in the store (the aisle shown by → in FIG. 3). ..

In step S202, it is determined whether the current traveling position is in front of the product shelf 311 for which the product display status should be confirmed. If it is in front of the product shelf 311 to be confirmed (YES), the process proceeds to step S203. If it is not in front of the product shelf 311 to be confirmed (NO), the process returns to step S201.

In step S203, the movement is temporarily stopped in front of the product shelf 311 to be confirmed. This stop position is preferably the center position with respect to the width direction of the product shelf 311 to be confirmed, but any stop position may be used as long as the display state of the product shelf 311 to be confirmed can be confirmed in the store.

In step S204, the shooting range of the camera 302 that shoots the product shelf 311 is changed, and camera images 500 in a plurality of directions (for example, 3 directions in the left middle right × 9 directions in the upper middle lower 3 directions) are taken and taken. The camera image is recorded in the image recording unit 211 of the circulation robot 300.

At this time, an image ID is assigned to the camera image, and (3) GPS is used to use information indicating the exact position of the product shelf 311 on the route, and (4) pan angle and tilt angle. Information indicating the orientation of the location and the height for calculating the number of product shelves, (5) the date and time of shooting, and the time of occurrence when confirming whether there is any disturbance are added to the EXIF information and saved. Will be done.

From the information in (3), the front position of the product shelf 311 in the store can be specified, from the information in (4), the number of shelves can be specified, and from the information in (5), the time when the display status is confirmed is It is identifiable.

In step S205, it is determined whether or not all the determined directions have been photographed. If the shooting is finished (YES), the shooting process is finished. If the shooting is not completed (NO), the process returns to step S204.

FIG. 9 will be described.

FIG. 9 is a sub-flow chart showing an example of a process for determining turbulence executed by the robot system. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S301, at this stop position, one camera image 500 in a certain direction taken in step S204 is read out.

In step S302, the read camera image 500 is analyzed, and as the input information, the feature amount of the product 313 (detection of the product rectangle, specification of the product type, specification of the background type, relative distance between the products, and between the products). (Relative color tones, differences between product types) are extracted, and using a trained model that has learned the beautiful display state in advance, it is determined whether or not the display state of the product is disturbed as output information. To do.

Further, the read camera image 500 is analyzed, and as the input information, the feature amount of the product 313 (detection of the product rectangle, specification of the product type, specification of the background type, relative distance between the products, relative between the products). Using a trained model that has learned the display state (the product is placed in front of the shelf) without blind spots by extracting the specific color and difference between product types), the output information of the product Determine if there is a blind spot in the display state.

If the display state of the product is disturbed as the output information, or if there is a blind spot in the display state of the product (both are applicable or one is applicable), the process proceeds to S303. If there is no disturbance in the display state of the product, or if there is no blind spot in the display state of the product (both do not apply), the process proceeds to S304.

In step S303, in the image ID of the read camera image, (1) flag information indicating a display state with disturbance and (2) flag information indicating a display state with a blind spot are added to EXIF information and saved. Will be done.

In step S304, (1) flag information indicating a clean display state without disturbance and (2) flag information indicating a correct display state without blind spots are added to the EXIF information and the like to the image ID of the read camera image. Is recorded and saved.

Next, returning to the circle 1, at this stop position, one camera image 500 in another direction taken in step S204 is read out. This loop is repeated with camera images 500 in multiple directions.

FIG. 10 will be described.

FIG. 10 is a sub-flow chart showing an example of a process of performing a rearrangement determination executed by the robot system. Each processing step is executed by the CPU 201 of the circulation robot 300.

In step S401, at this stop position, (1) flag information indicating a display state with disturbance and (2) flag information indicating a display state with a blind spot are added to EXIF information and saved in step S303. , Read one camera image 500.

In step S402, from the information of (3) and (4) of this image ID written in step S204, the detailed part of the product shelf (some steps, from the robot arm) where there is a problem in displaying the stop position of the circulation robot. Get the position of).

In step S403, the stereo camera 306 captures a detailed portion of the acquired product shelf whose display is disturbed from a plurality of directions.

In step S404, the position of the three-dimensional space of the product group in the display state with disturbance or the position of the three-dimensional space of the product group in the display state with a blind spot is recognized from the stereo image (not shown) in a plurality of directions. , The robot arm determines whether or not it is possible to rearrange the position of the product group in the display state with disorder or the position in the three-dimensional space of the product group in the display state with a blind spot. For example, if the position of the recognized display-state product group in the three-dimensional space is a position or shape that cannot be reached by the robot arm, or if the position or shape cannot be held by the robot arm, the position or shape that the robot arm collides with. If, it is determined that the sorting is not possible.

If it is determined that the sorting is possible (YES), the process proceeds to S405. If it is determined that the sorting is impossible (NO), the process proceeds to S407.

In step S405, flag information indicating that (6) the robot arm 305 can also be rearranged is added to the EXIF information and the like to be recorded and saved in the image ID read in step S401.

In step S406, the circulation robot 300 picks the target products 314 whose display state is a problem one by one by using the robot arm 305 according to the position of the recognized product group in the display state in the three-dimensional space. By moving it to a different place such as the front side of the shelf, it works to reduce the disorder of the display state and the blind spot.

In this way, the products having a disordered display state are sorted by using the robot arm 305 so that the disordered display state is eliminated, so that it is possible to quickly respond to the display state that causes a loss of sales opportunity.

In step S407, (6) flag information indicating that sorting is not possible even with the robot arm 305 is added to the EXIF information and the like to be recorded and saved in the image ID read in step S401.

Next, returning to circle 2, at this stop position, (1) flag information indicating a display state with disturbance and (2) flag information indicating a display state with a blind spot are added to EXIF information and the like in step S303. Another camera image 500 that has been recorded and saved is read out. This loop is repeated with the camera image 500 to which the flags of (1) and (2) are added.

FIG. 11 will be described.

FIG. 11 is a sub-flow chart showing an example of a process of notifying display support executed by the robot system. Each processing step is executed by the circulation robot 300, the in-store controller 100, the in-store terminal 102, and the CPU 201 of the electronic shelf label 700.

At this stop position, the circulation robot 300 is recorded and saved in the image ID in step S407, and (6) flag information indicating that the robot arm 305 cannot be rearranged is added to the EXIF information and the like in step S501. Read one 500 sheet.

In step S502, the circulation robot 300 has a problem in displaying the stop position of the circulation robot from the information of (3) and (4) of the image ID written in the EXIF information in step S204. Get the position of the step).

In step S503, the circulation robot 300 transmits the position of the detailed portion (number of stages) of the product shelf having a problem in the acquired display to the in-store controller 100.

The in-store controller 100 collates the position of the detailed part (number of stages) of the product shelf with the transmitted display with the table corresponding to the electronic shelf label, and supports only the ID of the collated electronic shelf label 700. Send notification display instructions.

The electronic shelf label 700 that has received the support notification display instruction turns on the support notification lamp 702. This allows store staff to visually and quickly find the location of product shelves that cannot be sorted by robots and have display problems. In addition, since the electronic shelf tag 700 is a different color (red), it should be discolored, the pan support mark confirmed by the store staff should be displayed, and the store staff should visually quickly find the location of the product shelf that has a problem with the display. It suffices if it can be done. In addition, in the case of stores where the electronic shelf label 700 cannot be installed, an indicator light such as a patrol light (registered trademark) capable of wireless communication that receives a support notification display instruction is provided for each location of the product shelf that has a problem in display. Even if it is turned on, it has the same effect.

Further, the in-store controller 100 collates the position of the detailed part (number of stages) of the product shelf having a problem in the transmitted display with the table corresponding to the electronic shelf label, and determines the location of the collated electronic shelf label 700. A support notification display instruction recognizable by the in-store map 400 is sent to the in-store terminal 102.

The in-store terminal 102 that has received the support notification display instruction can display the location of the electronic shelf label 700 collated in the in-store map screen with the display contents as shown in FIG. 3 and the target. The support notification mark is identified and displayed only on the ID of the electronic shelf label 700. This allows store staff to visually quickly find out that a product shelf has a problem in display because it cannot be sorted by the robot.

FIG. 12 will be described.

FIG. 12 is a sub-flow chart showing an example of a process of notifying the completion of the response executed by the robot system. Each processing step is executed by the CPU 201 of the in-store controller 100 and the electronic shelf label 700.

In step S601, the electronic shelf label 700 determines whether or not the instruction to turn off the support notification lamp 702 in the electronic shelf label 700 shown in FIG. 6 has been received. If accepted (YES), proceed to S602. If it is not accepted (NO), it returns to the state of S601. After the support notification lamp 702 in the electronic shelf label 700 is turned on, the store staff who has corrected the display disorder presses the support notification lamp 702 to turn it off to determine whether or not the reception has been accepted.

In step S602, the electronic shelf label 700 turns off the support notification lamp 702 in the electronic shelf label 700. Further, the store staff who has corrected the display disorder presses the support notification lamp 702 to the in-store controller 100, and sends a response completion registration request in order to register the completion of the response as sales promotion data. After that, in order to register the time when the store staff corrects the display disorder at this time, the in-store controller 100 newly registers the time received here as the response completion time in the improvement table. Furthermore, from the difference between the already registered occurrence time and the response completion time, how much time has passed until the store staff corrects the display disorder is calculated and registered in the improvement table.

In step S701, the electronic shelf label 700 determines whether or not the registration instruction of the correct answer button 703 in the electronic shelf label 700 shown in FIG. 6 has been accepted. If accepted (YES), proceed to S702. If it is not accepted (NO), it returns to the state of S701. The store staff who visually confirmed the disturbance of the display made a judgment as to whether or not the reception was accepted by pressing the correct answer button 703 because the result of the disturbance determination processing by the circulation robot 300 was correct.

In step S702, the electronic shelf label 700 transmits a teacher data registration request to the in-store controller 100 so that the store staff who has corrected the display disorder presses the correct answer button 703 and visually confirms it as teacher data. .. After that, the in-store controller 100 is instructed to read the camera image ID once read in step S501 again and register the EXIF information of the camera image ID in association with the (7) correct answer flag.

In step S801, the electronic shelf label 700 determines whether or not the registration instruction of the error button 704 in the electronic shelf label 700 shown in FIG. 6 has been accepted. If accepted (YES), proceed to S802. If it is not accepted (NO), it returns to the state of S801. The judgment as to whether or not the acceptance has been made may be made by the store staff who visually confirmed the disturbance of the display pressing the mistake button 704 because the result of the disturbance determination processing by the circulation robot 300 was wrong.

In step S702, the electronic shelf label 700 requests teacher data registration to the in-store controller 100 so that the store staff who has corrected the display disorder presses the error button 704 and visually confirms it as teacher data. Send. After that, the in-store controller 100 is instructed to read the camera image ID once read in step S501 again and register the EXIF information of the camera image ID in association with the (8) error flag.

FIG. 13 will be described.

FIG. 13 is a system configuration diagram showing an example of the configuration of the robot system of the present invention.

The robot system has a configuration in which a circulation robot 300, an in-store controller 100, a plurality of in-store terminals 102, a plurality of electronic shelf labels 700, a data management cloud 200, and a wireless communication router 101 are connected via an Internet line or a LAN. It has become.

Further, the circulation robot 300, the in-store controller 100, the plurality of in-store terminals 102, the plurality of electronic shelf labels 700, and the wireless communication router 101 are arranged in the store.

The data management cloud 200 includes a camera image 500 taken by the circulation robot 300, EXIF information in the camera image 500, a learned model installed in the circulation robot 300, an electronic shelf label compatible table stored in the in-store controller 100, and an improvement table. Save it for backup. As appropriate, the latest trained model additionally trained with the new teacher image is installed in the circulation robot 300. In addition, the data management cloud 200 collects new teacher images from a large number of stores to improve the determination accuracy of disorder determination, blind spot determination, and sortability determination on a daily basis.

The data management cloud 200 uses a large number of teacher images (learning datasets) on which products are displayed to display image features (detection of product rectangles, identification of product types, identification of types such as shelves and backgrounds, and between products. Relative distance, relative color between products, difference between product types, relative distance between shelves and backgrounds, relative color between shelves and backgrounds, difference between shelves and background types, etc. ), And learned the display state without any disturbance (such as when the product is placed at the beginning of the store) and the display state without blind spot (such as when the product is placed in front of the shelf). A trained model is created, and the latest trained model is periodically transmitted to the circulation robot 300.

The data management cloud 200 stores the camera image 500 received from the circulation robot 300, the electronic shelf label compatible table received from the in-store controller 100, and the improvement table.

The circulation robot 300 is constantly connected to the in-store controller 100 and the data management cloud 200 by wireless communication (Wi-Fi).

The in-store controller 100 is also wirelessly communicated (Wi-Fi) and is constantly connected to a large number of electronic shelf labels 700 in the store and in-store terminals 102 carried by a large number of store staff.

FIG. 14 will be described.
Next, an example of an electronic shelf label compatible table and an improvement table used by the robot system in the present embodiment will be described with reference to the data structure diagram of FIG.

The electronic shelf label compatible table is a master table that associates the ID of the electronic shelf label with the position and height of the electronic shelf label.

The ID of the electronic shelf label is a unique number of the electronic shelf label, and the ID of the electronic shelf label uniquely related to the position and height read from the EXIF information of the camera image 600 is specified.

The product shelf number is a unique name of the product shelf, and identifies which product is placed on the product shelf.

The number of stages is a unique number of stages of the electronic shelf label, and the ID of the electronic shelf label uniquely related to the position and height read from the EXIF information of the camera image 600 is specified.

The height above the ground is a unique height of the electronic shelf label, and like the number of stages, the ID of the electronic shelf label uniquely related to the position and height read from the EXIF information of the camera image 600 is specified.

The longitude information and the latitude information are unique positions of the electronic shelf label, and the ID of the electronic shelf label uniquely related to the position and height read from the EXIF information of the camera image 600 is specified.

The improvement table is a master table that associates the ID of the electronic shelf label with the occurrence of disturbance and blind spots and the completion of response.

The occurrence time is the time when the circulation robot confirms the occurrence of disturbance or blind spot.

The response completion time is the time when the circulation robot or the store staff completes the response to line up neatly.

The elapsed time is the elapsed time from the confirmation of the occurrence of disturbance or blind spot to the time when the circulation robot or the store staff completes the response to arrange them neatly.

In addition, the EXIF information for each image ID is overwritten with the following flags, location information, and date and time.

(1) A flag indicating either that there is turbulence or that there is no turbulence.

(2) A flag indicating either that there is a blind spot or that there is no blind spot.

(3) Information indicating the exact position of the place in front of the product shelf 311 on the route using GPS. (4) Information indicating the orientation of the location and the height for calculating the number of product shelves using the pan angle and tilt angle.

(5) The date and time when the product was photographed and the time when the circulation robot confirmed whether there was any disturbance or blind spot.

(6) A flag indicating either that the robot arm 305 can be rearranged or that it cannot be rearranged.

(7) A flag indicating the correct answer.

(8) A flag indicating an error.

FIG. 15 shows the circulation robot 300, the in-store controller 100, the plurality of in-store terminals 102, the plurality of electronic shelf labels 700, the data management cloud 200, and the hardware of the information processing device applicable to the wireless communication router 101 according to the embodiment of the present invention. It is a block diagram which shows the wear configuration. Since each device has the same configuration, the same reference numerals will be used for description.

The circulation robot 300, the in-store controller 100, the plurality of in-store terminals 102, the plurality of electronic shelf labels 700, the data management cloud 200, and the wireless communication router 101 are provided by the CPU (Central Processing Unit) 201 and the ROM (Central Processing Unit) 201 via the system bus 204. A configuration is adopted in which a Read Only Memory) 202, a RAM (Random Access Memory) 203, an input controller 205, a video controller 206, a memory controller 207, a communication I / F controller 208, and the like are connected. The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

Further, in the ROM 202 or the external memory 211 (external memory), in order to realize a function executed by each server or each PC, the BIOS (Basic Input / Output System) or OS (Operating System) which is a control program of the CPU 201. Various programs and the like, which will be described later, are stored. In addition, information necessary for carrying out the present invention is stored. The external memory may be a database.

The RAM 203 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing the process from the ROM 202 or the external memory 211 into the RAM 203 and executing the loaded program.

The input controller 205 controls input from a pointing device such as a keyboard (KB) 209 or a mouse (not shown). The input controller 205 also controls the camera image input from the camera device 213. The student PC 141 is equipped with a Web camera or the like as a camera device and captures a camera image used for face recognition, but a separate camera device may be prepared for face recognition.

The video controller 206 controls the display on a display such as the display 210. The display may be a display such as a liquid crystal display. These are used by the administrator as needed.

The memory controller 207 is an external storage device (hard disk (HD)) for storing boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA (Personal Computer). Controls access to external memory 211 such as Compact Flash® memory connected via an adapter to the Memory Card International Association card slot.

The communication I / F controller 208 connects and communicates with an external device via the network, and executes communication control processing on the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

The CPU 201 can display the outline font on the display 210 by executing the outline font expansion (rasterization) process on the display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction by a mouse cursor (not shown) or the like on the display 210.

Various programs described later for realizing the present invention are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as needed.

Although one embodiment has been described above, the present invention can take an embodiment as, for example, a system, an apparatus, a method, a program, a recording medium, or the like, and specifically, is composed of a plurality of devices. It may be applied to a system or a device consisting of one device.

Further, the program in the present invention is a program in which a computer can execute the processing method of the flowchart shown in each figure, and the storage medium of the present invention stores a program in which the computer can execute the processing method in each figure. The program in the present invention may be a program for each processing method of each device in each figure.

As described above, a recording medium on which a program for realizing the functions of the above-described embodiment is recorded is supplied to the system or device, and the computer (or CPU or MPU) of the system or device stores the program in the recording medium. Needless to say, the object of the present invention can be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon. A disk, solid state drive, etc. can be used.

Further, by executing the program read by the computer, not only the function of the above-described embodiment is realized, but also the OS (operating system) or the like running on the computer is actually operated based on the instruction of the program. Needless to say, there are cases where a part or all of the processing is performed and the processing realizes the functions of the above-described embodiment.

Further, the program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, and then the function expansion board is based on the instruction of the program code. It goes without saying that there are cases where the CPU or the like provided in the function expansion unit performs a part or all of the actual processing, and the processing realizes the functions of the above-described embodiment.

Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. It goes without saying that the present invention can also be applied when it is achieved by supplying a program to a system or device. In this case, by reading the recording medium in which the program for achieving the present invention is stored into the system or device, the system or device can enjoy the effect of the present invention.

Further, by downloading and reading a program for achieving the present invention from a server, database, or the like on the network by a communication program, the system or device can enjoy the effect of the present invention.

It should be noted that all the configurations in which the above-described embodiments and modifications thereof are combined are also included in the present invention.

100 In-store controller 102 In-store terminal 200 Data management cloud 300 Circulation robot 311 Product shelf 312-314 Product 302 Camera 305 Robot arm 306 Stereo camera 400 In-store map 500 Camera image 600 Camera image 700 Electronic shelf label

Claims (11)

A robot system equipped with a robot arm that can hold products.
Shooting means to shoot the displayed products,
Disturbance determining means for determining whether or not the displayed state of the photographed product is disturbed, and
A sorting means for rearranging products having a disordered display state by using the robot arm so that the disordered display state is eliminated.
A robot system characterized by being equipped with. A sorting determination means for determining whether or not a product having a disordered display state can be rearranged by using the robot arm so that the disordered display state is eliminated.
When the determination is not sortable, a notification means for notifying that the display state is disturbed, and
The robot system according to claim 1, further comprising. A robot system equipped with a robot arm that can hold products.
Shooting means to shoot the displayed products,
A blind spot determination means for determining whether or not there is a blind spot in the display state of the photographed product,
A sorting means for rearranging products having a blind spot in the display state by using the robot arm so that the blind spot in the display state disappears.
A robot system characterized by being equipped with. A sorting determination means for determining whether or not a product having a blind spot in the display state can be rearranged by using the robot arm so that the blind spot in the display state disappears.
If the determination is not sortable, a notification means for notifying that there is a blind spot in the display state, and
The robot system according to claim 3, further comprising. The robot system according to any one of claims 1 to 4, wherein the photographing means photographs the displayed product from a height of a customer's line of sight. The robot system according to claim 5, wherein the imaging means captures a plurality of locations by changing the imaging direction from the height of the customer's line of sight. The robot system according to any one of claims 1 to 6, wherein the photographing means photographs the displayed product at a viewing angle of the customer's line of sight. A control method for a robot system equipped with a robot arm that can hold products.
The shooting process of shooting the displayed products and
The disorder determination process for determining whether or not the display state of the photographed product is disturbed, and
A rearrangement process in which the products having a disordered display state are rearranged by using the robot arm so that the disordered display state is eliminated.
A control method characterized by including. A control method for a robot system equipped with a robot arm that can hold products.
The shooting process of shooting the displayed products and
A blind spot determination step for determining whether or not there is a blind spot in the display state of the photographed product, and
A rearrangement process in which products having a blind spot in the display state are rearranged by using the robot arm so that the blind spot in the display state disappears.
A control method characterized by including. A program that can be read and executed by a robot system equipped with a robot arm that can hold products.
The robot system
Shooting means to shoot the displayed products,
Disturbance determining means for determining whether or not the displayed state of the photographed product is disturbed, and
A sorting means for rearranging products having a disordered display state by using the robot arm so that the disordered display state is eliminated.
A program to function as. A program that can be read and executed by a robot system equipped with a robot arm that can hold products.
The robot system
Shooting means to shoot the displayed products,
A blind spot determination means for determining whether or not there is a blind spot in the display state of the photographed product,
A sorting means for rearranging products having a blind spot in the display state by using the robot arm so that the blind spot in the display state disappears.
A program to function as.

Images