JP7005418B2 - Product reading system - Google Patents

Description

Embodiments of the present invention relate to a product reading system.

Conventionally, in retail stores such as convenience stores and supermarkets, the payment process of the product is performed after the payment device is made to read the product specific information such as a barcode. Many of the checkout devices are operated by retail store staff and manually read the barcode printed on the product. In recent years, there has been a demand for labor saving in the work (product reading work) of having the settlement device read the product to be settled due to labor shortage.

As a conventional technique, a device provided with a robot arm for gripping an article such as a product has been proposed. However, the robot arm provided in the conventional device moves the grasped article to a predetermined place, or puts the grasped article in a storage part such as a bag or a box. Therefore, with the conventional technology, it is not possible to save labor in reading a product such as a barcode of the product.

Japanese Unexamined Patent Publication No. 2016-167246

An object of the present invention is to provide a product reading system capable of saving labor in reading product specific information in a product in order to solve the above-mentioned problems.

According to the embodiment, the product reading system includes a camera, an arm device, a scanner, and a controller. The camera takes a picture of the table on which the product is placed from above. The arm device has a first arm and a second arm, each of which is provided with a gripping mechanism. The scanner reads the product identification information existing in the appearance of the product at the reading position. When the shape of the product detected from the image taken by the camera is bag-shaped, the controller causes the gripping mechanism of the first arm and the gripping mechanism of the second arm to grip both ends of the product, respectively. The product is moved to the reading position of the scanner while the gripping position of the first arm and the gripping position of the second arm are pulled from each other.

FIG. 1 is a schematic diagram schematically showing an external configuration of a product reading system (product reading support system) according to an embodiment. FIG. 2 is a block diagram showing a configuration example of a control system of the robot 1 as a product reading system according to an embodiment. FIG. 3 is a flowchart for explaining an operation example of the robot as the product reading system according to the embodiment. FIG. 4 is a flowchart for explaining an operation example of the robot as the product reading system according to the embodiment. FIG. 5 is a diagram for explaining an operation example when the product reading system according to the embodiment grips a bag-shaped product with an arm. FIG. 6 is a diagram for explaining an example of a state in which the product reading system according to the embodiment holds a box-shaped product with an arm. FIG. 7 is a flowchart for explaining an operation example of a barcode reading process by an arm scanner in the product reading system according to the embodiment. FIG. 8 is a diagram for explaining an operation example of the arm scanner when the height of the cylindrical product is within a predetermined height in the product reading system according to the embodiment. FIG. 9 is a diagram for explaining an operation example of the arm scanner when the height of the cylindrical product exceeds a predetermined height in the product reading system according to the embodiment. FIG. 10 is a flowchart for explaining an operation example of the product recognition process for the image taken by the camera in the product reading system according to the embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 is a schematic diagram schematically showing an external configuration of a product reading support system (robot) 1 as a product reading system according to an embodiment.
The product reading support system (hereinafter, also simply referred to as a robot) 1 is a device for acquiring (reading) information (product specific information) for specifying a product. In the configuration example shown in FIG. 1, the robot 1 supports the cash register business of acquiring information specifying a product for which the price should be settled by the checkout device (register) 2. For example, the robot 1 has a humanoid appearance that imitates the upper body of a person so that it can be easily used by the user.

The robot 1 is connected to a settlement device 2 that setstles the price of a product or the like. The robot 1 and the settlement device 2 constitute a transaction system for performing settlement operations of goods. The transaction system including the robot 1 and the checkout device 2 is installed in a retail store such as a convenience store or a supermarket, for example. In the transaction system, the robot 1 is provided in the vicinity of the checkout table 3 on which the checkout device 2 is installed. The user places a product to be settled on the checkout table 3 in front of the robot 1 with the checkout device 2. The robot 1 reads the product identification information from the product placed on the cash register table 3 by the user, and supplies the read product identification information to the settlement device 2.

The product specific information may be any information that can be read from the appearance of the product. The product identification information is a barcode printed on the product package. Further, the product identification information may be a character string such as a number or a character that can identify the product, an image of a label attached to the product, or a silhouette of the entire product. The robot 1 lifts and moves the product on the cash register table 3 so that the product identification information such as the barcode on the appearance of the product can be read by a scanner or a camera. The robot 1 also has a function of moving a scanner along a product that cannot be lifted to read product identification information such as a barcode.

In the configuration example shown in FIG. 1, the robot (main body) 1 has a sensor 11, a camera 12, a scanner 13, a first arm 14, a second arm 15, an arm scanner 16, a speaker 17, a microphone 18, a display 19, and the like. ..
The sensor 11 is a sensor that detects a person approaching the cash register table 3 or a product placed on the cash register table 3. For example, the sensor 11 is an infrared sensor that detects a person or a product using infrared rays. The sensor 11 may be any sensor that can detect the presence or absence of a person or a product.

The camera 12 captures an image around the cash register table 3. The camera 12 has a function of changing the shooting area by adjusting the shooting direction, the shooting distance, and the like. For example, the camera 12 has a function of setting a shooting area for shooting a person who uses the cash register 2 (or a person who puts a product on the cash register table 3) and shooting a person in the vicinity of the cash register table 3. .. Further, the camera 12 also has a function of setting a shooting area for shooting a product placed on the cash register table 3 and shooting an image on the cash register table 3.

The scanner 13 reads the product specific information from the appearance of the product. In the present embodiment, the scanner 13 is a barcode scanner that reads a barcode printed on a part of a product package as product identification information. The scanner 13 may be configured as long as it is product-specific information from the appearance of the product, and may be configured by a video camera or the like that reads a two-dimensional bar code, a character string, or the like.

The first arm 14 and the second arm 15 operate relative to the goods placed on the cash register table 3. The first arm 14 and the second arm 15 have arm portions 14a and 15a and gripping mechanisms 14b and 15b. The arm portions 14a and 15a are articulated robot arms attached to the robot 1 main body. The gripping mechanisms 14b and 15b are mechanisms for gripping the product. The arm portions 14a and 15a may be any as long as they can freely move the gripping position by the gripping mechanisms 14b and 15b in three dimensions. For example, in the first arm 14 and the second arm 15, the gripping mechanisms 14b and 15b are provided at the tip portions of the arm portions 14a and 15a, respectively. That is, the first arm 14 and the second arm 15 move the arm portions 14a and 15a in a state where the product is gripped by the gripping mechanisms 14b and 15b to lift or rotate the product.

The gripping mechanisms 14b and 15b are configured to pinch, pinch, and suck the product on the cash register table 3. For example, the gripping mechanisms 14b and 15b are composed of a plurality of fingers (two holding members, three to five fingers, and the like). Further, the gripping mechanisms 14b and 15b are not limited to those composed of a plurality of fingers, and may be a mechanism for adsorbing a part of the product. Further, the gripping mechanisms 14b and 15b may be configured by combining a plurality of fingers, a holding member, a suction mechanism, and the like.

Further, the arm scanner 16 is a scanner whose reading position is moved by the operation of the first arm 14. For example, the arm scanner 16 is provided at the tip end portion of the first arm 14, and the reading position is moved by the operation of the first arm 14. In the present embodiment, the arm scanner 16 reads a barcode as product identification information from the appearance of the product at the reading position moved by the first arm 14. The arm scanner 16 may be provided on the second arm 15 or on both the first arm 14 and the second arm 15. Further, the arm scanner 16 may be provided separately from the first arm 14 and the second arm 15.

The speaker 17 outputs sound. The speaker 17 outputs a voice such as guidance to the user. The microphone 18 inputs voice. The microphone 18 inputs the voice emitted by the user. The speaker 17 and the microphone 18 may constitute a dialogue device that interactively inputs and outputs voice with the user. The display 19 displays an image. For example, the display 19 displays a guide to the user by an image, text, or the like. The speaker 17, microphone 18, and display 19 function as a user interface.

Next, the configuration of the control system in the robot 1 as the product reading device according to the embodiment will be described.
FIG. 2 is a block diagram showing a configuration example of a control system of the robot 1 as a product reading system (product reading support system) according to an embodiment.
The robot 1 includes a system controller (control device) 21, a product reading device (product reader) 22, an arm device 23, a dialogue device (user interface) 24, a communication device (communication module) 25, and the like.

The system controller 21 comprehensively controls each device in the robot. In the configuration example shown in FIG. 2, the system controller 21 has a processor 31, a memory 32, and an interface (I / F) 33.
The processor 31 realizes various processes by executing a program. The processor 31 is, for example, a CPU. The memory 32 is composed of a memory such as a ROM, a RAM, and a rewritable non-volatile memory. For example, the memory 32 includes a non-volatile memory for storing programs, control data, and the like. Further, the memory 32 includes a working memory such as a RAM for temporarily storing data.

The program stored in the memory 32 includes various application programs in addition to the basic program (OS). The application program includes a program for executing various processes described later. The processor 31 realizes various processes described later by executing various programs stored in the memory 32.

The interface (I / F) 33 is an interface for communicating with each device in the robot 1.
The processor 31 gives a control instruction to each device and acquires information from each device via the interface 33. For example, the processor 31 acquires information indicating a reading result of product specific information, a detection result of a person, and the like from the product reading device 22. Further, the processor 31 instructs the arm device 23 to grip the product by using the first arm 14 and the second arm 15.

Next, a configuration example of the control system of the product reading device 22 will be described.
The product reading device 22 reads product specific information from the product. Further, the product reading device 22 has a function of detecting a person, a product, or the like. In the configuration example shown in FIG. 2, the product reading device 22 includes a processor 41, a memory 42, an interface (I / F) 43, an image memory 44, a camera driving mechanism 45, a sensor 11, a camera 12, a scanner 13, and the like.

The processor 41 realizes various processes by executing a program. The processor 41 is, for example, a CPU. The memory 42 is composed of a memory such as a ROM, a RAM, and a rewritable non-volatile memory. For example, the memory 42 includes a non-volatile memory for storing programs, control data, and the like. Further, the memory 42 includes a working memory such as a RAM for temporarily storing data.

The interface (I / F) 43 is an interface for communicating with the system controller 21.
The image memory 44 stores an image taken by the camera 12. The camera drive mechanism 45 controls the camera 12. The camera drive mechanism 45 may move the direction of the camera 12 or the like to control the area to be photographed by the camera 12.

The processor 41 realizes various processes by executing a program stored in the storage areas 42a to 42e of the memory 42. For example, the storage area 42a shall store a program for executing a process for identifying a person and an object. The processor 41 functions as a person identification unit or an object identification unit by executing a program stored in the storage area 42a. For example, the processor 41 confirms whether the user is in front of the cash register 2 (in front of the cash register table 3) by using an image taken by the camera 12 as a person identification unit. Further, the processor 41 may infer the characteristics (gender, age, etc.) of the user from the image taken by the camera 12. Further, the processor 41 may confirm whether or not the product is on the cash register table 3 as the object identification unit.

The storage area 42b shall store a program for performing a process of recognizing the size and shape of the article. The processor 41 functions as an appearance recognition unit that recognizes the size and shape of an object by executing a program stored in the storage area 42b. For example, the processor 41 recognizes information indicating features such as the size and shape of the product on the cash register table 3 from the image taken by the camera 12. In the present embodiment, as an appearance recognition unit, the processor 41 confirms whether the product is a bag-shaped product, a box-shaped product, a tubular-shaped product, or a bag-shaped product. ..

The storage area 42c shall store a program for executing a process of detecting a position. The processor 41 functions as a position detection unit for detecting a position by executing a program stored in the storage area 42c. For example, the processor 41 detects the position of the product on the cash register table 3 based on the image taken by the camera 12.

The storage area 42d shall store a program for executing a process of detecting a barcode as product identification information. The processor 41 functions as a barcode detection unit that detects a barcode attached to a product by executing a program stored in the storage area 42d. For example, the processor 41 detects a barcode attached to a product on the checkout table 3 from an image read by the scanner 13 or the arm scanner 16.

The storage area 42e shall store a program for executing a process of recognizing characters as product-specific information. The processor 41 functions as a character recognition unit (OCR processing unit) that recognizes characters attached to a product by executing a program stored in the storage area 42e. For example, the processor 41 recognizes characters as product identification information attached to a product on the cash register table 3 from an image taken by the camera 12.

Next, a configuration example of the control system of the arm device 23 will be described.
The arm device 23 operates the first arm 14 and the second arm 15. In the configuration example shown in FIG. 2, the arm device 23 includes a processor 51, a memory 52, an interface (I / F) 53, an arm drive mechanism 54, a first arm 14, a second arm 15, an arm scanner 16, and the like.

The processor 51 realizes various processes by executing a program. The processor 51 is, for example, a CPU. The memory 52 is composed of a memory such as a ROM, a RAM, and a rewritable non-volatile memory. For example, the memory 52 includes a non-volatile memory for storing programs, control data, and the like. Further, the memory 52 includes a working memory such as a RAM for temporarily storing data.
The interface (I / F) 53 is an interface for communicating with the system controller 21.

The arm drive mechanism 54 controls the movements of the first arm 14 and the second arm 15. For example, the arm drive mechanism 54 moves the first arm 14 and the second arm 15, respectively, in response to an operation instruction from the system controller 21. For example, the arm drive mechanism 54 grips the product on the cash register table 3 in response to an instruction from the system controller 21. Further, the arm drive mechanism 54 moves the gripped product according to the instruction. Further, the arm drive mechanism 54 moves the first arm 14 in response to an instruction to move the reading position by the arm scanner 16.

Next, a configuration example of the control system of the dialogue device 24 will be described.
The dialogue device 24 functions as a user interface. The dialogue device 24 provides information to the user by voice, image, or the like. Further, the dialogue device 24 inputs information from the user and outputs information as a response to the input information. In the configuration example shown in FIG. 2, the dialogue device 24 includes a processor 61, a memory 62, an interface (I / F) 63, a speaker 17, a microphone 18, a display 19, and the like.

The processor 61 realizes various processes by executing a program. The processor 61 is, for example, a CPU. The memory 62 is composed of a memory such as a ROM, a RAM, and a rewritable non-volatile memory. The interface (I / F) 63 is an interface for communicating with the system controller 21.

The processor 61 outputs sound from the speaker 17 in response to an instruction from the system controller 21 or the like. Further, the processor 61 supplies the voice information input to the microphone 18 to the system controller 21. As a result, the processor 61 performs interactive voice input / output with the user using the speaker 17 and the microphone 18. Further, the processor 61 displays information on the display 19 in response to an instruction from the system controller 21 or the like.

Next, a configuration example of the control system of the communication device 25 will be described.
The communication device 25 is a communication interface for communicating with an external device. In the present embodiment, the communication device 25 communicates with the settlement device 2. In the configuration example shown in FIG. 2, the communication device 25 includes a processor 71, a memory 72, an interface (I / F) 73, a communication unit 74, and the like.

The processor 71 realizes various processes by executing a program. The processor 71 is, for example, a CPU. The memory 72 is composed of a memory such as a ROM, a RAM, and a rewritable non-volatile memory. The interface (I / F) 73 is an interface for communicating with the system controller 21. The communication unit 74 is an interface for communicating with an external device. In the configuration example shown in FIG. 2, the communication unit 74 includes an interface for communicating with the settlement device 2.

The robot 1 as a product reading system may be integrally configured with the settlement device 2. Further, the robot 1 may be provided with a function of settling the price of the product. For example, the robot 1 may be provided with a payment function using a credit card, electronic money, or the like. In this case, the robot 1 can not only read the product specific information but also make a payment with a credit card, electronic money, or the like based on the product specific information.

Next, the operation of the product reading business support process (product reading process) by the robot 1 as the product reading device according to the embodiment will be described.
3 and 4 are flowcharts for explaining an operation example of the robot 1 according to the embodiment.
In the standby state, the processor 41 of the product reading device 22 monitors the presence or absence of a person by using the sensor 11 that detects a person with the vicinity of the cash register table 3 as the detection range (ACT 11). When the sensor 11 detects a person (ACT11, YES), the processor 41 takes an image with the camera 12 in which the shooting area is set in the shooting area for detecting a person near the cash register table 3 (ACT12). The shooting area for detecting a person may be any shooting area as long as it can shoot a person detected by the sensor 11 or a person who appears in front of the cash register table 3 due to payment of products or the like.

For example, the processor 41 sets the area including the space in front of the cash register table 3 in the camera 12 as a shooting range for person detection by the camera drive mechanism 45. Further, the processor 41 may estimate the position of the person detected by the sensor 11 and control the direction and position of the camera 12 according to the estimated position of the person. The processor 41 stores the image taken by the camera 12 in the image memory 44. Further, the processor 41 may continuously capture a plurality of images with the camera 12 and store the captured plurality of images in the image memory 44.

When the camera 12 captures an image in the vicinity of the cash register table 3, the processor 41 determines whether or not a person is present in the captured image captured by the camera 12 (ACT 13). As a result, the processor 41 can detect a person (user) in the vicinity of the cash register table 3 from the captured image captured by the camera 12. The processor 41 determines whether or not there is a user in the vicinity of the cash register table 3 based on the detection result of a person from the captured image. If it cannot be determined that there is a user (ACT 13, NO), the processor 41 returns to ACT 11 and executes the above process again.

When it is determined that there is a user (ACT 13, YES), the processor 41 notifies the controller 21 that the user has been detected. Upon receiving the notification that the user has been detected, the system controller 21 provides guidance to the user using the dialogue device 24 (ACT 14). For example, the system controller 21 outputs a voice instruction such as "Welcome" from the speaker 17 by outputting a voice instruction to the dialogue device 24. Further, the system controller 21 may display the guidance on the display 19 together with the voice guidance.

When a user is detected, the processor 41 captures an image with a camera 12 in which a shooting area for product detection including a product placement position on the cash register table 3 is set as a shooting area (ACT 15). The photographing area for product detection may be any area including the area where the product to be settled (registered) placed on the cash register table 3 is placed. For example, the photographing area for product detection may be an area on which the product can be placed on the cash register table 3 or an area including an arrangement position of the product provided on the cash register table 3. Here, the processor 41 may store the image of the shooting area for product detection shot by the camera 12 in the image memory 44.

When the shooting area for product detection is photographed by the camera 12, the processor 41 determines whether or not the product exists in the image (photographed image for product detection) captured by the camera 12 (ACT 16). As a result, the product reading device 22 can detect the product placed on the cash register table 3. The processor 41 may determine the presence or absence of the product on the cash register table 3 by a method other than the method of detecting the product from the image taken by the camera 12.

When it is determined that there is no product on the cash register table 3 (ACT16, NO), the processor 41 returns to ACT15 and executes the above process again. That is, if the product cannot be detected, the processor 41 uses the dialogue device 24 to provide guidance by voice or display to urge the product to be placed on the cash register table 3. For example, the processor 41 may utter "Please put the product on the checkout table" from the speaker 17 of the dialogue device 24. Along with such guidance, the processor 41 repeatedly executes image shooting and product detection on the cash register table 3.

When it is determined that the product is on the cash register table 3 (ACT16, YES), the processor 41 determines whether the product on the cash register table 3 should be gripped by the arm device 23. That is, the processor 41 identifies the shape of the product on the cash register table 3 (ACT 17). In the present embodiment, the processor 41 specifies whether the shape of the product is a bag shape, a box shape, or a cylindrical shape.

Here, the bag-shaped product is one that can be gripped by the arm device 23 as long as it is within a predetermined gripping size, and the shape of the appearance is deformed. The bag-shaped product may have a shape that can be lifted or rotated while being held by the arm device 23, as long as it is within a predetermined size. Further, the box-shaped product is a solid article having a shape that can be gripped by the arm device 23 as long as it is within a predetermined gripping size. The box-shaped product is not limited to the shape of a hexahedron, and may be any shape as long as it is within a predetermined size and can be lifted or rotated while being gripped by the arm device 23.

Whether or not a bag-shaped product or a box-shaped product can be gripped may be determined not by the gripping size but by the weight of the product. In this case, the weight of the product on the cash register table 3 may be measured by a weight sensor or the like provided on the cash register table 3.

Further, the tubular product is a product other than a bag shape and a box shape, and has a shape that cannot be gripped (should not be gripped) by the arm device 23. The tubular product is not limited to a cylindrical article, but also includes an article having a shape that cannot be grasped by the arm device 23, such as a polygonal column, a cone, and a pyramid. For example, a cylindrical product is assumed to be an item such as a PET bottle, a can, a bottle, or a cup.

When it is determined that the shape of the product on the cash register 3 is bag-shaped (ACT18, YES), the processor 41 determines whether the size of the product is within the gripping size of the bag (ACT19). In the present embodiment, the arm device 23 cannot (should not) grip a bag-shaped product having a size larger than the grip size of the bag. Whether or not the bag-shaped product can be gripped may be determined by the weight of the product.

When it is determined that the product is bag-shaped, the processor 41 determines whether or not to grip the product (whether it can be gripped) based on the size of the product. The processor 41 specifies (estimates) the size of the product from the image area of the product in the image (photographed image for product detection) captured on the cash register table 3. In the present embodiment, the size of the bag-shaped product is the length of the diagonal line specified from the edge position of the product detected in the captured image for product detection. However, the size of the bag-shaped product may be defined by the number of pixels in the image area of the product, an estimated value of the area of the image area of the product, or the like.

For example, the processor 41 detects four points that are edges of a rectangular region that is an image region of the product in the captured image for product detection. The processor 41 uses the length of the diagonal line of the rectangle consisting of the detected four points as the size of the product. In this case, as the gripping size of the bag, the maximum value of the size (diagonal length) of the product that can be lifted by pinching both ends with the two arms 14 and 15 is set. For example, if the grip size of the bag is 500 mm, the processor 41 determines whether or not the length of the diagonal line specified from the edge of the product in the captured image is within 500 mm.

When it is determined that the size of the product determined to be bag-shaped exceeds the gripping size of the bag (ACT19, NO), the processor 41 determines that the product cannot be gripped by the arm device 23. If it is determined that the product cannot be gripped by the arm device 23, the processor 41 proceeds to the ACT 24 and performs a product identification information reading process by the arm scanner 16.

When it is determined that the product determined to be bag-shaped is within the bag gripping size (ACT 19, YES), the processor 41 instructs the system controller 21 to grip the product with the arm device 23. Instruct the system controller 21 to grip the product with the arm device 23. The processor 41 specifies position information indicating the position of the product and shape information indicating the shape of the product such as shape, size, center, and orientation, based on a photographed image taken on the cash register table 3. The processor 41 notifies the system controller 21 of the position information and the shape information of the product specified from the captured image together with the fact that the product is the gripping target.

The processor 31 of the system controller 21 sets the grip for gripping the product by the arm device 23 based on the information from the product reading device 22 (ACT 20). When the product is bag-shaped, the processor 31 performs gripping settings for the product based on an algorithm for gripping the bag-shaped product and the position information and shape information of the product. Here, it is assumed that the processor 31 sets the grip on the arm device 23 so that the two arms 14 and 15 grip both ends of the bag-shaped product.

FIG. 5 is a diagram for explaining an operation example when the bag-shaped product P1 is gripped by the two arms 14 and 15.
The arm device 23 moves the arms 14 and 15 so that the gripping mechanisms 14b and 15b provided at the tips of the arm portions 14a and 15a pinch both ends of the bag-shaped product. The arms 14 and 15 are moved so that the gripping mechanisms 14b and 15b rub the upper surface of the cash register table 3 from the outside of both ends of the product toward the center of the product in parallel with the product. In the arm device 23, after the gripping mechanism 14b (15b) of one arm 14 (15) has pinched one end in the longitudinal direction of the product, the gripping mechanism 15b (14b) of the other arm 15 (14) is the product. Move to pinch the opposite end in the longitudinal direction.

In the example shown in FIG. 5, after the first arm 14 picks one end of the product P1 with the gripping mechanism 14b, the second arm 15 moves the gripping mechanism 15b in the direction of the arrow a. When the gripping position of the gripping mechanism 15b reaches the end of the product, the second arm 15 picks the end of the product. Thereby, the gripping mechanisms 14b and 15b can pinch both ends in the longitudinal direction of the product.
The processor 31 of the system controller 21 sets the grip according to the position information and the shape information of the product so that the arm device 23 can grip both ends of the product by the above-mentioned operation.

When it is determined that the shape of the product on the cash register table 3 is box-shaped (ACT21, YES), the processor 41 determines whether the size of the box-shaped product is within the gripping size of the box (ACT22). ). In the present embodiment, the arm device 23 cannot (should not) grip a box-shaped product having a width exceeding the gripping size of the box. That is, when it is determined that the product is box-shaped, the processor 41 determines whether or not the product is gripped (whether it can be gripped) based on the width of the product.

For example, the processor 41 specifies (estimates) the width of the product from the image (photographed image for product detection) taken on the cash register table 3. Since the photographed image for product detection is an image of the cash register table 3 taken from above, the box-shaped product placed on the cash register table 3 is photographed as a rectangular image in the photographed image for product detection. The processor 41 detects four points that are edges of the rectangular region of the product in the captured image for product detection. Further, when the product determined to be box-shaped is not captured as a completely rectangular image, the processor 41 may detect four points that are edges of the rectangular region circumscribing the image region of the product.

The processor 41 estimates the width of the product from four points which are the edges of the rectangular region of the product detected from the captured image for product detection. The width of the product shall be the width of the portion gripped by the gripping mechanisms 14b and 15b. In this case, as the gripping size of the box, the maximum value of the width of the product that the gripping mechanisms 14b and 15b of the arms 14 and 15 can hold from the side surface of the product is set. For example, if the grip size of the box is 100 mm, the processor 41 determines whether or not the width of the box-shaped product detected from the captured image for product detection is within 100 mm. Whether or not the box-shaped product can be gripped may be determined by the weight of the product.

When the processor 41 determines that the width of the box-shaped product on the cash register table 3 exceeds the gripping size of the box (ACT22, NO), the processor 41 determines that the product cannot be gripped by the arm device 23. If it is determined that the product cannot be gripped, the processor 41 proceeds to the ACT 24 and performs a product identification information reading process using the arm scanner 16.

When it is determined that the width of the product is within the gripping size of the box (ACT22, YES), the processor 41 instructs the system controller 21 to grip the product by the arm device 23. The processor 41 specifies position information indicating the position of the product and shape information indicating the shape of the product such as shape, size, center, and orientation, based on a photographed image taken on the cash register table 3. The processor 41 notifies the system controller 21 of the position information and the shape information of the product together with the fact that the product is the gripping target.

The processor 31 of the system controller 21 sets the grip for gripping the product by the arm device 23 based on the information from the product reading device 22 (ACT 22). When the product is box-shaped, the processor 31 performs gripping settings for the product based on an algorithm for gripping the box-shaped product and the position information and shape information of the product.

FIG. 6 is a diagram for explaining an example of a state in which the box-shaped product P2 is gripped by the two arms 14 and 15.
In the example shown in FIG. 6, the arm device 23 shall grip both sides of the box-shaped product by the gripping mechanisms 14b and 15b provided at the tip portions of the arm portions 14a and 15a. For example, the arm device 23 moves both ends of the arm portions 14a and 15a (grip mechanisms 14b and 15b) from above in the direction perpendicular to the product toward the cash register table 3 to grip both sides of the product. The processor 31 performs gripping settings according to the position information and shape information of the product so that the arm device 23 can grip both sides of the product by the operation as described above.

Further, when it is determined that the shape of the product on the cash register table 3 is cylindrical (ACT21, NO), the processor 41 determines that the product cannot be gripped by the arm device 23. If it is determined that the product cannot be gripped by the arm device 23, the processor 41 proceeds to the ACT 24 and performs a product identification information reading process by the arm scanner 16.

When the product grip setting is performed by the system controller 21, the processor 51 of the arm device 23 drives each arm 14.15 according to the grip setting to grip the product (ACT30).
For example, when gripping a bag-shaped product, the processor 51 moves the gripping mechanism 14b from the outside of the product toward the center of the product according to the grip setting to pinch one end of the product. After the gripping mechanism 14b has pinched one end of the product, the processor 51 moves the second arm 15 according to the gripping setting to pinch the other end of the product with the gripping mechanism 15b, as shown in FIG. As a result, the arm device 23 grips the two arms 14 and 15 so as to pinch both ends of the product.

When gripping a box-shaped product, the processor 51 moves the tip portions (grip mechanism) of the arms 14 and 15 from above the cash register table 3 to both sides of the product based on the position specified in the grip setting. The processor 51 grips one end of the product from above with the arm 14, and grips the other end of the product with the arm 15. As a result, as shown in FIG. 6, the arm device 23 grips both sides of the product with the two arms 14 and 15, respectively.

When the arms 14 and 15 grip the product, the processor 51 lifts the gripped product and moves it to the reading position of the scanner 13 in response to an instruction from the system controller 21 (ACT 31). When the lifted product is moved to the reading position of the scanner 13, the processor 51 rotates the product at the reading position of the scanner 13 in a predetermined direction and angle (ACT 32). The processor 51 rotates the product held by the arms 14 and 15 in the rotation procedure specified by the system controller 21.

For example, the processor 51 rotates the product 90 times in the forward direction (downward toward the scanner 13) and then rotates 180 ° C. in the reverse direction opposite to the front direction. If the product is box-shaped, the system controller 21 may be rotated in the lateral direction orthogonal to the front direction and the rear direction so that the side surface of the box can be read by the scanner 13.

The product is rotated by the arm device 23, and the processor 31 of the system controller 21 instructs the product reading device 22 to read the product identification information (bar code) of the product. In response to this, the processor 41 of the product reading device 22 performs a reading process of reading a barcode from the product rotated by the arm device 23 using the scanner 13 (ACT 33). In the present embodiment, it is assumed that the product identification information read by the scanner 13 is a barcode.

That is, the processor 41 executes the barcode reading process by the scanner 13 while the arm device 23 rotates the product. The scanner 13 reads the barcode from the appearance of the product rotated at the reading position by the arms 14 and 15. As a result, the processor 41 determines whether or not the scanner 13 has read the barcode as the product identification information (ACT 34).

When the barcode is read by the scanner 13 (ACT34, YES), the processor 41 outputs the product identification information read as the barcode as a reading result (ACT43). For example, when a bar code is read from a product, the processor 41 outputs information indicating the product specified by the bar code read by the scanner 13 to the checkout device 2 via the communication device 25.

Further, when the barcode cannot be read by the scanner 13 (ACT 34, NO), the processor 41 determines whether or not to retry the barcode reading process based on a predetermined condition (ACT 35). For example, it is assumed that the processor 41 executes the rotation of the product by the arm device 23 described above twice and attempts to read the barcode.
When it is determined that the barcode reading process by the scanner 13 is to be retried (ACT35, YES), the processor 41 returns to the ACT 32 and repeats the above-mentioned process.

If it is determined not to retry (ACT35, NO), the processor 41 executes the product identification information (percode) reading process (reading process in the stretched state) by the scanner 13 in a state where the product package is stretched. Whether or not it is determined (ACT36). The stretched state of the product package is a state in which the two arms 14 and 15 gripping both ends of the product are pulled between the gripping positions.

The enlargement of the product package assumes that the barcode printed on the package cannot be read due to creases or wrinkles in the product package. That is, a product whose barcode can be read by pulling both ends of the package (between the gripping positions by the two arms) is the target of stretching. In the present embodiment, it is assumed that the bar code reading process in the stretched state is executed for the bag-shaped product. In this case, the processor 41 determines whether or not to perform the reading process in the stretched state for the product, depending on whether or not the product is in the shape of a bag.

When it is determined to execute the reading process in the stretched state (ACT36, YES), the processor 41 instructs the system controller 21 to execute the reading process in the stretched state. Upon receiving an instruction to read the stretched state, the processor 31 of the system controller 21 sends an instruction to stretch the package of the product to the arm device 23 (ACT37).

The arm device 23 operates so as to pull both ends of the product held by both arms 14 and 15 from each other in response to an instruction from the system controller 21. Here, the gripping mechanism 14b of the first arm 14 and the gripping mechanism 15b of the second arm 15 grip both ends of the product, respectively. Therefore, the processor 51 controls the gripping mechanism 14b (the gripping position of the first arm 14) and the gripping mechanism 15b (the gripping position of the second arm 15) to move in opposite directions with a constant power. That is, the processor 51 operates the arm portion 14a and the arm portion 15a by the arm drive mechanism 54 so as to pull between the gripping positions of the arms 14 and 15 with a constant force. When the arms 14 and 15 are pulled with a constant force between the gripping positions, the arms 14 and 15 grip both ends of the product in a stretched state.

When the product is pulled between the gripping positions, the processor 51 rotates the product to be gripped in the state of being pulled between the gripping positions (the state in which the package is stretched) at the reading position of the scanner 13 (ACT38). For example, the processor 51 of the arm device 23 lifts the product to the reading position of the scanner 13 while pulling between the gripping positions. After lifting the product to the reading position of the scanner 13, the processor 51 rotates the product at the reading position of the scanner 13 in a predetermined direction and angle. The rotation procedure for rotating the product to be gripped while the arms 14 and 15 are pulled may be the same procedure as that for the ACT 32.

While rotating the stretched product on the arm device 23, the processor 31 of the system controller 21 instructs the product reading device 22 to read the barcode (product specific information) of the product. In response to this, the processor 41 of the product reading device 22 performs a process of reading a barcode as product identification information from the product rotated by the arm device 23 using the scanner 13 (ACT 39).

That is, the processor 41 executes the barcode reading process by the scanner 13 while the arm device 23 rotates the product in the stretched state. The scanner 13 reads the barcode from the appearance of the product rotated at the reading position by the arms 14 and 15. As a result, the processor 41 determines whether or not the scanner 13 has read the barcode as the product identification information (ACT 40).

When the barcode is read by the scanner 13 (ACT 40, YES), the processor 41 outputs the product identification information read as the barcode as a reading result (ACT 43). For example, when a bar code is read from a product, the processor 41 outputs information indicating the product specified by the bar code read by the scanner 13 to the settlement device 2 via the communication device 25.

Further, when the scanner 13 cannot read the product specific information (ACT 40, NO), the processor 41 determines whether or not to retry the reading process in the stretched state based on a predetermined condition (ACT 41). When it is determined to retry the barcode reading process in the stretched state (ACT35, YES), the processor 41 returns to the ACT 38 and repeats the above-mentioned process.

When it is determined not to retry (ACT 41, NO), the processor 41 executes a product recognition process (reading process of product specific information) based on the captured image captured by the camera 12 (ACT 42). In the product recognition process based on the captured image of the camera 12, it is assumed that the product whose barcode cannot be read due to dirt or the like is the processing target. In the product recognition process, the processor 41 photographs the product held by the arm device 23 with the camera 12, and recognizes the product from the captured image captured by the camera 12. The product recognition process will be described in detail later.

The processor 41 outputs the recognition result by the product recognition process as the reading result of the product identification information (ACT43). For example, when the product cannot be recognized by the product recognition process, the processor 41 outputs to the settlement device 2 that the product identification information cannot be read via the communication device 25. When the product can be recognized by the product recognition process, the processor 41 outputs the information indicating the recognized product to the settlement device 2 via the communication device 25.

Next, the barcode reading process by the arm scanner 16 will be described.
FIG. 7 is a flowchart for explaining an operation example of the barcode reading process by the arm scanner 16.
According to the above-mentioned operation example, the system targets a product whose shape is not bag-shaped or box-shaped, or a product whose size exceeds a specified value, to be read by the arm scanner 16. The arm scanner 16 is provided at the tip of the arm 14, and the reading position is operated by the movement of the arm 14. In the reading process by the arm scanner 16, the product surface is scanned by the arm scanner 16 by moving the tip of the arm 14 along the product placed on the cash register table 3.

When the processor 41 of the product reading device 22 determines that the product is the target of the reading process by the arm scanner 16, the processor 41 requests the system controller 21 to perform the reading process by the arm scanner 16. Further, the processor 41 supplies various information (the shape of the product, the center position of the product, the edge position of the product, etc.) specified from the image for product detection taken by the camera 12 to the system controller 21. When the reading process by the arm scanner 16 is executed, the processor 31 of the system controller 21 confirms the shape of the product.

When the product is determined to be bag-shaped (ACT61, YES), the processor 31 of the system controller 21 instructs the arm device 23 to move the arm scanner 16 so as to scan the upper surface of the product. For example, the processor 31 determines a position (movement path) to be scanned by the arm scanner 16 based on the edge position of the product.

For example, the processor 31 sets a first path for moving the arm scanner 16 in a square shape so as to connect the four edges presumed to be the four corners of the product. Further, the processor 31 sets a second path for moving the arm scanner 16 along a diagonal line consisting of four edges when the barcode cannot be read in the first path. As a result, the arm scanner 16 can scan the entire upper surface of the product placed on the cash register table 3. Further, by setting the first path and the second path, the arm scanner 16 can scan the entire product in the second path after first scanning the vicinity of the outer periphery of the product in the first path.

The processor 51 of the arm device 23 moves the arm scanner 16 according to the instruction from the system controller 21 to scan the upper surface of the bag-shaped product (ACT 62). The processor 51 drives the arm 14 so that the arm scanner 16 moves along the path instructed by the system controller 21 by the arm drive mechanism 54. The arm scanner 16 scans a barcode from the upper surface of a bag-shaped product placed on the cash register table 3 while moving the scan position by the arm 14.

The processor 51 of the arm device 23 supplies the reading result by the arm scanner 16 to the system controller 21. The processor 31 of the system controller 21 outputs the reading result of the barcode from the arm device 23 to the settlement device 2 via the communication device 25 as the reading result of the product identification information.

If the product is determined to be box-shaped (ACT63, YES), the processor 31 of the system controller 21 instructs the arm device 23 to move the arm scanner 16 so as to scan the upper surface of the product. For example, the processor 31 determines a position (movement path) to be scanned by the arm scanner 16 based on the edge position of the product. For example, the processor 31 sets a square-shaped first path connecting the four edges presumed to be the four corners of the product and a second path along the diagonal of the four edges, as in the above example.

The processor 51 of the arm device 23 moves the arm scanner 16 according to the instruction from the system controller 21 to scan the upper surface of the box-shaped product (ACT64). The processor 51 drives the arm 14 so that the arm scanner 16 moves along the path instructed by the system controller 21 by the arm drive mechanism 54. The arm scanner 16 scans the barcode from the upper surface of the box-shaped product placed on the cash register table 3 while moving the scanning position by the arm 14.

The processor 51 of the arm device 23 supplies the reading result of the barcode by the arm scanner 16 to the system controller 21. The processor 31 of the system controller 21 determines whether or not the arm scanner 16 can read the barcode from the upper surface of the product based on the reading result from the arm device 23 (ACT65). When the barcode is read from the upper surface of the product (ACT65, NO), the processor 31 outputs the barcode reading result to the settlement device 2 via the communication device 25.

If the barcode cannot be read from the top surface of the product (ACT65, YES), the processor 31 instructs the arm device 23 to scan the side surface of the product with the arm scanner 16. For example, the processor 31 sets a path for the arm scanner 16 to scan the side surface of the product based on the edge position of the product.

The processor 51 of the arm device 23 moves the arm scanner 16 according to the instruction from the system controller 21 to scan the side surface of the box-shaped product (ACT66). The processor 51 scans the barcode from the side surface of the product while moving the arm scanner 16 along the side surface of the product on the cash register table 3 by the arm drive mechanism 54. The processor 31 of the system controller 21 outputs the reading result of the barcode from the arm device 23 to the settlement device 2 via the communication device 25 as the reading result of the product identification information.

When it is determined that the product is tubular (neither bag-shaped nor box-shaped) (ACT63, NO), the processor 31 of the system controller 21 has a predetermined height (for example, height 100 mm) of the product on the cash register table 3. ) Is within (ACT70). Here, the height of the product is the height from the upper surface of the cash register table 3 (the surface on which the product is placed) to the top of the product placed on the cash register table 3. The height of the product is estimated from, for example, a photographed image taken by the camera 12. Further, a height detection sensor for detecting the height of the product may be provided. The predetermined height is a threshold value for determining whether the arm 14 can be moved above the product. That is, the arm 14 can move freely at a position lower than the predetermined height, and cannot move freely at a height higher than the predetermined height.

When the height of the product is within the predetermined height (ACT70, YES), the processor 31 reads the barcode from the peripheral surface of the product with the arm scanner 16 by moving the arm 14 above the product (ACT71). That is, for a product whose tubular height is within a predetermined height, the arm portion 14a is moved above the product in a circular motion to read the peripheral surface of the product with the arm scanner 16.

FIG. 8 is a diagram for explaining an operation example of the arm scanner 16 when the height of the cylindrical product P3 is within a predetermined height.
In the present embodiment, the arm 14 can move freely at a low position within a predetermined height. Therefore, if the height of the product is within the predetermined height, the arm 14 can freely move above the product. As a result, the arm scanner 16 provided at the tip of the arm portion 14a extending from above the product by moving the arm portion 14a above the product can freely move around the peripheral surface of the product. In the example shown in FIG. 8, the arm portion 14a is moved as shown by the arrow b1 in the space including the upper part of the product. As shown by the arrow b2, the arm scanner 16 provided at the tip of the arm portion 14a moves along the peripheral surface of the product as the arm portion 14a moves in the space including the upper part of the product.

The processor 31 of the system controller 21 instructs the arm device 23 to scan the barcode by moving the arm scanner 16 on the peripheral surface of the product by a predetermined angle (for example, 360 degrees). However, the bar code on the peripheral surface of the product facing the scanner 13 may be readable by the scanner 13. In this case, the arm scanner 16 may move about 270 degrees along the peripheral surface of the product.

The processor 51 of the arm device 23 drives the arm 14 by the arm drive mechanism 54 so that the arm scanner 16 moves along the peripheral surface of the product according to the instruction from the system controller 21. As a result, the arm scanner 16 scans the barcode printed on the peripheral surface of the product while moving along the peripheral surface of the product placed on the cash register table 3.

The processor 51 of the arm device 23 supplies the reading result of the barcode by the arm scanner 16 to the system controller 21. The processor 31 of the system controller 21 outputs the reading result of the barcode from the arm device 23 to the settlement device 2 via the communication device 25 as the product identification information of the product.

When the height of the product exceeds a predetermined height (ACT70, YES), the processor 31 reads the barcode from the peripheral surface of the product with the arm scanner 16 by moving the arm 14 to the side of the product. When moving the arm 14 to the side of the product, it is difficult to rotate the arm scanner 16 360 degrees. Therefore, the processor 31 divides the peripheral surface of the product into several areas and scans the barcode. In the present embodiment, an operation example in which the product is divided into an upper part and a lower part and the lower part and the upper part are scanned for half a circumference will be described.

FIG. 9 is a diagram for explaining an operation example of the arm scanner 16 when the height of the tubular product P4 exceeds a predetermined height.
In the example shown in FIG. 9, the arm scanner 16 provided at the tip of the arm 14 scans the lower peripheral surface and the upper peripheral surface of the cylindrical product P4 by half a circumference. For example, as shown by the arrow c1, the arm scanner 16 scans the barcode by making a half turn clockwise from the initial position of the lower part of the product P4. If the barcode cannot be read by this, the arm scanner 16 moves to the initial position of the upper part of the product P4 and scans the barcode by making a half turn clockwise around the peripheral surface of the product, as shown by the arrow c2.

If the barcode cannot be read even after this, the arm scanner 16 moves to the initial position at the bottom of the product P4 again, and makes a half turn counterclockwise (dotted arrow c3) around the peripheral surface of the product P4 to read the barcode. to scan. If the barcode cannot be read even after this, the arm scanner 16 moves to the initial position of the upper part of the product P4 again, makes a half turn counterclockwise around the peripheral surface of the product P4, and scans the barcode. By these operations, the arm 14 moves the arm scanner 16 from the side of the product P4, so that the entire peripheral surface of the product P4 is scanned by the arm scanner 16.

First, the processor 31 instructs the arm device 23 to move the arm scanner 16 from the initial position of the lower part of the product by half a turn clockwise along the peripheral surface of the product to scan the barcode. The processor 51 of the arm device 23 operates the arm 14 to move the arm scanner 16 to the initial position at the bottom of the product. When the arm scanner 16 is moved to the initial position at the bottom of the product, the processor 51 operates the arm 14 on the side of the product to move the arm scanner 16 clockwise along the peripheral surface of the product by half a turn ( ACT72). As a result, the arm scanner 16 scans the barcode on the peripheral surface of the lower half of the product.

If the barcode cannot be read from the peripheral surface of the lower half of the product (ACT73, YES), the processor 31 instructs the arm device 23 to scan the upper half of the product clockwise. The processor 51 of the arm device 23 operates the arm 14 in response to an instruction from the system controller 21 to move the arm scanner 16 to the initial position on the upper part of the product. When the arm scanner 16 moves to the initial position on the upper part of the product, the processor 51 moves the arm scanner 16 clockwise along the peripheral surface of the product by half a turn by operating the arm 14 on the side of the product ( ACT74). As a result, the arm scanner 16 scans the barcode on the peripheral surface of the upper half of the product.

If the barcode cannot be read from the peripheral surface of the upper half of the product (ACT75, YES), the processor 31 instructs the arm device 23 to scan the remaining half of the peripheral surface of the lower part of the product. The processor 51 of the arm device 23 operates the arm 14 in response to an instruction from the system controller 21 to move the arm scanner 16 to the initial position at the bottom of the product. When the arm scanner 16 moves to the initial position at the bottom of the product, the processor 51 operates the arm 14 on the side of the product to move the arm scanner 16 counterclockwise by half a turn along the peripheral surface of the product. (ACT76). As a result, the arm scanner 16 scans the barcode on the peripheral surface of the remaining half circumference at the bottom of the product.

If the barcode cannot be read from the remaining half-circumferential surface at the bottom of the product (ACT77, YES), the processor 31 instructs the arm device 23 to scan the remaining half-circumferential surface at the top of the product. .. The processor 51 of the arm device 23 operates the arm 14 in response to an instruction from the system controller 21 to move the arm scanner 16 to the initial position on the upper part of the product. When the arm scanner 16 moves to the initial position at the top of the product, the processor 51 moves the arm 14 to the side of the product to move the arm scanner 16 counterclockwise by half a turn along the peripheral surface of the product. (ACT78). As a result, the arm scanner 16 scans the barcode on the peripheral surface of the remaining half circumference at the top of the product.

The processor 31 of the system controller 21 acquires the reading result of the barcode by the arm scanner 16 from the arm device 23 operated as described above. The processor 31 of the system controller 21 outputs the reading result of the barcode from the arm device 23 to the settlement device 2 via the communication device 25 as the product identification information of the product.

Next, the product recognition process for the captured image captured by the camera 12 will be described.
FIG. 10 is a flowchart for explaining an operation example of the product recognition process for the captured image captured by the camera 12.
In the operation example shown in FIGS. 3 and 4 described above, the system controller 21 executes the product recognition process based on the captured image of the camera 12 when the barcode cannot be read by the scanner 13. The system controller 21 instructs the arm device 23 to move the product to the shooting position of the camera 12 when performing the product recognition process based on the captured image of the camera 12. The processor 51 of the arm device 23 moves the product held by the arms 14 and 15 to the shooting position of the camera 12 according to the instruction from the system controller 21 (ACT81).

When the product moves to the shooting position of the camera 12, the system controller 21 rotates the product held by the arms 14 and 15 in order to shoot an image including the product identification information. It is assumed that the system controller 21 detects the image of the product identification information from the captured image taken by the camera 12 by the product reading device 22 while rotating the product by the arm device 23.

In the present embodiment, the product is rotated toward the camera 12 in the front-back direction (front (front) and vice versa) by a predetermined angle (for example, 15 degrees), and the image of the product identification information is taken from the image taken by the camera 12. Shall be detected. Further, in the present embodiment, it is assumed that the barcode cannot be read by the scanner 13 due to dirt or the like, and the product is recognized by the numerical string attached to the lower side of the barcode in the barcode area. Therefore, in the following processing, the product reading device detects the image of the barcode region (bar code image) as the image of the product identification information, and recognizes the product from the number string in the barcode image.

That is, the system controller 21 determines the rotation angle of the product held by the arms 14 and 15, and instructs the arm device 23 to rotate the product at the determined angle. The arm device 23 rotates the product held by the arms 14 and 15 in response to an instruction from the system controller 21 (ACT 82). For example, the system controller 21 rotates the product forward (front) by a predetermined angle (for example, 15 degrees). When the forward rotation angle reaches 90 degrees, the system controller 21 rotates the product backward by a predetermined angle (for example, 15 degrees), and rotates the product until the backward rotation angle reaches 180 degrees.

When the arm device 23 rotates the product, the system controller 21 instructs the product reading device 22 to detect an image (bar code image) of the product identification information from the image taken by the camera 12. The processor 41 of the product reading device 22 captures an image by the camera 12 in response to an instruction from the system controller 21 (ACT83). When the camera 12 captures an image, the processor 41 of the product reading device 22 detects a barcode image from the captured image captured by the camera 12 (ACT 84). The processor 41 of the product reading device 22 notifies the system controller 21 of information indicating whether or not the barcode image is detected.

When the barcode image is not detected from the captured image of the camera 12 (ACT85, NO), the system controller 21 further rotates the product to determine whether to detect the barcode image (ACT86). When it is determined that the bar code image is detected by further rotating the product (ACT86, NO), the system controller 21 returns to the ACT 82 and executes the above-mentioned process again. When it is determined that the detection of the barcode image is finished (ACT86, YES), the system controller 21 assumes that the product recognition by the captured image of the camera 12 has failed, and ends the process.

When the barcode image can be detected from the captured image of the camera 12 (ACT85, NO), the system controller 21 adjusts the orientation of the product and captures the image for product recognition processing with the camera 12. For example, the system controller 21 acquires a photographed image taken by the camera 12 in a state where the product is rotated (tilted) by a predetermined angle (for example, 5 degrees) in the left-right direction toward the camera 12. In this case, the system controller 21 rotates the product held by the arms 14 and 15 to the right by 5 degrees, and causes the camera 12 to take an image. When the rotation angle to the right reaches 45 degrees, the system controller 21 returns the product to the original angle, then rotates the product to the left by 5 degrees, and causes the camera 12 to take an image. When the rotation angle to the left reaches 45 degrees, the system controller 21 causes the product reading device 22 to perform product recognition based on the captured images taken at each angle.

That is, the system controller 21 determines the orientation of the product held by the arms 14 and 15, and instructs the arm device 23 of the determined orientation of the product. The arm device 23 adjusts the orientation of the product gripped by the arms 14 and 15 in response to an instruction from the system controller 21 (ACT87). For example, the system controller 21 instructs the arm device 23 to rotate the product 5 degrees to the right until the rotation angle to the right reaches 45 degrees. Further, when the rotation angle to the right reaches 45 degrees, the system controller 21 instructs the arm device 23 to rotate the product to the left by 5 degrees after returning the product to the original angle.

When the arm device 23 adjusts the orientation of the product, the system controller 21 instructs the product reading device 22 to take an image for product recognition. The product reading device 22 captures an image with the camera 12 in response to an instruction from the system controller 21 (ACT88). The product reading device 22 stores the captured image taken by the camera 12 in the image memory 44 as an image for product recognition (ACT 89).

When the image taken by the camera 12 is saved as the image for product recognition, the system controller 21 determines whether or not to end the shooting of the image for product recognition by the camera 12 (ACT90). When it is determined that the shooting by the camera 12 is not completed (ACT 90, NO), the system controller 21 returns to the ACT 87 and executes the above-mentioned process again.

When it is determined that the shooting by the camera 12 is finished (ACT90, NO), the system controller 21 instructs the product reading device 22 to recognize the product by image processing. The processor 41 of the product reading device 22 executes the product recognition process based on the captured image (image for the product recognition process) of the camera 12 stored in the image memory 44 (ACT91). The processor 41 of the product reading device 22 recognizes the numbers under the barcode based on the plurality of images taken by the camera 12 by executing the character recognition program.

For example, the processor 41 extracts a barcode image from each captured image and detects a number under the barcode from the barcode image. When all the numbers under the barcode are detected, the processor 41 recognizes the detected numbers and uses the numerical information in which the recognized numbers are arranged as the product recognition result. The processor 41 notifies the system controller 21 of the product recognition result. The system controller 21 transmits the product recognition result from the product recognition device to the settlement device 2 via the communication device 25. The settlement device 2 identifies the product from the numerical information under the barcode and executes the settlement process of the product.

As described above, the product reading system according to the present embodiment includes a camera, an arm device, a scanner for reading a barcode, a processor, a controller, and the like. The camera takes a picture of the product placed on the cash register. The processor determines whether or not the product is bag-shaped from the image taken by the camera. When the product is bag-shaped, the controller causes the gripping mechanism of the first arm and the gripping mechanism of the second arm to grip both ends of the product, respectively, and the gripping position of the first arm and the gripping mechanism of the second arm. The product is moved to the reading position of the scanner while the gripping position is pulled from each other. The arm device grips both ends of the product by the gripping mechanism of the first arm and the gripping mechanism of the second arm, and reads the product in a state where both ends of the product gripped by the two gripping mechanisms are pulled from each other. Put up in position. The scanner reads a barcode as product identification information from a product that the arm device grips with both ends pulled.

As a result, the product reading system according to the present embodiment can read the barcode with the product package stretched in both directions. As a result, even if the product has wrinkles or creases in the package, the scanner can make the barcode printed on the package easy to read, and the scanner can reliably read the barcode.

Further, the controller of the product reading system according to the present embodiment rotates the product to be gripped in a state where the gripping position of the first arm and the gripping position of the second arm are pulled from each other at the reading position of the scanner. As a result, the product reading system according to the present embodiment can reliably read the barcode printed on the package by the scanner even if the product has wrinkles or folds on the package.

Further, when the scanner cannot read the barcode of the product, the controller of the product reading system according to the present embodiment rotates the product held by the arm in the shooting area of the camera to cause the camera to shoot an image. The processor recognizes the product identification information by image recognition for the image of the rotating product taken by the camera. As a result, the product reading system can specify the product identification information by image recognition for the image taken by the camera while moving the product held by the arm for the product whose barcode cannot be read even when both ends are pulled. can.

Further, in the controller of the product reading system according to the present embodiment, the scanner reads the barcode as the product identification information, the processor extracts the barcode image from the image taken by the camera, and writes it under the barcode. The product identification information is recognized by the character recognition of the displayed number. As a result, the product reading system can read the barcode even when both ends are pulled. For the product, the number written under the barcode in the image taken by the camera while moving the product held by the arm. Product specific information can be specified by character recognition.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 ... Product reading system (robot), 2 ... Settlement device (registration), 3 ... Cash register stand (table), 11 ... Sensor, 12 ... Camera, 13 ... Scanner (fixed scanner), 14 ... 1st arm, 15 ... No. 2 arms, 14b, 15b ... gripping mechanism, 16 ... arm scanner, 21 ... system controller (control device), 22 ... product reading device (product reader), 23 ... arm device, 25 ... communication device, 31, 41, 51, 61, 71 ... Processor.

Claims (5)

A camera that shoots the table on which the product is placed from above,
An arm device having a first arm and a second arm, each provided with a gripping mechanism, and
A scanner that reads the product specific information that exists in the appearance of the product at the reading position,
When the shape of the product detected from the image taken by the camera is bag-shaped, both ends of the product are gripped by the gripping mechanism of the first arm and the gripping mechanism of the second arm, respectively, and the first. A controller that moves the product to the reading position of the scanner while pulling the gripping position of the arm and the gripping position of the second arm to each other.
Product reading system with. The controller rotates the product that grips the gripping position of the first arm and the gripping position of the second arm while pulling each other at the reading position of the scanner.
The product reading system according to claim 1. It has a processor that recognizes product specific information by image recognition for an image taken by the camera.
When the scanner cannot read the product identification information of the product, the controller rotates the product held by the first and second arms in the shooting area of the camera, and the controller is based on an image taken by the camera. Make the processor recognize the product specific information of the product,
The product reading system according to any one of claims 1 or 2. The scanner reads the barcode as product identification information and
The processor extracts a barcode image from an image taken by the camera, and recognizes product identification information by character recognition of a number written under the barcode.
The product reading system according to claim 3. A communication device having a communication device that outputs product identification information of the product held by the first and second arms to a payment device that performs payment processing of the product.
The product reading system according to any one of claims 1 to 4.

Images