JP7106331B2 - Product reading system - Google Patents

Description

Embodiments of the present invention relate to merchandise reading systems.

2. Description of the Related Art Conventionally, retail stores such as convenience stores and supermarkets perform checkout processing for the product after making the checkout device read product specific information such as a bar code. Most of the checkout devices are operated by retail store clerks and read barcodes printed on merchandise manually by the clerks. In recent years, due to labor shortages, etc., there is a demand for labor-saving work (merchandise reading work) for the settlement device to read the merchandise to be settled.

As a conventional technique, a device provided with a robot arm for gripping an article such as a product has been proposed. However, a robot arm provided in a conventional device moves a gripped article to a predetermined location or puts the gripped article into a storage section such as a bag or a box. For this reason, with the conventional technology, it is not possible to save labor in the product reading work of reading the bar code of the product.

JP 2016-167246 A

SUMMARY OF THE INVENTION It is an object of the present invention to provide a product reading system capable of saving labor in reading product identification information on products.

According to an embodiment, a product reading system has a camera, an arm device, a scanner, and a controller. The camera shoots the table on which the product is placed from above. The arm device grips the product placed on the table. The scanner reads product identification information present on the exterior of the product at the reading position. The controller causes the arm device to grip the product with grip settings that use different algorithms according to the shape of the product detected from the image captured by the camera, and the product gripped by the arm device is transferred to the scanner. Rotate at the reading position to cause the scanner to read the product identification information.

FIG. 1 is a schematic diagram schematically showing the external configuration of a product reading system (product reading support system) according to an embodiment. FIG. 2 is a block diagram showing a configuration example of the control system of the robot 1 as the commodity reading system according to the embodiment. FIG. 3 is a flowchart for explaining an operation example of the robot as the commodity reading system according to the embodiment. FIG. 4 is a flowchart for explaining an operation example of the robot as the commodity reading system according to the embodiment. FIG. 5 is a diagram for explaining an operation example when the product reading system according to the embodiment grips a bag-like product with an arm. FIG. 6 is a diagram for explaining an example of a state in which the product reading system according to the embodiment grips a box-shaped product with an arm; FIG. 7 is a flowchart for explaining an operation example of barcode reading processing by an arm scanner in the product reading system according to the embodiment. FIG. 8 is a diagram for explaining an operation example of the arm scanner when the height of the tubular product is within a predetermined height in the product reading system according to the embodiment. FIG. 9 is a diagram for explaining an operation example of an arm scanner when the height of a cylindrical product exceeds a predetermined height in the product reading system according to the embodiment; FIG. 10 is a flowchart for explaining an operation example of product recognition processing for an image captured by a camera in the product reading system according to the embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 is a schematic diagram schematically showing the external configuration of a product reading support system (robot) 1 as a product reading system according to an embodiment.
A product reading support system (hereinafter also simply referred to as a robot) 1 is a device for acquiring (reading) information for specifying a product (product specific information). In the configuration example shown in FIG. 1, the robot 1 assists the cash register operation of acquiring information specifying the product for which payment is to be made at the checkout device (cash register) 2 . For example, the robot 1 has a humanoid appearance that imitates the upper body of a person so that the user can easily use it.

The robot 1 is connected to a settlement device 2 that settles the price of goods. The robot 1 and the settlement device 2 constitute a transaction system for performing commodity settlement operations. A transaction system including the robot 1 and the settlement device 2 is installed, for example, in retail stores such as convenience stores and supermarkets. In a transaction system, a robot 1 is provided near a checkout table 3 for settlement on which a settlement device 2 is installed. A user puts a product to be settled on a cash register table 3 in front of a robot 1 by using a settlement device 2. - 特許庁The robot 1 reads product specific information from the product placed on the register table 3 by the user, and supplies the read product specific information to the settlement device 2 .

The product identification information may be any information that can be read from the appearance of the product. The product identification information is a barcode printed on the package of the product. Further, the product identification information may be a character string such as numbers and characters that can identify the product, an image of a label attached to the product, or a silhouette of the product as a whole. A robot 1 lifts and moves a commodity on a register table 3 so that commodity specifying information such as a bar code on the appearance of the commodity can be read by a scanner or a camera. The robot 1 also has a function of moving the scanner along the product that cannot be lifted to read product specific information such as a bar code.

In the configuration example shown in FIG. 1, a robot (main body) 1 has a sensor 11, a camera 12, a scanner 13, a first arm 14, a second arm 15, an arm scanner 16, a speaker 17, a microphone 18, a display 19, and the like. .
The sensor 11 is a sensor that detects a person approaching the cash register table 3 or that a product is placed on the cash register table 3 . For example, the sensor 11 is an infrared sensor that uses infrared rays to detect people or goods. The sensor 11 may be any sensor that can detect the presence or absence of a person or a product.

A camera 12 captures an image around the register table 3 . The camera 12 has a function of changing the photographing area by adjusting the photographing direction or the photographing distance. For example, the camera 12 has a function of setting a photographing area for photographing a person using the cash register 2 (or a person placing an item on the cash register table 3) and photographing a person near the cash register table 3. . The camera 12 also has a function of setting a photographing area for photographing the product placed on the cash register table 3 and photographing an image on the cash register table 3 .

The scanner 13 reads product identification information from the appearance of the product. In this embodiment, the scanner 13 is assumed to be a bar code scanner that reads a bar code printed on a part of the product package as the product identification information. Note that the scanner 13 may be any product specific information from the appearance of the product, and may be configured with a video camera or the like for reading two-dimensional bar codes, character strings, and the like.

The first arm 14 and the second arm 15 move relative to the product placed on the register table 3 . The first arm 14 and the second arm 15 have arm portions 14a, 15a and gripping mechanisms 14b, 15b. The arm portions 14a and 15a are articulated robot arms attached to the main body of the robot 1 . The gripping mechanisms 14b and 15b are mechanisms for gripping products. The arm portions 14a and 15a may be any device that allows the gripping positions of the gripping mechanisms 14b and 15b to move freely three-dimensionally. For example, in the first arm 14 and the second arm 15, the grasping mechanisms 14b and 15b are provided at the tip portions of the arm portions 14a and 15a, respectively. That is, the first arm 14 and the second arm 15 lift or rotate the product by moving the arm portions 14a and 15a while the product is gripped by the gripping mechanisms 14b and 15b.

The gripping mechanisms 14b and 15b are configured to pinch, pinch, or suck the merchandise on the register table 3. As shown in FIG. For example, the gripping mechanisms 14b, 15b are composed of multiple fingers (such as two clamping members or three to five fingers). Moreover, the gripping mechanisms 14b and 15b are not limited to those configured by a plurality of fingers, and may be a mechanism for sucking a part of the product. Also, the gripping mechanisms 14b and 15b may be configured by combining a plurality of fingers, a holding member, a mechanism for attracting, and the like.

Also, the arm scanner 16 is a scanner whose reading position is moved by the operation of the first arm 14 . For example, the arm scanner 16 is provided at the tip of the first arm 14, and the reading position is moved by the operation of the first arm 14. FIG. In this embodiment, the arm scanner 16 is assumed to read the bar code as the product identification information from the appearance of the product at the reading position moved by the first arm 14 . Note that the arm scanner 16 may be provided on the second arm 15 or may be provided on both the first arm 14 and the second arm 15 . Also, the arm scanner 16 may be provided separately from the first arm 14 and the second arm 15 .

The speaker 17 outputs sound. The speaker 17 outputs voice such as guidance to the user. A microphone 18 inputs voice. The microphone 18 inputs voice uttered by the user. The speaker 17 and the microphone 18 may constitute an interactive device for inputting/outputting voices interactively with the user. The display 19 displays images. For example, the display 19 displays guidance to the user by means of images or text. Speaker 17, microphone 18 and display 19 function as a user interface.

Next, the configuration of the control system in the robot 1 as the product reading device according to the embodiment will be described.
FIG. 2 is a block diagram showing a configuration example of a control system of the robot 1 as a product reading system (a product reading support system) according to the embodiment.
The robot 1 has a system controller (control device) 21, a product reading device (product reader) 22, an arm device 23, a dialogue device (user interface) 24, a communication device (communication module) 25, and the like.

The system controller 21 comprehensively controls each device in the robot. In the configuration example shown in FIG. 2 , the system controller 21 has a processor 31 , a memory 32 and an interface (I/F) 33 .
The processor 31 implements various processes by executing programs. Processor 31 is, for example, a CPU. The memory 32 is composed of memories such as ROM, RAM, and rewritable non-volatile memory. For example, memory 32 includes non-volatile memory that stores programs, control data, and the like. The memory 32 also includes working memory such as RAM for temporarily storing data.

Programs stored in the memory 32 include a basic program (OS) and various application programs. The application programs include programs for executing various processes described later. The processor 31 implements various processes described later by executing various programs stored in the memory 32 .

An interface (I/F) 33 is an interface for communicating with each device in the robot 1 .
The processor 31 gives control instructions to each device and acquires information from each device via the interface 33 . For example, the processor 31 acquires information indicating the reading result of the product identification information, the detection result of the person, and the like from the product reading device 22 . The processor 31 also instructs the arm device 23 to grip the product using the first arm 14 and the second arm 15 .

Next, a configuration example of the control system of the commodity reading device 22 will be described.
The product reading device 22 reads product identification information from products. The commodity reading device 22 also has a function of detecting a person, a commodity, and the like. In the configuration example shown in FIG. 2, the product reading device 22 has a processor 41, a memory 42, an interface (I/F) 43, an image memory 44, a camera driving mechanism 45, a sensor 11, a camera 12, a scanner 13, and the like.

The processor 41 implements various processes by executing programs. Processor 41 is, for example, a CPU. The memory 42 is composed of memories such as ROM, RAM, and rewritable non-volatile memory. For example, memory 42 includes non-volatile memory that stores programs, control data, and the like. The memory 42 also includes working memory such as RAM for temporarily storing data.

An interface (I/F) 43 is an interface for communicating with the system controller 21 .
The image memory 44 stores images captured by the camera 12 . A camera driving mechanism 45 controls the camera 12 . The camera drive mechanism 45 may control the area captured by the camera 12 by moving the orientation of the camera 12 .

The processor 41 implements various processes by executing programs stored in storage areas 42 a to 42 e of the memory 42 . For example, storage area 42a stores a program for executing processing for identifying persons and objects. The processor 41 functions as a person identifying section or an object identifying section by executing a program stored in the storage area 42a. For example, the processor 41, as a person identification unit, uses an image captured by the camera 12 to confirm whether the user is in front of the cash register 2 (in front of the cash register table 3). Also, the processor 41 may estimate the user's features (gender, age, etc.) from the image captured by the camera 12 . Moreover, the processor 41 may check whether there is any product on the cash register table 3 as the product identification unit.

The storage area 42b stores a program for recognizing the size and shape of an article. The processor 41 functions as an appearance recognition unit that recognizes the size and shape of an object by executing a program stored in the storage area 42b. For example, the processor 41 recognizes information indicating characteristics such as the size and shape of the product on the cash register table 3 from the image captured by the camera 12 . In this embodiment, the processor 41, as an appearance recognition unit, confirms whether the product is a bag-shaped product, a box-shaped product, a cylindrical product, or a bag-shaped product. .

The storage area 42c is assumed to store a program for executing the process of detecting the position. The processor 41 functions as a position detector that detects a position by executing a program stored in the storage area 42c. For example, processor 41 detects the position of the product on checkout table 3 based on the image captured by camera 12 .

The storage area 42d is assumed to store a program for executing processing for detecting a bar code as product identification information. The processor 41 functions as a barcode detector that detects the barcode attached to the product by executing the program stored in the storage area 42d. For example, the processor 41 detects the barcode attached to the product on the cash register table 3 from the image read by the scanner 13 or the arm scanner 16 .

The storage area 42e stores a program for executing processing for recognizing characters as product identification information. The processor 41 functions as a character recognition section (OCR processing section) that recognizes characters attached to products by executing a program stored in the storage area 42e. For example, the processor 41 recognizes characters as product identification information attached to the product on the cash register table 3 from the image captured by the camera 12 .

Next, a configuration example of the control system of the arm device 23 will be described.
The arm device 23 operates the first arm 14 and the second arm 15 . In the configuration example shown in FIG. 2, arm device 23 has processor 51, memory 52, interface (I/F) 53, arm drive mechanism 54, first arm 14, second arm 15, arm scanner 16, and the like.

The processor 51 implements various processes by executing programs. Processor 51 is, for example, a CPU. The memory 52 is composed of memories such as ROM, RAM, and rewritable non-volatile memory. For example, memory 52 includes non-volatile memory that stores programs, control data, and the like. The memory 52 also includes working memory such as RAM for temporarily storing data.
An interface (I/F) 53 is an interface for communicating with the system controller 21 .

The arm drive mechanism 54 controls movements of the first arm 14 and the second arm 15 . For example, the arm drive mechanism 54 moves the first arm 14 and the second arm 15 according to operation instructions from the system controller 21 . For example, the arm driving mechanism 54 grips the product on the cash register table 3 according to an instruction from the system controller 21 . In addition, the arm drive mechanism 54 moves the gripped product according to the instruction. Further, the arm drive mechanism 54 moves the first arm 14 according to the instruction to move the reading position of the arm scanner 16 .

Next, a configuration example of the control system of the interactive device 24 will be described.
Dialog device 24 functions as a user interface. The interactive device 24 provides information to the user by means of sounds, images, and the like. Further, the interactive device 24 inputs information from the user and outputs information as a response to the input information. In the configuration example shown in FIG. 2, the interactive device 24 has a processor 61, a memory 62, an interface (I/F) 63, a speaker 17, a microphone 18, a display 19, and the like.

The processor 61 implements various processes by executing programs. Processor 61 is, for example, a CPU. The memory 62 is composed of memories such as ROM, RAM, and rewritable non-volatile memory. An interface (I/F) 63 is an interface for communicating with the system controller 21 .

The processor 61 outputs sound from the speaker 17 according to instructions from the system controller 21 or the like. The processor 61 also supplies the audio information input to the microphone 18 to the system controller 21 . As a result, the processor 61 uses the speaker 17 and the microphone 18 to input and output voice in an interactive manner with the user. In addition, the processor 61 displays information on the display 19 according to instructions from the system controller 21 or the like.

Next, a configuration example of the control system of the communication device 25 will be described.
The communication device 25 is a communication interface for communicating with external devices. In this embodiment, the communication device 25 communicates with the settlement device 2 . In the configuration example shown in FIG. 2, the communication device 25 has a processor 71, a memory 72, an interface (I/F) 73, a communication section 74, and the like.

The processor 71 implements various processes by executing programs. Processor 71 is, for example, a CPU. The memory 72 is composed of memories such as ROM, RAM, and rewritable non-volatile memory. An interface (I/F) 73 is an interface for communicating with the system controller 21 . The communication unit 74 is an interface for communicating with an external device. In the configuration example shown in FIG. 2 , the communication unit 74 includes an interface for communicating with the settlement device 2 .

In addition, the robot 1 as a product reading system may be configured integrally with the checkout device 2 . In addition, the robot 1 may be equipped with a function of adjusting the price of the product. For example, the robot 1 may be provided with a settlement function using a credit card or electronic money. In this case, the robot 1 can not only read the product specific information, but also make payments with a credit card or electronic money based on the product specific information.

Next, operation of support processing (product reading processing) for product reading work by the robot 1 as the product reading apparatus according to the embodiment will be described.
3 and 4 are flowcharts for explaining an operation example of the robot 1 according to the embodiment.
In the standby state, the processor 41 of the product reading device 22 monitors the presence or absence of a person using the sensor 11 for detecting a person with the detection range near the register table 3 (ACT 11). When the sensor 11 detects a person (ACT 11, YES), the processor 41 shoots an image with the camera 12 set in the person detection shooting area near the cash register 3 (ACT 12). The photographing area for person detection may be any photographing area that can photograph a person detected by the sensor 11 or a person appearing in front of the cashier table 3 at the checkout of a product or the like.

For example, the processor 41 uses the camera driving mechanism 45 to set the area including the space in front of the cash register table 3 to the camera 12 as the photographing range for person detection. The processor 41 may also estimate the position of the person detected by the sensor 11 and control the orientation and position of the camera 12 according to the estimated position of the person. Note that the processor 41 stores the image captured by the camera 12 in the image memory 44 . Alternatively, the processor 41 may continuously capture a plurality of images with the camera 12 and store the captured images in the image memory 44 .

When the camera 12 captures an image near the register table 3, the processor 41 determines whether a person is present in the captured image captured by the camera 12 (ACT 13). Thereby, the processor 41 can detect a person (user) in the vicinity of the cash register table 3 from the captured image captured by the camera 12 . The processor 41 determines whether or not there is a user near the register table 3 based on the detection result of the person from the photographed image. If it cannot be determined that there is a user (ACT13, NO), the processor 41 returns to ACT11 and executes the above process again.

When determining that there is a user (ACT13, YES), the processor 41 notifies the controller 21 that the user has been detected. Upon receiving the notification that the user has been detected, the system controller 21 guides the user using the interactive device 24 (ACT 14). For example, the system controller 21 outputs a greeting voice such as “Welcome” from the speaker 17 by outputting a voice instruction to the interactive device 24 . Also, the system controller 21 may display the guidance on the display 19 together with the voice guidance.

When the user is detected, the processor 41 captures an image with the camera 12 in which the product detection image capturing region including the product placement position on the register table 3 is set as the image capturing region (ACT 15). The photographing area for product detection may include an area where the product to be settled (registered) placed on the cash register table 3 is placed. For example, the photographing area for product detection may be an area where the product can be placed on the cash register table 3 or an area including the arrangement position of the product provided on the cash register table 3 . Here, the processor 41 may store in the image memory 44 the image of the product detection imaging area captured by the camera 12 .

When the photographing area for product detection is photographed by the camera 12, the processor 41 determines whether the product is present in the image photographed by the camera 12 (photographed image for product detection) (ACT 16). Thereby, the commodity reading device 22 can detect the commodity placed on the register table 3 . Note that the processor 41 may use a method other than the method of detecting the product from the captured image of the camera 12 to determine the presence or absence of the product on the register table 3 .

If it is determined that there is no product on the register table 3 (ACT16, NO), the processor 41 returns to ACT15 and executes the above-described processing again. In other words, if the product cannot be detected, the processor 41 uses the interactive device 24 to provide voice or display guidance to prompt the user to place the product on the register table 3 . For example, the processor 41 may utter "Please place the product on the cash register" from the speaker 17 of the interactive device 24. FIG. Along with such guidance, the processor 41 repeatedly executes image capturing and product detection on the register table 3 .

If it is determined that there is a product on the cash register table 3 (ACT 16 , YES), the processor 41 determines whether the product on the cash register table 3 should be gripped by the arm device 23 . That is, the processor 41 identifies the shape of the product on the cash register table 3 (ACT 17). In this embodiment, the processor 41 specifies whether the shape of the product is bag-like, box-like, or cylindrical.

Here, the bag-shaped product is one that can be gripped by the arm device 23 as long as it is within a predetermined gripping size, and the shape of the exterior is deformed. As long as the size of the bag-like product is within a predetermined size, it may be of a shape that allows the arm device 23 to lift or rotate the product in a gripped state. A box-shaped product is a solid product having a shape that can be gripped by the arm device 23 as long as it is within a predetermined gripping size. The box-shaped product is not limited to a hexahedral shape, and any shape that can be lifted or rotated while being held by the arm device 23 can be used as long as it is within a predetermined size.

Whether or not a bag-shaped product or a box-shaped product can be gripped may be determined based on the weight of the product instead of the grip size. In this case, the weight of the product on the cash register table 3 may be measured by a weight sensor or the like provided on the cash register table 3 .

In addition, a cylindrical product is an item other than bag-like or box-like, and has a shape that cannot be (should not be) gripped by the arm device 23 . Cylindrical products are not limited to cylindrical products, and include products with shapes that cannot be gripped by the arm device 23, such as polygonal prisms, cones, and pyramids. For example, cylindrical products are assumed to be articles such as PET bottles, cans, bottles, and cups.

When determining that the shape of the product on the register table 3 is bag-like (ACT18, YES), the processor 41 determines whether the size of the product is within the size of the bag to be gripped (ACT19). In this embodiment, it is assumed that the arm device 23 cannot (should not) grip a bag-like product having a size exceeding the gripping size of the bag. It should be noted that whether or not a bag-shaped product can be gripped may be determined based on the weight of the product.

When determining that the product is bag-shaped, the processor 41 determines whether the product can be gripped (whether it can be gripped) based on the size of the product. The processor 41 identifies (estimates) the size of the product from the image area of the product in the image of the cash register table 3 (the photographed image for product detection). In the present embodiment, the size of the bag-like product is the length of the diagonal specified from the edge position of the product detected in the photographed image for product detection. However, the size of the bag-shaped product may be defined by the number of pixels of the image area of the product, an estimated area of the image area of the product, or the like.

For example, the processor 41 detects four points that are edges of a rectangular area that is the image area of the product in the photographed image for product detection. The processor 41 determines the length of the diagonal line of the detected four-point rectangle as the size of the product. In this case, as the size of the bag to be gripped, the maximum value of the size (diagonal length) of the product that can be lifted by pinching both ends with the two arms 14 and 15 is set. For example, if the gripping size of the bag is 500 mm, the processor 41 determines whether the length of the diagonal line specified from the edge of the product in the photographed image is within 500 mm.

When determining that the size of the product determined to be bag-like exceeds the grip size of the bag (ACT 19 , NO), the processor 41 determines that the arm device 23 cannot grip the product. If it is determined that the product cannot be gripped by the arm device 23, the processor 41 proceeds to ACT 24 and performs reading processing of the product specific information by the arm scanner 16. FIG.

If it is determined that the bag-shaped product is within the gripping size of the bag (ACT 19 , YES), the processor 41 instructs the system controller 21 to grip the product with the arm device 23 . The system controller 21 is instructed to grip the product with the arm device 23 . The processor 41 identifies position information indicating the position of the product and shape information indicating the shape of the product, such as shape, size, center, and direction, based on the photographed image of the register table 3 . The processor 41 notifies the system controller 21 of the position information and shape information of the product specified from the photographed image, together with the fact that the product is to be gripped.

The processor 31 of the system controller 21 performs gripping settings for gripping the product with the arm device 23 based on the information from the product reading device 22 (ACT 20). If the product is bag-shaped, the processor 31 performs gripping settings for the product based on an algorithm for gripping the bag-shaped product and the position information and shape information of the product. Here, it is assumed that the processor 31 sets the gripping setting for the arm device 23 so that the two arms 14 and 15 grip both ends of the bag-shaped product.

FIG. 5 is a diagram for explaining an operation example when the two arms 14 and 15 grip the bag-shaped product P1.
The arm device 23 moves the arms 14 and 15 so that the gripping mechanisms 14b and 15b provided at the ends of the arms 14a and 15a grip both ends of the bag-shaped product. The arms 14 and 15 are moved so that the gripping mechanisms 14b and 15b scrape the upper surface of the register table 3 parallel to the merchandise from the outside of both ends of the merchandise toward the center of the merchandise. In the arm device 23, after the gripping mechanism 14b (15b) of one arm 14 (15) grips one end of the product in the longitudinal direction, the gripping mechanism 15b (14b) of the other arm 15 (14) grips the product. Move so as to pinch one end on the opposite side in the longitudinal direction.

In the example shown in FIG. 5, after the first arm 14 grips one end of the product P1 with the gripping mechanism 14b, the second arm 15 moves the gripping mechanism 15b in the direction of arrow a. In the second arm 15, when the gripping position of the gripping mechanism 15b reaches the end of the product, the gripping mechanism 15b grips the end of the product. Thereby, the gripping mechanisms 14b and 15b can grip both ends of the product in the longitudinal direction.
The processor 31 of the system controller 21 performs gripping settings according to the position information and shape information of the product so that the arm device 23 can grip both ends of the product by the above-described operation.

When determining that the product on the cash register table 3 is box-shaped (ACT 21, YES), the processor 41 determines whether the size of the box-shaped product is within the gripping size of the box (ACT 22). ). In this embodiment, it is assumed that the arm device 23 cannot (should not) grip a box-shaped product having a width exceeding the gripping size of the box. That is, when determining that the product is box-shaped, the processor 41 determines whether or not the product can be gripped (whether it can be gripped) based on the width of the product.

For example, the processor 41 identifies (estimates) the width of the product from an image of the cash register table 3 (photographed image for product detection). Since the photographed image for product detection is an image of the register table 3 photographed from above, the box-shaped product placed on the register table 3 is photographed as a rectangular image in the photographed image for product detection. The processor 41 detects four points that are the edges of the rectangular area of the product in the photographed image for product detection. In addition, when a product determined to be box-shaped is not captured as a perfect rectangular image, the processor 41 may detect four points that are edges of a rectangular area circumscribing the image area of the product.

The processor 41 estimates the width of the product from the four edges of the rectangular area of the product detected from the photographed image for product detection. The width of the product is assumed to be the width of the portion gripped by the gripping mechanisms 14b and 15b. In this case, as the gripping size of the box, the maximum width of the product that can be gripped from the sides of the product by the gripping mechanisms 14b and 15b of the arms 14 and 15 is set. For example, if the gripping size of the box is 100 mm, the processor 41 determines whether the width of the box-shaped product detected from the photographed image for product detection is within 100 mm. It should be noted that whether or not a box-shaped product can be gripped may be determined based on the weight of the product.

When the processor 41 determines that the width of the box-shaped product on the cash register table 3 exceeds the gripping size of the box (ACT 22 , NO), it determines that the arm device 23 cannot grip the product. If it is determined that the product cannot be gripped, the processor 41 advances to ACT 24 and performs reading processing of product specific information using the arm scanner 16 .

When determining that the width of the product is within the gripping size of the box (ACT 22 , YES), the processor 41 instructs the system controller 21 to grip the product with the arm device 23 . The processor 41 identifies position information indicating the position of the product and shape information indicating the shape of the product, such as shape, size, center, and direction, based on the photographed image of the register table 3 . The processor 41 notifies the system controller 21 of the position information and shape information of the product, together with the fact that the product is to be gripped.

Based on the information from the product reading device 22, the processor 31 of the system controller 21 performs gripping settings for gripping the product with the arm device 23 (ACT 22). If the product is box-shaped, the processor 31 performs gripping settings for the product based on an algorithm for gripping a box-shaped product and position information and shape information of the product.

FIG. 6 is a diagram for explaining an example of a state in which a box-shaped product P2 is gripped by two arms 14 and 15. As shown in FIG.
In the example shown in FIG. 6, the arm device 23 grips both sides of a box-shaped product with gripping mechanisms 14b and 15b provided at the tip portions of the respective arm portions 14a and 15a. For example, the arm device 23 moves the tips of the arms 14a and 15a (grasping mechanisms 14b and 15b) toward the cash register table 3 from above in the direction perpendicular to the product to grip both sides of the product. The processor 31 performs gripping settings according to the position information and shape information of the product so that the arm device 23 can grip both sides of the product by the above-described operation.

When determining that the shape of the product on the cash register table 3 is cylindrical (ACT 21 , NO), the processor 41 determines that the shape of the product cannot be gripped by the arm device 23 . If it is determined that the product cannot be gripped by the arm device 23, the processor 41 proceeds to ACT 24 and performs reading processing of the product specific information by the arm scanner 16. FIG.

When the system controller 21 implements the product gripping settings, the processor 51 of the arm device 23 drives the arms 14 and 15 according to the gripping settings to grip the product (ACT 30).
For example, when gripping a bag-shaped product, the processor 51 moves the gripping mechanism 14b from the outside of the product toward the center of the product according to the grip setting to pick up one end of the product. After the gripping mechanism 14b picks one end of the product, the processor 51 moves the second arm 15 according to the gripping setting to pick up the opposite end of the product with the gripping mechanism 15b, as shown in FIG. As a result, the arm device 23 grips both ends of the product with the two arms 14 and 15 .

When gripping a box-shaped product, the processor 51 moves the tip portions (gripping mechanisms) of the arms 14 and 15 from above the cash register table 3 to both sides of the product based on the position designated by the grip setting. The processor 51 grips one end of the product with the arm 14 and grips the other end of the product with the arm 15 . Thereby, the arm device 23 grips both sides of the product with the two arms 14 and 15, respectively, as shown in FIG.

When the arms 14 and 15 grip the product, the processor 51 lifts the gripped product and moves it to the reading position of the scanner 13 according to instructions from the system controller 21 (ACT 31). When the lifted product is moved to the reading position of the scanner 13, the processor 51 rotates the product in a predetermined direction and angle at the reading position of the scanner 13 (ACT32). The processor 51 rotates the product gripped by the arms 14 and 15 according to the rotation procedure designated by the system controller 21 .

For example, the processor 51 rotates the item forward (downward toward the scanner 13) 90 degrees, and then rotates the item backward 180°. If the product is box-shaped, the system controller 21 may rotate the box in a lateral direction orthogonal to the forward and backward directions so that the scanner 13 can read the sides of the box.

While the product is rotated by the arm device 23, the processor 31 of the system controller 21 instructs the product reading device 22 to read the product identification information (barcode) of the product. In response to this, the processor 41 of the product reading device 22 uses the scanner 13 to read the barcode from the product rotated by the arm device 23 (ACT 33). In this embodiment, the product identification information read by the scanner 13 is assumed to be a bar code.

That is, the processor 41 executes the barcode reading process by the scanner 13 while the arm device 23 rotates the product. The scanner 13 reads the bar code from the exterior of the product rotated to the reading position by the arms 14,15. Thereby, the processor 41 determines whether or not the scanner 13 has read the bar code as the product identification information (ACT34).

If the barcode is read by the scanner 13 (ACT34, YES), the processor 41 outputs the product identification information read as the barcode as a read result (ACT43). For example, when a barcode is read from a product, processor 41 outputs information indicating the product specified by the barcode read by scanner 13 to settlement device 2 via communication device 25 .

If the barcode cannot be read by the scanner 13 (ACT34, NO), the processor 41 determines whether or not to retry the barcode reading process based on a predetermined condition (ACT35). For example, the processor 41 rotates the product by the arm device 23 described above twice and attempts to read the bar code.
When determining to retry the bar code reading process by the scanner 13 (ACT35, YES), the processor 41 returns to ACT32 and repeats the above-described process.

If it is determined not to retry (ACT 35, NO), the processor 41 executes the process of reading the product identification information (parcode) by the scanner 13 while the package of the product is stretched (reading process in the stretched state). (ACT 36). The state in which the product package is stretched is a state in which the two arms 14 and 15 gripping both ends of the product are pulled between gripping positions.

Enlarging the product package assumes that the barcode printed on the package cannot be read due to folds, wrinkles, or the like in the product package. In other words, products whose barcodes can be read by pulling both ends of the package (between the gripping positions of the two arms) are subject to stretching. In the present embodiment, it is assumed that barcode reading processing is executed in a stretched state for bag-shaped products. In this case, the processor 41 determines whether or not to perform the reading process for the product in the stretched state based on whether the product is bag-shaped or not.

If the processor 41 determines to execute the reading process in the enlarged state (ACT36, YES), the processor 41 instructs the system controller 21 to execute the reading process in the enlarged state. Upon receiving the instruction to read the stretched state, the processor 31 of the system controller 21 sends an instruction to stretch the product package to the arm device 23 (ACT 37).

The arm device 23 operates according to an instruction from the system controller 21 so as to mutually pull both ends of the product gripped by the arms 14 and 15 . Here, the gripping mechanism 14b of the first arm 14 and the gripping mechanism 15b of the second arm 15 respectively grip both ends of the product. Therefore, the processor 51 controls the gripping mechanism 14b (gripping position of the first arm 14) and the gripping mechanism 15b (gripping position of the second arm 15) to move in mutually opposite directions with constant power. That is, the processor 51 causes the arm drive mechanism 54 to operate the arm portions 14a and 15a so as to pull the arms 14 and 15 between the gripping positions with a constant force. When the arms 14 and 15 are pulled between the gripping positions with a constant force, the arms 14 and 15 grip both ends of the product while the package of the product is stretched.

When the product is pulled between the gripping positions, the processor 51 rotates the gripped product to the reading position of the scanner 13 while the product is pulled between the gripping positions (the package is stretched) (ACT38). For example, the processor 51 of the arm device 23 lifts the product to the reading position of the scanner 13 while pulling the product between the gripping positions. After lifting the product to the reading position of the scanner 13, the processor 51 rotates the product at the reading position of the scanner 13 in a predetermined direction and angle. The rotation procedure for rotating the product to be gripped while being pulled by the arms 14 and 15 may be the same procedure as in ACT 32 described above.

While the arm device 23 rotates the stretched product, the processor 31 of the system controller 21 instructs the product reading device 22 to read the bar code (product specific information) of the product. In response to this, the processor 41 of the product reading device 22 uses the scanner 13 to read the bar code as product identification information from the product rotated by the arm device 23 (ACT 39).

That is, the processor 41 executes the bar code reading process by the scanner 13 while the arm device 23 rotates the stretched product. The scanner 13 reads the bar code from the exterior of the product rotated to the reading position by the arms 14,15. Thereby, the processor 41 determines whether or not the scanner 13 has read the bar code as the product identification information (ACT 40).

If the barcode is read by the scanner 13 (ACT 40, YES), the processor 41 outputs the product identification information read as the barcode as a read result (ACT 43). For example, when a barcode is read from a product, processor 41 outputs information indicating the product specified by the barcode read by scanner 13 to settlement device 2 via communication device 25 .

If the scanner 13 fails to read the product identification information (ACT 40, NO), the processor 41 determines whether or not to retry the reading process in the stretched state based on a predetermined condition (ACT 41). If it is determined to retry the process of reading the bar code in the stretched state (ACT35, YES), the processor 41 returns to ACT38 and repeats the above-described process.

If it is determined not to retry (ACT41, NO), the processor 41 executes product recognition processing (reading processing of product identification information) using the image captured by the camera 12 (ACT42). Product recognition processing based on images captured by the camera 12 assumes products whose barcodes cannot be read due to dirt or the like to be processed. In product recognition processing, the processor 41 captures an image of the product gripped by the arm device 23 with the camera 12 and recognizes the product from the captured image captured by the camera 12 . The product recognition processing will be described later in detail.

The processor 41 outputs the recognition result of the product recognition process as the reading result of the product specifying information (ACT43). For example, if the product cannot be recognized in the product recognition process, the processor 41 outputs to the settlement device 2 via the communication device 25 that the product specifying information could not be read. When the product is recognized by the product recognition process, the processor 41 outputs information indicating the recognized product to the settlement device 2 via the communication device 25 .

Next, barcode reading processing by the arm scanner 16 will be described.
FIG. 7 is a flowchart for explaining an operation example of barcode reading processing by the arm scanner 16 .
According to the operation example described above, the present system targets products that are not bag-shaped or box-shaped, or products that exceed a specified size for reading processing by the arm scanner 16 . The arm scanner 16 is provided at the tip of the arm 14, and the reading position moves as the arm 14 moves. In the reading process by the arm scanner 16 , the product surface is scanned by the arm scanner 16 by moving the tip of the arm 14 along the product placed on the register table 3 .

When the processor 41 of the product reading device 22 determines that the product is to be read by the arm scanner 16 , it requests the system controller 21 to read the product by the arm scanner 16 . The processor 41 also supplies the system controller 21 with various types of information (product shape, product center position, product edge position, etc.) specified from the image for product detection captured by the camera 12 . When executing reading processing by the arm scanner 16, the processor 31 of the system controller 21 confirms the shape of the product.

When determining that the product is bag-shaped (ACT61, YES), the processor 31 of the system controller 21 instructs the arm device 23 to move the arm scanner 16 so as to scan the upper surface of the product. For example, the processor 31 determines the position (moving path) to be scanned by the arm scanner 16 based on the edge position of the product.

For example, the processor 31 sets the first path for moving the arm scanner 16 in a square shape so as to connect the four edges presumed to be the four corners of the product. Further, the processor 31 sets a second path along which the arm scanner 16 is moved along a diagonal line consisting of four edges if the bar code cannot be read on the first path. As a result, the arm scanner 16 can scan the entire top surface of the product placed on the cash register table 3 . Further, by setting the first path and the second path, the arm scanner 16 can first scan the vicinity of the periphery of the product on the first path and then scan the entire product on the second path.

The processor 51 of the arm device 23 moves the arm scanner 16 according to the instruction from the system controller 21 to scan the upper surface of the bag-shaped product (ACT62). The processor 51 drives the arm 14 so that the arm scanner 16 moves along the path instructed by the system controller 21 by the arm drive mechanism 54 . The arm scanner 16 scans the bar code from the upper surface of the bag-like product placed on the register table 3 while moving the scanning position by the arm 14 .

The processor 51 of the arm device 23 supplies the result of reading by the arm scanner 16 to the system controller 21 . The processor 31 of the system controller 21 outputs the reading result of the bar code from the arm device 23 to the settlement device 2 via the communication device 25 as the reading result of the commodity specifying information.

If the product is box-shaped (ACT63, YES), the processor 31 of the system controller 21 instructs the arm device 23 to move the arm scanner 16 so as to scan the upper surface of the product. For example, the processor 31 determines the position (moving path) to be scanned by the arm scanner 16 based on the edge position of the product. For example, the processor 31 sets the square-shaped first route connecting the four edges presumed to be the four corners of the product and the second route along the diagonals of the four edges, as in the above example.

The processor 51 of the arm device 23 moves the arm scanner 16 according to the instruction from the system controller 21 to scan the upper surface of the box-shaped product (ACT64). The processor 51 drives the arm 14 so that the arm scanner 16 moves along the path instructed by the system controller 21 by the arm drive mechanism 54 . The arm scanner 16 scans the bar code from the upper surface of the box-shaped product placed on the register table 3 while moving the scanning position by the arm 14 .

The processor 51 of the arm device 23 supplies the result of reading the bar code by the arm scanner 16 to the system controller 21 . The processor 31 of the system controller 21 determines whether or not the arm scanner 16 has read the bar code from the upper surface of the product based on the reading result from the arm device 23 (ACT65). If the barcode is read from the upper surface of the product (ACT65, NO), the processor 31 outputs the barcode reading result to the settlement device 2 via the communication device 25 .

If the barcode cannot be read from the upper surface of the product (ACT65, YES), the processor 31 instructs the arm device 23 to scan the side surface of the product with the arm scanner 16 . For example, processor 31 sets a path for arm scanner 16 to scan the sides of the item based on the edge position of the item.

The processor 51 of the arm device 23 moves the arm scanner 16 according to the instruction from the system controller 21 to scan the sides of the box-shaped product (ACT66). The processor 51 moves the arm scanner 16 along the side of the product on the register table 3 by the arm drive mechanism 54 and scans the bar code from the side of the product. The processor 31 of the system controller 21 outputs the reading result of the bar code from the arm device 23 to the settlement device 2 via the communication device 25 as the reading result of the commodity specifying information.

When it is determined that the product is cylindrical (neither bag-like nor box-like) (ACT 63, NO), the processor 31 of the system controller 21 determines that the height of the product on the register table 3 is a predetermined height (for example, 100 mm). ) is determined (ACT70). Here, the height of the product is the height from the upper surface of the register table 3 (the surface on which the product is placed) to the top of the product placed on the register table 3 . The height of the product is estimated, for example, from the captured image captured by the camera 12 . Also, a height detection sensor for detecting the height of the product may be provided. The predetermined height is a threshold for determining whether the arm 14 can be moved above the product. In other words, the arm 14 can move freely at a position lower than a predetermined height, and cannot move freely at a height higher than the predetermined height.

If the height of the product is within the predetermined height (ACT70, YES), the processor 31 moves the arm 14 above the product to read the bar code from the peripheral surface of the product with the arm scanner 16 (ACT71). That is, for products whose cylindrical height is within a predetermined height, the arm portion 14a is moved in a circle above the product, and the arm scanner 16 reads the peripheral surface of the product.

FIG. 8 is a diagram for explaining an operation example of the arm scanner 16 when the height of the cylindrical product P3 is within a predetermined height.
In this embodiment, the arm 14 can freely move at a low position within a predetermined height. Therefore, if the height of the product is within a predetermined height, the arm 14 can freely move above the product. As a result, the arm portion 14a is moved above the product so that the arm scanner 16 provided at the tip of the arm portion 14a extending from above the product can freely move along the peripheral surface of the product. In the example shown in FIG. 8, the arm portion 14a is moved as indicated by an arrow b1 in a space including above the product. The arm scanner 16 provided at the tip of the arm portion 14a moves along the peripheral surface of the product as the arm portion 14a moves in the space including above the product, as indicated by the arrow b2.

The processor 31 of the system controller 21 instructs the arm device 23 to scan the barcode when the arm scanner 16 is moved by a predetermined angle (for example, 360 degrees) on the peripheral surface of the product. However, the bar code on the peripheral surface of the product facing the scanner 13 may be read by the scanner 13 . In this case, the arm scanner 16 may move about 270 degrees along the peripheral surface of the product.

The processor 51 of the arm device 23 drives the arm 14 by the arm driving mechanism 54 so that the arm scanner 16 moves along the peripheral surface of the product according to the instruction from the system controller 21 . Thereby, the arm scanner 16 scans the bar code printed on the peripheral surface of the product while moving along the peripheral surface of the product placed on the cash register table 3 .

The processor 51 of the arm device 23 supplies the result of reading the bar code by the arm scanner 16 to the system controller 21 . The processor 31 of the system controller 21 outputs the reading result of the bar code from the arm device 23 to the settlement device 2 via the communication device 25 as product identification information of the product.

If the height of the product exceeds the predetermined height (ACT70, YES), the processor 31 causes the arm scanner 16 to read the bar code from the peripheral surface of the product by moving the arm 14 to the side of the product. It is difficult to rotate the arm scanner 16 360 degrees when moving the arm 14 to the side of the product. Therefore, the processor 31 divides the circumferential surface of the product into several areas and scans the barcode. In the present embodiment, an operation example will be described in which a product is divided into an upper portion and a lower portion, and the lower portion and the upper portion are each scanned by half the circumference.

FIG. 9 is a diagram for explaining an operation example of the arm scanner 16 when the height of the cylindrical product P4 exceeds a predetermined height.
In the example shown in FIG. 9, the arm scanner 16 provided at the tip of the arm 14 scans the lower peripheral surface and the upper peripheral surface of a cylindrical product P4 by half the circumference. For example, the arm scanner 16 scans the bar code clockwise from the initial position at the bottom of the article P4, as indicated by an arrow c1. If the barcode cannot be read, the arm scanner 16 moves to the initial position above the product P4 and rotates the peripheral surface of the product clockwise halfway to scan the barcode, as indicated by an arrow c2.

If the barcode is still unreadable, the arm scanner 16 moves again to the initial position under the product P4, rotates the peripheral surface of the product P4 halfway counterclockwise (dotted arrow c3), and scans the barcode. to scan. If the barcode is still unreadable, the arm scanner 16 moves again to the initial position above the product P4 and rotates counterclockwise halfway around the peripheral surface of the product P4 to scan the barcode. By these operations, the arm 14 moves the arm scanner 16 from the side of the product P4, thereby scanning the entire circumferential surface of the product P4 with the arm scanner 16.

First, the processor 31 instructs the arm device 23 to scan the bar code by moving the arm scanner 16 clockwise halfway along the circumference of the product from the initial position under the product. The processor 51 of the arm device 23 operates the arm 14 to move the arm scanner 16 to the initial position below the product. When the arm scanner 16 is moved to the initial position under the product, the processor 51 moves the arm 14 to the side of the product to move the arm scanner 16 half clockwise along the peripheral surface of the product ( ACT 72). Thereby, the arm scanner 16 scans the bar code on the lower half of the peripheral surface of the product.

If the barcode cannot be read from the lower half circumference of the product (ACT73, YES), the processor 31 instructs the arm device 23 to scan the upper half circumference of the product clockwise. The processor 51 of the arm device 23 operates the arm 14 according to the instruction from the system controller 21 to move the arm scanner 16 to the initial position above the product. When the arm scanner 16 moves to the initial position above the product, the processor 51 moves the arm scanner 16 along the peripheral surface of the product by moving the arm 14 to the side of the product by a half clockwise rotation ( ACT 74). Thereby, the arm scanner 16 scans the bar code on the upper half circumference of the product.

If the barcode cannot be read from the upper half circumference of the product (ACT75, YES), the processor 31 instructs the arm device 23 to scan the remaining lower half circumference of the product. The processor 51 of the arm device 23 operates the arm 14 according to an instruction from the system controller 21 to move the arm scanner 16 to the initial position below the product. When the arm scanner 16 moves to the initial position under the product, the processor 51 moves the arm scanner 16 along the peripheral surface of the product counterclockwise halfway by operating the arm 14 on the side of the product. (ACT 76). Thereby, the arm scanner 16 scans the bar code on the remaining half circumference of the lower part of the product.

If the barcode cannot be read from the remaining half of the circumference of the lower part of the product (ACT77, YES), the processor 31 instructs the arm device 23 to scan the remaining half of the upper part of the product. . The processor 51 of the arm device 23 operates the arm 14 according to the instruction from the system controller 21 to move the arm scanner 16 to the initial position above the product. When the arm scanner 16 moves to the initial position above the product, the processor 51 moves the arm 14 to the side of the product, thereby moving the arm scanner 16 counterclockwise halfway around the product. (ACT78). Thereby, the arm scanner 16 scans the bar code on the remaining half circumference of the upper part of the product.

The processor 31 of the system controller 21 acquires the reading result of the bar code by the arm scanner 16 from the arm device 23 operated as described above. The processor 31 of the system controller 21 outputs the reading result of the bar code from the arm device 23 to the settlement device 2 via the communication device 25 as product identification information of the product.

Next, description will be given of product recognition processing for a captured image captured by the camera 12. FIG.
FIG. 10 is a flow chart for explaining an operation example of product recognition processing for a captured image captured by the camera 12 .
In the operation examples shown in FIGS. 3 and 4 described above, the system controller 21 executes product recognition processing based on the image captured by the camera 12 when the barcode cannot be read by the scanner 13 . The system controller 21 instructs the arm device 23 to move the product to the photographing position of the camera 12 when product recognition processing is performed using the image captured by the camera 12 . The processor 51 of the arm device 23 moves the product gripped by the arms 14 and 15 to the photographing position of the camera 12 according to the instruction from the system controller 21 (ACT81).

When the product moves to the photographing position of the camera 12, the system controller 21 rotates the product gripped by the arms 14 and 15 in order to photograph the image containing the product specific information. It is assumed that the system controller 21 detects the image of the product identification information from the image captured by the camera 12 of the product reading device 22 while rotating the product with the arm device 23 .

In this embodiment, the product is rotated by a predetermined angle (for example, 15 degrees) in the front-rear direction (forward (front) and backward) toward the camera 12, and the image of the product-specific information is obtained from the image captured by the camera 12. shall be detected. In this embodiment, assuming that the barcode cannot be read by the scanner 13 due to dirt or the like, the commodity is recognized by the number string appended below the barcode in the barcode area. Therefore, in the following processing, the product reading device detects the image of the barcode area (barcode image) as the image of the product identification information, and recognizes the product from the number string in the barcode image.

That is, the system controller 21 determines the angle to rotate the product gripped by the arms 14 and 15, and instructs the arm device 23 to rotate the product at the determined angle. The arm device 23 rotates the product gripped by the arms 14 and 15 according to the instruction from the system controller 21 (ACT82). For example, the system controller 21 rotates the product forward (front) by a predetermined angle (for example, 15 degrees). When the forward rotation angle reaches 90 degrees, the system controller 21 rotates the product backward by a predetermined angle (for example, 15 degrees) until the backward rotation angle reaches 180 degrees.

When the arm device 23 rotates the product, the system controller 21 instructs the product reading device 22 to detect an image (barcode image) of product specifying information from the image captured by the camera 12 . The processor 41 of the commodity reading device 22 takes an image with the camera 12 according to the instruction from the system controller 21 (ACT83). When the camera 12 takes an image, the processor 41 of the commodity reading device 22 detects the barcode image from the image taken by the camera 12 (ACT84). The processor 41 of the product reading device 22 notifies the system controller 21 of information indicating whether or not the barcode image has been detected.

If the barcode image is not detected from the image captured by the camera 12 (ACT85, NO), the system controller 21 determines whether to further rotate the product and detect the barcode image (ACT86). If it is determined to further rotate the product and detect the barcode image (ACT86, NO), the system controller 21 returns to ACT82 and executes the above-described processing again. If the system controller 21 determines to end the detection of the barcode image (ACT86, YES), the system controller 21 assumes that the product recognition based on the image captured by the camera 12 has failed, and ends the process.

Also, if the barcode image can be detected from the image captured by the camera 12 (ACT85, NO), the system controller 21 adjusts the orientation of the product and captures an image for product recognition processing with the camera 12 . For example, the system controller 21 acquires a photographed image photographed by the camera 12 in a state in which the product is rotated (tilted) by a predetermined angle (for example, 5 degrees) in the horizontal direction toward the camera 12 . In this case, the system controller 21 rotates the product gripped by the arms 14 and 15 clockwise by 5 degrees and causes the camera 12 to take an image. When the rotation angle to the right reaches 45 degrees, the system controller 21 returns the product to its original angle, rotates the product to the left by 5 degrees, and causes the camera 12 to take an image. When the rotation angle to the left reaches 45 degrees, the system controller 21 causes the product reading device 22 to perform product recognition based on the photographed images photographed at each angle.

That is, the system controller 21 determines the orientation of the product gripped by the arms 14 and 15 and instructs the arm device 23 of the determined orientation of the product. The arm device 23 adjusts the orientation of the product gripped by the arms 14 and 15 according to the instruction from the system controller 21 (ACT87). For example, the system controller 21 instructs the arm device 23 to rotate the product 5 degrees to the right until the rotation angle to the right reaches 45 degrees. Further, when the rotation angle to the right reaches 45 degrees, the system controller 21 instructs the arm device 23 to rotate the product to the left by 5 degrees after returning the product to its original angle.

When the arm device 23 adjusts the orientation of the product, the system controller 21 instructs the product reading device 22 to capture an image for product recognition. The product reading device 22 takes an image with the camera 12 according to the instruction from the system controller 21 (ACT88). The product reading device 22 stores the captured image captured by the camera 12 in the image memory 44 as an image for product recognition (ACT89).

When the image captured by the camera 12 is saved as the product recognition image, the system controller 21 determines whether or not to end the capturing of the product recognition image by the camera 12 (ACT90). If the system controller 21 determines not to finish photographing with the camera 12 (ACT 90, NO), the system controller 21 returns to ACT 87 and executes the above-described processing again.

When it is determined that the photographing by the camera 12 is finished (ACT 90, NO), the system controller 21 instructs the product reading device 22 to recognize the product by image processing. The processor 41 of the product reading device 22 executes product recognition processing based on the image captured by the camera 12 (image for product recognition processing) stored in the image memory 44 (ACT91). The processor 41 of the product reading device 22 recognizes numbers under the barcode based on a plurality of images captured by the camera 12 by executing a character recognition program.

For example, the processor 41 extracts the barcode image from each captured image and detects the numbers under the barcode from the barcode image. When all the numbers under the bar code are detected, the processor 41 recognizes the detected numbers, and uses numerical information in which the recognized numbers are arranged as the product recognition result. The processor 41 notifies the system controller 21 of the product recognition result. The system controller 21 transmits the product recognition result from the product recognition device to the settlement device 2 via the communication device 25 . The settlement device 2 identifies the product from the numerical information under the barcode, and executes settlement processing for the product.

As described above, the product reading system according to this embodiment includes a camera, an arm device, a scanner for reading barcodes, a processor, a controller, and the like. The camera shoots the product placed on the cash register table. The processor identifies the shape of the product from the image taken by the camera. The controller performs gripping settings for the arm device to grip the product according to the shape of the product. The arm device grips the item according to the grip setting, and rotates the gripped item at the reading position of the scanner. The scanner reads a bar code as product identification information from the product rotated at the reading position by the arm device. As a result, the product reading system according to the present embodiment can reliably grip the product with the grip setting according to the shape, and can rotate the product so that the barcode can be easily read at the reading position of the scanner. .

In addition, when the product is bag-shaped, the controller of the product reading system according to the embodiment picks up the first end of the product with the first arm and sandwiches the product with the first end with the second arm. Pinch the second end on the opposite side. As a result, the commodity reading system can reliably pick up both ends of the bag-shaped commodity and move the bar code of the bag-shaped commodity so that the scanner can easily read it.

Further, when the product is box-shaped, the controller of the product reading system according to the embodiment grips the product so that both ends of the product are sandwiched between the gripping mechanism of the first arm and the gripping mechanism of the second arm. As a result, the product reading system can reliably hold both ends of the box-shaped product, and move the bar code of the box-shaped product so that the scanner can easily read it.

Further, in the product reading system according to the embodiment, when the product identification information of the product cannot be read by the scanner, the controller rotates the product gripped by the arm device within the imaging area of the camera and causes the camera to capture an image. The processor recognizes the product identification information by character recognition on the image captured by the camera. As a result, the product reading system can specify the product identification information by recognizing images captured by the camera while moving the product gripped by the arm, even for products whose barcode cannot be read by the scanner.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.
The content described in the scope of claims as originally filed for this application will be added below.
[1]
A camera that shoots the table on which the product is placed from above,
an arm device for gripping the product placed on the table;
a scanner for reading product-specific information present on the exterior of the product at a reading position;
The product is gripped by the arm device with a grip setting corresponding to the shape of the product detected from the image captured by the camera, and the product gripped by the arm device is rotated at a reading position of the scanner to cause the scanner to rotate. a controller that causes the to read product-specific information;
A product reading system having a
[2]
The arm device has a first arm and a second arm each provided with a gripping mechanism, and the controller controls the gripping mechanism provided on the first arm when the product is bag-shaped. a gripping setting for gripping a first end of the product, and gripping a second end of the product on the table opposite to the first end with a gripping mechanism provided on the second arm; do,
The product reading system according to [1].
[3]
When the shape of the product is box-shaped, the controller sets a gripping setting such that the gripping mechanism provided on the first arm and the gripping mechanism provided on the second arm respectively grip both ends of the product.
The product reading system according to [2].
[4]
a processor that recognizes product-specific information by image recognition of an image captured by the camera;
When the scanner cannot read the product identification information of the product, the controller rotates the product gripped by the arm device in the photographing area of the camera, and transmits the product to the processor based on the image photographed by the camera. to recognize the product specific information of
The commodity reading system according to any one of [1] to [3].
[5]
a communication device that outputs product identification information of the product gripped by the arm device to a checkout device that performs checkout processing for the product;
The product reading system according to any one of [1] to [4].

DESCRIPTION OF SYMBOLS 1... Commodity reading system (robot), 2... Settlement apparatus (cash register), 3... Cash register stand (stand), 11... Sensor, 12... Camera, 13... Scanner (fixed scanner), 14... First arm, 15... Second 2 arms, 14b, 15b... grasping mechanism, 16... arm scanner, 21... system controller (control device), 22... product reader (product reader), 23... arm device, 25... communication device, 31, 41, 51, 61, 71... Processors.

Claims (5)

A camera that shoots the table on which the product is placed from above,
an arm device for gripping the product placed on the table;
a scanner for reading product-specific information present on the exterior of the product at a reading position;
The arm device is caused to grip the product with a grip setting using a different algorithm according to the shape of the product detected from the image captured by the camera, and the product gripped by the arm device is positioned at the reading position of the scanner. a controller that rotates to cause the scanner to read product identification information;
A product reading system having a The arm device has a first arm and a second arm each provided with a gripping mechanism,
When the product has a bag shape, the controller moves the gripping mechanism provided on the first arm and the gripping mechanism provided on the second arm along the placement surface of the product on the table, The gripping mechanism provided on the first arm grips the first end of the product, and the gripping mechanism provided on the second arm grips the product on the side opposite to the first end across the product on the table. With a gripping setting that grips a certain second end,
The commodity reading system according to claim 1. When the shape of the product is box-shaped, the controller sets the gripping setting such that the gripping mechanism provided on the first arm and the gripping mechanism provided on the second arm respectively grip both ends of the product from above . ,
The commodity reading system according to claim 2. a processor that recognizes product-specific information by image recognition of an image captured by the camera;
When the scanner cannot read the product identification information of the product, the controller rotates the product gripped by the arm device in the photographing area of the camera, and transmits the product to the processor based on the image photographed by the camera. to recognize the product specific information of
The product reading system according to any one of claims 1 to 3. a communication device that outputs product identification information of the product gripped by the arm device to a checkout device that performs checkout processing for the product;
The product reading system according to any one of claims 1 to 4.

Images