JP2021099629A - Information processing system, information processing device, and information processing method - Google Patents

Abstract

To provide an information processing system, an information processing device, and an information processing method capable of specifying what commodity an object is with high accuracy.SOLUTION: An information processing system 10 includes an imaging part 16 for generating an image signal by imaging, and an information processing device 18. The information processing device acquires a picked-up image corresponding to the image signal, and specifies an object corresponding to a partial image included in at least one of two picked-up images on the basis of a difference image between the two picked-up images having different imaging times.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to an information processing system, an information processing device, and an information processing method.

Conventionally, in order to settle the price of a product, an information processing system that identifies which product the object is has been required. For example, there is known an information processing system that identifies which product an object is based on a plurality of images obtained by capturing one or more objects placed in a predetermined area from a plurality of directions (Patent Document 1). ).

JP-A-2017-220198

There is room for improvement in conventional information processing systems.

An object of the present disclosure made in view of this point is to provide an improved information processing system, information processing apparatus and information processing method.

The information processing system according to the embodiment of the present disclosure includes an imaging unit that generates an image signal by imaging and an information processing device. The information processing device acquires a captured image corresponding to the image signal, and corresponds to a partial image included in at least one of the two captured images based on a difference image of the two captured images having different imaging times. Identify the object to be used.

The information processing device according to the embodiment of the present disclosure includes a communication unit and a control unit. The communication unit can communicate with an imaging unit that generates an image signal by imaging. The control unit acquires a captured image corresponding to the image signal by the communication unit, and is included in at least one of the two captured images based on the difference image of the two captured images having different imaging times. Identify the object that corresponds to the image.

The information processing method according to the embodiment of the present disclosure includes generating an image signal by imaging. The information processing method acquires a captured image corresponding to the image signal, and corresponds to a partial image included in at least one of the two captured images based on a difference image of the two captured images having different imaging times. Includes identifying the object to be.

According to one embodiment of the present disclosure, an improved information processing system, information processing apparatus and information processing method may be provided.

It is a figure which shows the whole structure of the settlement system which concerns on one Embodiment of this disclosure. It is a figure which shows the whole structure of the information processing system shown in FIG. It is a functional block diagram of the information processing system shown in FIG. It is a figure which shows an example of the captured image which concerns on one Embodiment of this disclosure. It is a figure which shows an example of the captured image which concerns on one Embodiment of this disclosure. It is a figure which shows an example of the captured image which concerns on one Embodiment of this disclosure. It is a figure which shows an example of the difference image which concerns on one Embodiment of this disclosure. It is a figure which shows an example of the difference image which concerns on one Embodiment of this disclosure. It is a flowchart which shows the object identification process of the information processing system which concerns on one Embodiment of this disclosure. It is a figure which shows an example of the captured image which concerns on other embodiment of this disclosure. It is a figure which shows an example of the captured image which concerns on other embodiment of this disclosure. It is a flowchart which shows the object identification process of the information processing system which concerns on other embodiment of this disclosure. It is a figure which shows the whole structure of the information processing system which concerns on still another Embodiment of this disclosure. It is a flowchart which shows the object identification process of the information processing system which concerns on still another Embodiment of this disclosure. It is a figure which shows an example of the difference image which concerns on still another Embodiment of this disclosure. It is a flowchart which shows the object recognition with respect to the difference image of the information processing system which concerns on still another Embodiment of this disclosure. It is a figure which shows the whole structure of the information processing system which concerns on still another Embodiment of this disclosure.

Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings.

As shown in FIG. 1, the settlement system 11 includes at least one information processing system 10 and a server 12. The server 12 may be a physical server or a cloud server. The information processing system 10 and the server 12 can communicate with each other via the network 13. The network 13 may be wireless, wired, or a combination of wireless and wired.

The information processing system 10 may be arranged in any store such as a convenience store and a supermarket. The information processing system 10 may be configured as a cash register terminal of a store. The information processing system 10 generates an captured image by capturing an image of a product placed on a mounting table 14 described later by a customer. The information processing system 10 identifies which product in the store the object in the generated captured image is. In the present disclosure, the "object in the captured image" means an image of an object drawn as an image in the captured image. The information processing system 10 transmits the specified product information to the server 12 via the network 13. The product information may include the product name, the number of products, and the like. The server 12 calculates the amount billed to the customer based on the product information. The server 12 transmits the billed amount to the information processing system 10 via the network 13. The information processing system 10 requests the customer to pay the billed amount by presenting the billed amount to the customer.

As shown in FIG. 2, the information processing system 10 includes a camera 16 (imaging unit) and an information processing device 18. The information processing system 10 may include a mounting table 14, a support column 15, and a display device 17.

The mounting table 14 includes a mounting surface 14a. At the time of checkout, the customer places the product he / she wants to purchase on the mounting surface 14a. In the present embodiment, the mounting surface 14a has a substantially rectangular shape. However, the mounting surface 14a may have any shape.

The support pillar 15 extends from the side portion of the mounting table 14 toward one of the mounting surfaces 14a in the normal direction. One of the normal directions may be a direction away from the mounting surface 14a among the normal directions of the mounting surface 14a. The camera 16 is located at the tip of the support column 15.

The camera 16 generates an image signal corresponding to the captured image by imaging. In the present embodiment, the camera 16 images the mounting surface 14a of the mounting table 14. However, the range imaged by the camera 16 may be at least a part of the mounting surface 14a, and may be a range exceeding the mounting surface 14a. Further, the range of imaging by the camera 16 is not limited to the mounting surface 14a of the mounting table 14. For example, the range imaged by the camera 16 may be in the shopping cart 214 as shown in FIG. 17 described later.

The camera 16 may be arranged so that the mounting surface 14a of the mounting table 14 can be imaged. The camera 16 may be arranged so that the optical axis is perpendicular to the mounting surface 14a. The camera 16 may be arranged at the tip of the support column 15. The camera 16 may be fixed to, for example, the tip of the support column 15 so that the mounting surface 14a of the mounting table 14 can be imaged.

The camera 16 can acquire a signal instructing the start of imaging from the information processing device 18. The camera 16 may start imaging when it acquires a signal instructing the start of imaging. The camera 16 outputs the image signal generated by the imaging to the information processing device 18 via the communication line. The camera 16 continuously performs imaging at an arbitrary frame rate. The camera 16 can generate image signals having different imaging times by continuously performing imaging. However, the camera 16 may capture a still image at any time based on a predetermined trigger.

The display device 17 may be configured to include an arbitrary display such as a liquid crystal display. The display device 17 displays an image based on the signal acquired from the information processing device 18. For example, the display device 17 may acquire a signal indicating the billing amount of the product from the information processing device 18. The display device 17 may display an image showing the billed amount based on the signal indicating the billed amount of the product.

The display device 17 may have a touch screen function. The display device 17 may be arranged near the mounting table 14. The display device 17 may be arranged so as to face the customer when the customer places the product on the mounting table 14.

The information processing device 18 may be arranged in the vicinity of the mounting table 14. As shown in FIG. 3, the information processing device 18 includes a control unit 23. The information processing device 18 may include a communication unit 20, an input unit 21, and a storage unit 22.

The communication unit 20 includes an arbitrary communication module. The arbitrary communication module includes a communication module capable of communicating with the camera 16 via a communication line, a module capable of communicating with a display device 17 via a communication line, and a communication module capable of communicating with a server 12 via a network 13. May include. The communication line between the camera 16 and the communication unit 20 and the communication line between the display device 17 and the communication unit 20 may be configured to include wired or wireless.

The input unit 21 includes one or more interfaces capable of detecting the user input. For example, the input unit 21 may include at least one of a physical key, a capacitance key, and a touch screen. In the present embodiment, the input unit 21 is a touch screen. The input unit 21, which is a touch screen, is provided integrally with the display device 17.

The storage unit 22 includes an arbitrary storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage unit 22 stores various programs that make the control unit 23 function and various information used by the control unit 23.

The storage unit 22 may store the product data of the store. The product data can be used for object detection, object recognition, etc., which will be described later. The product data may include a product image, characters attached to the product package, and the like associated with an identifier that identifies the product. Hereinafter, the identifier that identifies the product shall be the product name for convenience.

The control unit 23 includes one or more processors and a memory. The processor may include a general-purpose processor that loads a specific program and executes a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The control unit 23 may be either a SoC (System-on-a-Chip) in which one or a plurality of processors cooperate, or a SiP (System In a Package).

The control unit 23 may acquire the above-mentioned product data from the server 12 via the network 13 by the communication unit 20. The control unit 23 may store the acquired product data in the storage unit 22.

The control unit 23 may transmit a signal instructing the start of imaging to the camera 16 by the communication unit 20. The control unit 23 may transmit a signal instructing the start of imaging when the information processing device 18 is activated. When the input unit 21 detects the user input instructing the start of imaging, the control unit 23 may transmit a signal instructing the start of imaging. The user input can be input by the clerk from the input unit 21 at the time of opening the store or the like.

The control unit 23 acquires an image signal from the camera 16 by the communication unit 20. By acquiring the image signal, the control unit 23 can acquire the captured image corresponding to the image signal. The control unit 23 may store the acquired captured image in the storage unit 22. By generating image signals having different imaging times by the camera 16, the control unit 23 can acquire captured images having different imaging times. Here, when the customer purchases a plurality of products, the customers can place the products on the mounting surface 14a of the mounting table 14 one after another. By placing the products on the mounting surface 14a of the mounting table 14 one after another by the customer, the products are mounted on the mounting table 14 one after another as the imaging time of the captured image advances. Can be reflected. For example, the control unit 23 can acquire the captured images 30, 31 and 32 as shown in FIGS. 4, 5 and 6.

The captured image 30 as shown in FIG. 4 is an image captured at the imaging time t0. At the imaging time t0, no product or the like is placed on the mounting surface 14a of the mounting table 14. The captured image 30 includes an image corresponding to the mounting surface 14a of the mounting table 14.

The captured image 31 as shown in FIG. 5 is an image captured at the imaging time t1. The imaging time t1 is a time after the imaging time t0. At the imaging time t1, butter is placed on the mounting surface 14a of the mounting table 14. The captured image 31 includes a partial image 31a. The partial image 31a is a part of the captured image 31. The partial image 31a corresponds to butter.

The captured image 32 as shown in FIG. 6 is an image captured at the imaging time t2. The imaging time t2 is a time after the imaging time t1. At the imaging time t2, the butter and the rice ball are placed on the mounting surface 14a of the mounting table 14. The rice balls are placed after the customer places the butter on the mounting table 14. The rice balls are layered on a piece of butter. The captured image 32 includes a partial image 32a and a partial image 32b. The partial image 32a and the partial image 32b are a part of the captured image 32. The partial image 32a corresponds to butter. The partial image 32b corresponds to a rice ball. Due to the onigiri being superimposed on a part of the butter, the partial image 32a is partially missing, unlike the partial image 31a as shown in FIG.

The control unit 23 generates a difference image based on two captured images having different imaging times. For example, the control unit 23 may generate a difference image by calculating the difference in the brightness values of each pixel included in each of the two captured images. Here, when the control unit 23 determines that there is a difference between two captured images having different imaging times, the control unit 23 may generate a difference image. Hereinafter, an example of the process of determining whether or not there is a difference between the two captured images will be described.

The control unit 23 detects the number of objects in the captured image by object detection. In the present disclosure, "object detection" means that an object to be detected is detected separately from the background and other objects in the captured image. The object to be detected is a product of a store in which the information processing system 10 is arranged. The control unit 23 may detect the number of objects by detecting the objects in the captured image by a known object detection method. As an example of a known object detection method, there is a method using segmentation, semantic segmentation and / or a bounding box. The control unit 23 may detect the number of objects in the captured image by a known object authentication method. As an example of a known object authentication method, there is a method using machine learning such as deep learning and / or feature amount pattern matching.

The control unit 23 determines whether or not the number of individually detected objects in the captured image has changed. The control unit 23 compares the number of objects detected by the latest detection process with the number of objects detected by the detection process immediately before the latest object detection process, and thereby individually in the captured image. It may be determined whether or not the number of detected objects has changed. With such a configuration, it can be determined whether or not there is a difference between the two captured images.

When the control unit 23 determines that the number of individually detected objects has changed, the control unit 23 determines that the captured image when the number of objects has changed and the captured image at the imaging time earlier than the imaging time of the captured image. Is used to generate a difference image. As an image captured at an imaging time before the imaging time of the captured image when the number of objects changes, an captured image captured at a predetermined time before the imaging time of the captured image when the number of objects changes May be used. The predetermined time may be appropriately set assuming the time required from the customer placing one product on the mounting table 14 to mounting the next product on the mounting table 14. Alternatively, as the captured image at the imaging time before the imaging time of the captured image when the number of objects changes, the captured image before the number of objects changes may be used. The captured image before the number of the objects changes may be the latest one among the captured images before the number of the objects changes. In the present embodiment, the captured image before the number of objects changes is used as the captured image at the imaging time before the capturing time of the captured image when the number of objects changes.

For example, in the detection process, the control unit 23 detects that the number of objects in the captured image 30 is zero. In the next detection process, the control unit 23 detects the number of objects in the captured image 31 as one. The control unit 23 takes an image by comparing one of the number of objects detected by the latest detection process with zero of the number of objects detected by the detection process immediately before the latest detection process. It is determined that the number of individually detected objects in the image 31 has changed. The control unit 23 generates a difference image 40 as shown in FIG. 7 by using the captured image 31 in which the number of objects is changed to one and the captured image 30 in which the number of objects is changed to one. To do. The difference image 40 includes a partial image 40a. The partial image 40a is a region where non-zero pixels are continuous. The partial image 40a can be generated based on the difference value between the partial image 31a of the captured image 31 and the portion of the captured image 30 corresponding to the partial image 31a. The partial image 40a corresponds to the partial image 31a included in the captured image 31 of the two captured images, that is, the captured image 30 and the captured image 31.

For example, in the detection process, the control unit 23 detects two objects in the captured image 32. It is assumed that the detection process immediately before the detection process is the detection process for the captured image 31. The control unit 23 takes an image by comparing two of the number of objects detected by the latest detection process with one of the number of objects detected by the detection process immediately before the latest detection process. It is determined that the number of individually detected objects in the image 32 has changed. The control unit 23 generates a difference image 41 as shown in FIG. 8 by using the captured image 32 in which the number of objects is changed to two and the captured image 31 before the number of objects is changed to two. To do. The difference image 41 includes a partial image 41a. The partial image 41a is a region where non-zero pixels are continuous. The partial image 41a can be generated based on the difference value between the partial image 3b of the captured image 32 and the portion of the captured image 31 corresponding to the partial image 32b. The partial image 41a corresponds to the partial image 32b included in the captured image 32 of the two captured images, that is, the captured image 31 and the captured image 32.

The control unit 23 identifies an object corresponding to a partial image included in at least one of the captured images 30 to 32 by executing object recognition on the difference images 40 and 41. Object recognition for a difference image may include identifying an identifier, or name, of an object corresponding to the partial image by analyzing the partial image in the difference image. Object recognition for a difference image may include detecting an object from the difference image. The control unit 23 may execute a known object recognition method on the difference image. As an example of a known object recognition method, there is a method using machine learning such as deep learning, character recognition, and / or feature amount pattern matching. The control unit 23 detects the partial image 40a by executing the object recognition on the difference image 40, and identifies that the product name of the object corresponding to the partial image 40a is "butter". The control unit 23 detects the partial image 41a corresponding to the object by executing the object recognition for the difference image 41, and identifies that the product name of the object corresponding to the partial image 41a is "rice ball". To do.

When the input unit 21 detects the user input instructing the accounting request, the control unit 23 ends the generation of the difference image. The user input instructing the accounting request can be input from the input unit 21 by the customer after the customer has placed all the products on the mounting table 14. Alternatively, if the control unit 23 does not change the number of objects individually detected in the captured image even after a predetermined time has elapsed after determining that the number of objects in the captured image has changed, the control unit 23 generates a difference image. May be terminated. The predetermined time may be longer than the time expected to be required from the customer placing one product on the mounting table 14 to mounting the next product on the mounting table 14.

When the control unit 23 finishes the process of identifying the object, the control unit 23 transmits the specified product information to the server 12 via the network 13 by the communication unit 20. The product information may include the product name, the number of products, and the like. The server 12 acquires the product information from the information processing device 18 via the network 13. The server 12 calculates the billing amount to be billed to the customer based on the product information. The server 12 transmits a signal indicating the billing amount to the information processing device 18 via the network 13. The control unit 23 acquires a signal indicating the billing amount from the server 12 via the network 13 by the communication unit 20. The control unit 23 transmits a signal indicating the billed amount to the display device 17 by the communication unit 20.

FIG. 9 is a flowchart showing an object identification process of the information processing system 10 according to the embodiment of the present disclosure. When the control unit 23 starts acquiring the image signal from the camera 16, the object identification process may be started. The control unit 23 may store the image signal acquired from the camera 16 in the storage unit 22. Further, the control unit 23 may start the object identification process again after the customer's accounting is completed.

The control unit 23 detects the number of objects in the captured image (step S10). The control unit 23 determines whether or not the number of individually detected objects in the captured image has changed (step S11).

When the control unit 23 determines that the number of individually detected objects has changed (step S11: YES), the control unit 23 proceeds to the process of step S12. On the other hand, when the control unit 23 determines that the number of individually detected objects does not change (step S11: NO), the control unit 23 returns to the process of step S10.

In the process of step S12, the control unit 23 generates a difference image by using the captured image when the number of objects changes and the captured image at the imaging time before the imaging time of the captured image. In the process of step S13, the control unit 23 identifies the object by executing the object recognition for the difference image.

In the process of step S14, the control unit 23 determines whether or not the user input instructing the accounting request is detected by the input unit 21. When the control unit 23 determines that the user input instructing the accounting request has been detected by the input unit 21 (step S14: YES), the control unit 23 proceeds to the process of step S15. On the other hand, if the control unit 23 does not determine that the user input instructing accounting has been detected by the input unit 21 (step S14: Nо), the control unit 23 returns to the process of step S10.

In the process of step S15, the control unit 23 transmits the specified product information to the server 12 by the communication unit 20.

Here, the control unit 23 determines that the number of individually detected objects in the process of step S11 has changed, and even after a lapse of a predetermined time, the number of individually detected objects in the captured image has passed. If does not change, the process of step S15 may proceed.

As described above, in the information processing system 10 according to the present embodiment, the information processing device 18 generates a difference image from two captured images having different imaging times. Even if any of the two captured images includes a plurality of partial images corresponding to the object, the plurality of partial images can be included in separate difference images. For example, the captured image 32 as shown in FIG. 6 includes a partial image 32a corresponding to butter and a partial image 32b corresponding to rice balls. The partial image 40a corresponding to the partial image 32a is included in the difference image 40. The partial image 41a corresponding to the partial image 32b is included in the difference image 41. By executing the object recognition for such a difference image, the information processing apparatus 18 can identify the object more accurately than the case where the object recognition is executed for the captured image.

Further, in the information processing system 10 according to the present embodiment, the object can be specified by executing the object recognition on the difference image without having to image the object from a plurality of directions with a camera. In the information processing system 10, since it is not necessary to take an image of an object from a plurality of directions with a camera, it is not necessary to install a plurality of cameras toward the mounting table 14. With such a configuration, the information processing system 10 according to the present embodiment can identify an object with a simpler configuration.

Therefore, according to the present embodiment, an improved information processing system 10, an information processing device 18, and an information processing method can be provided.

Further, in the information processing system 10 according to the present embodiment, the captured image when the number of individually detected objects changes and the captured image at the imaging time before the imaging time of the captured image are used. A difference image is generated. With such a configuration, a difference image can be generated using a captured image having a difference more reliably.

Next, the information processing system 10 according to another embodiment of the present disclosure will be described. In another embodiment of the present disclosure, the control unit 23 uses the control unit 23 to capture images before and after the imaging time of the captured image including the partial image corresponding to the fingers as the two captured images having different imaging times used for generating the difference image. The captured image of is used. In another embodiment of the present disclosure, the storage unit 22 may store images of fingers and the like.

The control unit 23 determines whether or not the captured image acquired from the camera 16 includes a partial image corresponding to the fingers. The control unit 23 may determine whether or not the partial image corresponding to the finger is included by executing the object recognition in the captured image.

When the control unit 23 determines that the partial image corresponding to the finger is included in the captured image, the control unit 23 acquires the captured image at the imaging time before and after the imaging time of the captured image. For example, the control unit 23 has a second image than the imaging time of the captured image including the partial image corresponding to the finger and the captured image of the captured image including the partial image corresponding to the finger. The captured image at the imaging time after 2 hours may be acquired. The first time and the second time may be appropriately set in consideration of the time required for the customer to place the individual products on the mounting table 14. The first time and the second time may be the same or different. In the present embodiment, the control unit 23 controls the capturing image of the captured image at the first hour before the capturing time of the captured image including the partial image corresponding to the finger, and the capturing time of the captured image including the partial image corresponding to the finger. The captured image at the imaging time after the second hour is acquired.

For example, the control unit 23 acquires the captured image 33 as shown in FIG. The captured image 33 is an image captured at the imaging time t01. At the imaging time t01, the customer is trying to grab the butter by hand and place it on the mounting surface 14a of the mounting table 14. The captured image 33 includes a partial image 33a and a partial image 33b. The partial image 33a and the partial image 33b are a part of the captured image 33. The partial image 33a corresponds to the fingers. The partial image 33b corresponds to butter. The control unit 23 determines that the captured image 33 includes the partial image 33a corresponding to the fingers. Here, it is assumed that the imaging time one hour before the imaging time t01 is the imaging time t0 of the above-mentioned captured image 30. Further, it is assumed that the imaging time after the second hour from the imaging time t01 is the imaging time t1 of the above-mentioned captured image 31. The control unit 23 uses the above-mentioned captured image 30 and the above-mentioned captured image 31 to generate the above-mentioned difference image 40.

For example, the control unit 23 acquires the captured image 34 as shown in FIG. The captured image 34 is an image captured at the imaging time t12. At the imaging time t12, the customer is trying to grab the rice ball by hand and place it on the mounting surface 14a of the mounting table 14. The captured image 34 includes a partial image 34a, a partial image 34b, and a partial image 34c. The partial image 34a, the partial image 34b, and the partial image 34c are a part of the captured image 34. The partial image 34a corresponds to the fingers. The partial image 34b corresponds to a rice ball. The partial image 34c corresponds to butter. The control unit 23 determines that the captured image 34 includes the partial image 34a corresponding to the fingers. Here, it is assumed that the imaging time one hour before the imaging time t12 is the imaging time t1 of the above-mentioned captured image 31. Further, it is assumed that the imaging time after the second hour from the imaging time t12 is the imaging time t2 of the above-mentioned captured image 32. The control unit 23 uses the above-mentioned captured image 31 and the above-mentioned captured image 32 to generate the above-mentioned difference image 41.

FIG. 12 is a flowchart showing an object identification process of the information processing system 10 according to another embodiment of the present disclosure. When the control unit 23 starts acquiring the image signal from the camera 16, the object identification process may be started. The control unit 23 may store the image signal acquired from the camera 16 in the storage unit 22. Further, the control unit 23 may start the object identification process again after the customer's accounting is completed.

By executing object recognition in the captured image (step S20), the control unit 23 determines whether or not the partial image corresponding to the finger is included (step S21). When the control unit 23 determines that the partial image corresponding to the finger is included in the captured image (step S21: YES), the control unit 23 proceeds to the process of step S22. On the other hand, when it is not determined that the partial image corresponding to the finger is included in the captured image (step S21: NO), the process returns to the process of step S20.

In the process of step S22, the control unit 23 generates a difference image by using the captured images of the captured images before and after the captured time of the captured images including the partial images corresponding to the fingers.

The control unit 23 executes the processes of steps S23, S24, and S25 in the same manner as the processes of steps S13, S14, and S15 shown in FIG.

FIG. 13 is a diagram showing an overall configuration of the information processing system 110 according to still another embodiment of the present disclosure. The information processing system 110 includes a camera 16, an information processing device 18, and a weight sensor 19. The information processing system 110 may include a mounting table 14, a support column 15, and a display device 17.

The weight sensor 19 detects the weight applied to the mounting surface 14a of the mounting table 14. The weight sensor 19 outputs the detected weight to the information processing device 18 via the communication line. The weight sensor 19 is appropriately arranged at a position where the weight applied to the mounting surface 14a can be detected.

In the information processing device 18 according to the present embodiment, the communication unit 20 may include a communication module capable of communicating with the weight sensor 19 via a communication line. The communication line between the weight sensor 19 and the communication unit 20 may be configured to include wired or wireless.

In the information processing apparatus 18 according to the present embodiment, the control unit 23 has the time when the weight applied to the mounting surface 14a of the mounting table 14 changes as the above-mentioned two captured images having different imaging times used for generating the difference image. Use the captured images of the previous and next imaging times. For example, the control unit 23 can acquire the weight applied to the mounting surface 14a of the mounting table 14 by the communication unit 20 from the weight sensor 19 via the communication line. The control unit 23 determines whether or not the weight applied to the mounting surface 14a of the mounting table 14 has changed. When the control unit 23 determines that the weight applied to the mounting surface 14a of the mounting table 14 has changed, the control unit 23 acquires captured images of the imaging times before and after the time when the weight applied to the mounting surface 14a of the mounting table 14 has changed. .. As an image of the imaging time before the time when the weight changes, the control unit 23 may acquire an image captured at the imaging time 3 hours before the time when the weight applied to the mounting surface 14a changes. The third time may be appropriately set assuming the time required for the customer to place the individual products on the mounting table 14. As an image captured at the imaging time after the time when the weight changes, the control unit 23 determines the weight applied to the mounting surface 14a in the captured image captured after the time when the weight applied to the mounting surface 14a changes. The captured image at the imaging time closest to the time when is changed may be acquired.

For example, when a customer places butter on a mounting table 14 on which nothing is placed, the weight of the butter increases the weight applied to the mounting surface 14a of the mounting table 14. That is, the weight applied to the mounting surface 14a of the mounting table 14 changes. The control unit 23 shall acquire the above-mentioned captured images 30 and 31 as captured images at the captured images before and after the time when the weight applied to the mounting surface 14a of the mounting table 14 changes. The control unit 23 can generate the above-mentioned difference image 40 by using the captured image 30 and the captured image 31.

For example, when a customer places a rice ball on a mounting table 14 on which butter is placed, the weight of the rice ball increases the weight applied to the mounting surface 14a of the mounting table 14. That is, the weight applied to the mounting surface 14a of the mounting table 14 changes. The control unit 23 shall acquire the above-mentioned captured images 31 and 32 as captured images at the captured images before and after the time when the weight applied to the mounting surface 14a of the mounting table 14 changes. The control unit 23 can generate the above-mentioned difference image 41 by using the captured image 31 and the captured image 32.

When the weight applied to the mounting surface 14a of the mounting table 14 changes, the control unit 23 may transmit a signal instructing the camera 16 to take an image by the communication unit 20. Here, when the customer purchases a plurality of products, the customers can place the products on the mounting surface 14a of the mounting table 14 one after another. As the customer places the products on the mounting surface 14a of the mounting table 14 one after another, the weight applied to the mounting surface 14a may change at any time. When the weight applied to the mounting surface 14a changes at any time and the control unit 23 transmits a signal instructing the camera 16 to take an image when the weight applied to the mounting surface 14a changes, the control unit 23 performs imaging. The indicating signal can be transmitted at any time. The camera 16 may capture a still image at any time by using a signal instructing imaging from the control unit 23 as a predetermined trigger.

FIG. 14 is a flowchart showing an object identification process of the information processing system 110 according to still another embodiment of the present disclosure. When the control unit 23 starts acquiring the image signal from the camera 16, the object identification process may be started. The control unit 23 may store the image signal acquired from the camera 16 in the storage unit 22. Further, the control unit 23 may start the object identification process again after the customer's accounting is completed.

The control unit 23 acquires the weight applied to the mounting surface 14a of the mounting table 14 by the communication unit 20 from the weight sensor 19 via the communication line (step S30). The control unit 23 determines whether or not the weight applied to the mounting surface 14a of the mounting table 14 has changed (step S31).

When the control unit 23 determines that the weight applied to the mounting surface 14a of the mounting table 14 has changed (step S31: YES), the control unit 23 proceeds to the process of step S32. On the other hand, when the control unit 23 does not determine that the weight applied to the mounting surface 14a of the mounting table 14 has changed (step S31: NO), the control unit 23 returns to the process of step S30.

In the process of step S32, the control unit 23 generates a difference image by using the captured images at the imaging times before and after the time when the weight applied to the mounting surface 14a of the mounting table 14 changes.

The control unit 23 executes the processes of steps S33, S34, and S35 in the same manner as the processes of steps S13, S14, and S15 shown in FIG.

FIG. 15 is a diagram showing an example of a difference image 42 according to still another embodiment of the present disclosure. The difference image 42 includes a partial image corresponding to each of the two objects. For example, the difference image 42 includes a partial image 42a and a partial image 42b. The partial image 42a and the partial image 42b are a part of the difference image 42. Partial image 42a corresponds to chocolate. The partial image 42b corresponds to a rice ball.

Depending on the customer, a plurality of products may be placed on the mounting surface 14a of the mounting table 14 at a time. When a plurality of products are placed on the mounting surface 14a at a time by a customer, a difference image including a partial image corresponding to each of the plurality of objects, such as the difference image 42, can be generated.

In the information processing device 18 according to the present embodiment, when a plurality of objects are individually detected from the difference images, the control unit 23 executes object recognition for each partial image corresponding to each object. For example, the control unit 23 may detect the number of objects in the difference image. The control unit 23 may detect the number of objects by detecting the objects in the difference image by the above-mentioned known object detection method. The control unit 23 determines whether or not a plurality of objects are individually detected from the difference image. When it is determined that a plurality of objects are individually detected from the difference image, the control unit 23 executes object recognition for each partial image corresponding to each object.

For example, the control unit 23 detects two distinct objects from the difference image 42. The control unit 23 determines that a plurality of objects have been detected separately from the difference image 42. The control unit 23 executes object recognition for each of the partial image 42a and the partial image 42b corresponding to each object. The control unit 23 identifies the object corresponding to the partial image 42a as chocolate by executing the object recognition on the partial image 42a. The control unit 23 identifies the object corresponding to the partial image 42b as a rice ball by executing the object recognition on the partial image 42b.

FIG. 16 is a flowchart showing object recognition for a difference image of the information processing system 10 according to still another embodiment of the present disclosure. The object recognition for the difference image may be executed instead of the process of step S13 shown in FIG. 9, the process of step S23 shown in FIG. 12, and the process of step S33 shown in FIG. In this case, the control unit 23 executes the process of step S12 shown in FIG. 9, the process of step S22 shown in FIG. 12, or the process of step S32 shown in FIG. 14, and then the difference. Object recognition for the image may be started.

The control unit 23 detects the number of objects in the difference image (step S40). The control unit 23 determines whether or not a plurality of objects are individually detected from the difference image (step S41).

When the control unit 23 does not determine that a plurality of objects have been detected separately from the difference image (step S41: Nо), the control unit 23 proceeds to the process of step S42. For example, when the control unit 23 determines that one object has been detected from the difference image, the control unit 23 proceeds to the process of step S42. In the process of step S42, the control unit 23 executes object recognition for the difference image. After executing the process of step S42, the control unit 23 proceeds to the process of step S14 shown in FIG. 9, the process of step S24 shown in FIG. 12, or the process of step S34 shown in FIG.

When the control unit 23 determines that a plurality of objects are individually detected from the difference image (step S41: YES), the control unit 23 proceeds to the process of step S43. When proceeding to the process of step S43, the control unit 23 may store the number of objects detected from the difference image in the storage unit 22.

In the process of step S43, the control unit 23 executes object recognition on the partial image corresponding to one object included in the difference image. In the process of step S44, the control unit 23 determines whether or not the process of step S43 has been executed for all the partial images corresponding to the objects included in the difference image.

When the control unit 23 determines that the process of step S43 has been executed for all of the partial images corresponding to the objects (step S44: YES), the process of step S14 shown in FIG. 9 and the process of step S24 shown in FIG. The process proceeds to the process or the process of step S34 shown in FIG.

When the control unit 23 does not determine that the process of step S43 has been executed for all of the partial images corresponding to the objects (step S44: NO), the control unit 23 executes the process of step S43 again. In the process of step S43 to be executed again, the control unit 23 executes object recognition for the partial image corresponding to the object detected from the difference image, for which the object recognition is not executed.

As described above, according to the present embodiment, when a plurality of objects are individually detected from the difference image, object recognition is executed for each partial image corresponding to each object. By such processing, the object can be identified more accurately.

FIG. 17 is a diagram showing an overall configuration of the information processing system 210 according to still another embodiment of the present disclosure. The information processing system 210 includes a shopping cart 214, a camera 16, a display device 17, and an information processing device 18.

The shopping cart 214 may be used by a customer who visits the store. The customer can put the goods he / she wants to purchase in the shopping cart 214.

The camera 16 takes an image of the inside of the shopping cart 214. The camera 16 may be arranged so that the inside of the shopping cart 214 can be imaged. The camera 16 may be located at the upper edge of the shopping cart 214. The camera 16 may be fixed to the upper edge of the shopping cart 214 so that the inside of the shopping cart 214 can be imaged.

The camera 16 may include a communication module capable of wirelessly communicating with the information processing device 18. The camera 16 may transmit an image signal to the information processing device 18 by the wireless module.

As described above, the information processing apparatus 18 can identify an object corresponding to an image included in at least one of the two captured images based on the difference image of the two captured images having different imaging times.

The configuration according to the present disclosure is not limited to the embodiments described above, and can be modified or modified in many ways. For example, the functions and the like included in each component and the like can be rearranged so as not to be logically inconsistent, and a plurality of components and the like can be combined or divided into one.

10, 110, 210 Information processing system 11 Settlement system 12 Server 13 Network 14 Mounting stand 14a Mounting surface 15 Support pillar 16 Camera 17 Display device 18 Information processing device 19 Weight sensor 20 Communication unit 21 Input unit 214 Shopping basket 22 Storage unit 23 Control unit 30, 31, 32, 33, 34 Captured image 31a, 32a, 32b, 33a, 33b, 34a, 34b, 34c Partial image 40, 41, 42 Difference image 40a, 41a, 42a, 42b Partial image

Claims (11)

An imaging unit that generates an image signal by imaging,
Information that acquires a captured image corresponding to the image signal and identifies an object corresponding to a partial image included in at least one of the two captured images based on the difference image of the two captured images having different imaging times. An information processing system including a processing device. The information processing system according to claim 1.
The information processing device is an information processing system that identifies an object by performing object recognition on the difference image. The information processing system according to claim 2.
The information processing device is an information processing system that executes the object recognition for each partial image corresponding to each object when a plurality of objects are individually detected from the difference image. The information processing system according to any one of claims 1 to 3.
An information processing system that generates the difference image when the information processing device determines that there is a difference between the two captured images. The information processing system according to claim 4.
The information processing device detects the number of objects from the captured image, the captured image when the number of individually detected objects changes, and the imaging at an imaging time before the imaging time of the captured image. An information processing system that generates the difference image using an image. The information processing system according to claim 5.
The information processing device is an information processing system that uses an image captured before the number of objects changes as an image captured at an imaging time before the imaging time. The information processing system according to any one of claims 1 to 3.
The information processing device is an information processing system that uses captured images at imaging times before and after the imaging time of the captured image including a partial image corresponding to a finger as two captured images having different imaging times. The information processing system according to any one of claims 1 to 7.
Further provided with a mounting table on which the product, which is the object, is placed,
The image pickup unit is an information processing system in which the mounting surface of the above-mentioned stand is arranged so as to be able to image. The information processing system according to any one of claims 1 to 3.
A mounting table on which the product, which is the object, is placed, and
Further equipped with a weight sensor that detects the weight applied to the mounting surface of the above-mentioned stand.
The information processing device is an information processing system that uses captured images at imaging times before and after the time when the weight applied to the above-mentioned mounting surface changes as two captured images having different imaging times. A communication unit that can communicate with an imaging unit that generates an image signal by imaging,
An object corresponding to a partial image included in at least one of the two captured images based on a difference image of the two captured images having different imaging times by acquiring an captured image corresponding to the image signal by the communication unit. An information processing device including a control unit for specifying an image. Generating an image signal by imaging and
Acquiring a captured image corresponding to the image signal and identifying an object corresponding to a partial image included in at least one of the two captured images based on a difference image of the two captured images having different imaging times. Information processing methods, including.

Images