JP2021026724A - Information processing system, information processing device, information processing method and information processing program - Google Patents

Abstract

To reduce a processing time and processing cost in information processing for determining the stockout of commodity products.SOLUTION: An information processing system 1 determines stockout of commodity products displayed in a store Sh. The information processing system comprises: an imaging unit 14 which generates a photographed image obtained by imaging commodity products; a distance information acquisition unit 15 which acquires distance information indicating a distance between the imaging unit and a subject imaged in the photographed image; a dynamic body determination unit 16 which determines presence/absence of a dynamic body moving in a space between the imaging unit and the commodity product on the basis of the distance information; an image acquisition unit 17 which acquires a photographed image in which the dynamic body is not imaged as a specific photographed image on the basis of the determination result of the dynamic body determination unit; a storage unit 12 (22) which stores the specific photographed image; and a stockout determination unit 24 which determines presence/absence of the stockout on the basis of the specific photographed image.SELECTED DRAWING: Figure 9

Description

The present invention relates to an information processing system, an information processing device, an information processing method, and an information processing program.

If there is a shortage (out of stock) of displayed products at sales floors such as supermarkets and convenience stores (convenience stores), sales opportunities for the products will be lost and sales at the sales floor will decrease. Therefore, the clerk at the sales floor must regularly monitor the products, and if there is a shortage of products, promptly perform the work (product management) to replenish the missing products and eliminate the shortage of products. It doesn't become.

As such product management, product management is performed to manage a shortage of products based on an image captured by a camera on a shelf on which products are displayed. In this product management, if a moving object such as a person or a cart exists in front of the shelf, the moving object is imaged together with the shelf or the product, and the product in the area shadowed by the moving object is not imaged.

When the arithmetic processing for detecting the shortage of products is executed based on the captured image of the moving object, not only the arithmetic processing requires a long time, but also the shortage of products cannot be detected correctly. As a result, unnecessary processing time and processing cost may occur in the arithmetic processing. In particular, in a large-scale sales floor, the number of shelves reaches several hundreds, which may result in enormous processing time and processing cost.

In order not to cause such a problem, a technique has been proposed in which information processing for removing a moving body is executed on an image captured by a moving body to manage a shortage of products based on the image from which the moving body has been removed. (See, for example, Patent Documents 1 and 2).

The technique disclosed in Patent Document 1 calculates dominant image information (dominant color information) for each pixel based on a plurality of captured images, and uses a known background image generation technique to remove moving objects (moving objects). Removed image) is generated. The dominant image information is updated each time the captured image is captured. The technology detects the shortage of products based on the moving object removal image, and displays the shortage state (deficiency state) of the products on the monitor. The technology measures the duration of the shortage state and changes the display mode of the area where the product is short according to the duration. As a result, the clerk who checks the monitor can easily grasp the area where the replenishment of the product is delayed.

The technique disclosed in Patent Document 2 cuts a region of a moving body from an image captured by a moving body, and sets a corresponding region of a reference image (a captured image in which the moving body is not captured) as a cut region. By copying, a moving object removal image is generated. The technique then generates the latest moving object removal image by using the latest moving object removal image instead of the reference image. The technology detects the shortage state of the product based on the moving object removal image and calculates the recovery degree of the shortage state. The shortage state is recovered by replenishing the products and displaying the products in the rear in the front. The clerk can objectively grasp the display status of the product based on the calculated recovery degree.

As described above, the techniques disclosed in Patent Documents 1 and 2 execute information processing (moving object removing processing) for generating a moving object removing image for each captured image. As a result, these techniques require enormous processing time and processing cost for the moving object removal processing. In addition, the technology executes information processing (deficiency detection processing) for detecting the shortage state of the product for each moving object removal image. In this technique, since the shortage detection process is executed for a huge number of moving object removal images, a huge amount of processing time and processing cost may occur in the shortage detection process.

JP-A-2017-162375 Japanese Unexamined Patent Publication No. 2006-059038

An object of the present invention is to reduce processing time and processing cost in information processing for determining a shortage of products.

The present invention is an information processing system for determining a shortage of a product displayed in a store, and is an imaging unit that generates an image captured by the product, an imaging unit, and a subject imaged in the captured image. A distance information acquisition unit that acquires distance information indicating the distance between the image and the image, a moving object determination unit that determines the presence or absence of a moving object moving in the space between the image pickup unit and the product based on the distance information, and a moving object determination. Based on the judgment result of the unit, the image acquisition unit that acquires the captured image in which the moving object is not captured as the specific captured image, the storage unit that stores the specific captured image, and the presence or absence of the missing item based on the specific captured image. It is characterized by having a shortage determination unit for determining.

According to the present invention, it is possible to reduce the processing time and the processing cost in the information processing for determining the shortage of products.

It is a schematic diagram which shows the example of the image capture image generated by the image pickup apparatus provided in the information processing system which concerns on this invention. It is a schematic diagram which shows another example of the captured image of FIG. It is a system block diagram which shows the embodiment of the information processing system which concerns on this invention. It is a schematic diagram which shows the structural example of the store which uses the information processing system of FIG. It is a functional block diagram of the image pickup apparatus of FIG. It is a functional block diagram of the server provided in the information processing system of FIG. It is a functional block diagram of the information processing terminal provided in the information processing system of FIG. It is a flowchart of information processing of the information processing system of FIG. It is a flowchart of the moving body determination process included in the information processing of FIG. It is a flowchart of the shortage determination process included in the information processing of FIG. It is a schematic diagram which shows the example of the information generated by the shortage determination process of FIG. It is a schematic diagram which shows the example of the information displayed in the shortage determination process of FIG.

Hereinafter, embodiments of the information processing system, the information processing device, the information processing method, and the information processing program according to the present invention will be described with reference to the drawings.

● Information processing system ●
First, an embodiment of an information processing system (hereinafter referred to as "the present system") according to the present invention will be described.

This system determines the shortage of products displayed in a closed space. In addition, this system determines the presence or absence of a moving object moving between the image pickup unit that images the product and the product based on the distance information described later. Further, this system determines whether or not there is a shortage (missing item) of the products displayed in the closed space based on the captured image (specific captured image described later) in which the moving object is not captured.

The "closed space" is a three-dimensional space partitioned by an arbitrary range, and is a space in which the system can determine a shortage of products. The closed space includes a space that is physically partitioned and a space that is not physically partitioned. That is, for example, a closed space includes an internal space of a store such as a supermarket or a convenience store, an internal space of each store of a shopping mall or an airport duty-free shop where many stores are connected, and an outdoor space such as an outdoor sales floor of a store. ..

A "good" is an item that is displayed in a closed space and purchased by a customer. The products are displayed in a display space such as a shelf, a table, or a desk installed in a closed space in a state of being arranged (or stacked) according to the type of the product.

A "moving body" is an object that moves in a closed space. That is, for example, the moving body is a customer (person) moving in the aisle in the store or a shopping cart used by the customer (hereinafter referred to as "cart").

The moving object is not only an object that is constantly moving in the closed space (the space between the image pickup device and the displayed product), but also an object that is temporarily stopped in the closed space (for example, is stopped). Includes (customers) and objects temporarily placed in closed spaces (eg carts, shopping carts, etc.).

The "captured image" is an image in which a subject is captured by an imaging device such as a camera, and is, for example, a still image or each frame of a moving image. The captured image is classified into a specific captured image and a non-specific captured image depending on the presence or absence of a moving object. That is, the captured image includes a specific captured image and a non-specific captured image.

A "subject" is an object to be imaged by an imaging device such as a shelf, a product, or a moving object.

The "specific captured image" is an captured image in which a moving object is not captured. That is, the specific captured image is an captured image in which a moving object is not included in the subject. That is, the subject image captured in the captured image does not include the moving object image.

The "subject image" is an image of a subject included in the captured image. The "moving body image" is an image of a moving body included in the captured image.

FIG. 1 is a schematic view showing an example of a specific captured image.
The figure shows that the specific captured image includes the shelf image of the shelf and the product image of the product, and does not include the moving object image. The figure shows that the price tags P1-P12 of each product are arranged at the lower left for each product in the front view.

The "shelf image" is an image of a shelf included in the captured image. The "product image" is an image of a product included in the captured image.

The "non-specific captured image" is an captured image in which a moving object is captured among the captured images. That is, the non-specific captured image is an captured image in which a moving object is included in the subject. That is, the subject image captured in the captured image includes a moving object image.

FIG. 2 is a schematic view showing an example of a non-specific captured image.
The figure shows that the non-specific captured image includes the shelf image of the shelf Sc, the product image of the product, and the moving object image. In addition, the figure shows that a part of each of the shelf image and the product image is hidden behind the moving body image and hidden in the moving body image.

"Out of stock" indicates a state in which some or all of the products displayed in a predetermined state are missing. That is, for example, a shortage is a state in which one product displayed on the shelf is carried (purchased) by the customer and one product is missing, and a product displayed on the shelf is carried by the customer. Including the state where all the goods are carried and there is no such product.

The embodiment described below is an example of a case where the system detects a shortage of products displayed on a shelf installed in a store in a supermarket sales floor (that is, in a store). The content of the invention will be described.

● Configuration of Information Processing System FIG. 3 is a system configuration diagram showing an embodiment of this system.
FIG. 4 is a schematic view showing a configuration example of a store.
FIG. 4 shows that six shelves Sc (1-6) and six image pickup devices 10 (1-6) corresponding to each shelf Sc (1-6) are arranged in the store Sh. Is shown. Further, in FIG. 2, two customers Cu (1,2) shop using the cart C (1,2), and two users Us (1,2) use the information processing terminal 30 (1). , 2) is shown while moving in the store Sh.

The "user Us" is a person who uses the system 1, and is, for example, a manager or an employee of the store Sh.

The system 1 includes an image pickup device 10, a server 20, an information processing terminal 30, and a POS (Point Of Sales) terminal 40.

The image pickup device 10, the server 20, the information processing terminal 30, and the POS terminal 40 are each connected via a network N (communication line) that uses a wired communication method or a wireless communication method. The network N is, for example, a communication network such as the Internet, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), and Wi-Fi (registered trademark).

The image pickup device 10 takes an image of the shelf Sc and the products displayed on the shelf Sc in the store Sh, and generates an captured image. The image pickup device 10 is, for example, a camera device including a camera function for capturing a still image of a shelf Sc in a store Sh at a first time interval, and a circuit for executing information processing described later. The image pickup device 10 is a customer Cu (a cart C used by the customer Cu) that moves in a space between the shelf Sc in the store Sh, the products displayed on the shelf Sc, and the shelf Sc (product) and the image pickup device 10. Etc.), and are placed on the ceiling surface and wall surface of the store Sh so that images can be taken. In the present embodiment, the system 1 includes at least one (six in the present embodiment) imaging devices 10 (1-6). The specific configuration of the image pickup apparatus 10 will be described later.

The "first time interval" is a time interval in which the image pickup apparatus 10 periodically images a subject, and is, for example, 1/30 second (30 fsp).

The imaging device in the present invention may be an information processing device (computer) having a camera function including an imaging unit (see FIG. 5) described later. That is, for example, the image pickup device may be a smartphone having a camera function, a mobile phone, a tablet PC (Personal Computer), a notebook computer, or a digital camera having a communication function.

Further, the number of image pickup devices in the present invention is not limited to "6" as long as it can image the shelves for which product shortages are determined and the products displayed on the shelves.

Further, the imaging device in the present invention may capture a moving image or may capture a moving image and a still image.

Furthermore, the arrangement of the imaging device in the present invention is not limited to the ceiling surface and the wall surface of the store. That is, for example, the image pickup apparatus may be arranged on shelves arranged so as to face each other across the aisle. Specifically, the imaging device may be arranged on a shelf board of a shelf or a price rail attached to the shelf board.

The server 20 determines whether or not the product is out of stock based on the specific captured image among the captured images generated by the imaging device 10. The server 20 is, for example, a personal computer. The server 20 is arranged, for example, in the office Of of the store Sh. The specific configuration of the server 20 will be described later.

The server in the present invention may be a device such as a cloud server or NAS (Network Attached Storage) that can store information via the network N.

The information processing terminal 30 is carried by the user Us and displays the state of the product shortage on the user Us. The information processing terminal 30 is a portable terminal having a function of inputting information and a function of displaying information. The information processing terminal 30 is, for example, a smart phone, a mobile phone, a tablet PC, or a PDA (Personal Digital Assistant).

The POS terminal 40 executes information processing such as a product settlement process and a product sales data generation process. The POS terminal 40 is, for example, a known register terminal. The POS terminal 40 is arranged, for example, in the accounting area in the sales floor R. In the present embodiment, the system 1 includes three POS terminals 40 (1-3). Since the configuration of the POS terminal 40 and the processing of the POS terminal 40 are well known, detailed description of the configuration and processing of the POS terminal 40 will be omitted.

● Configuration of the image pickup device FIG. 5 is a functional block diagram of the image pickup device 10.

The image pickup device 10 includes a communication unit 11, a storage unit 12, a control unit 13, an image pickup unit 14, a distance information acquisition unit 15, a moving object determination unit 16, and an image acquisition unit 17.

The communication unit 11 communicates with the server 20 via the network N. The communication unit 11 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 12 stores information necessary for the image pickup apparatus 10 to execute information processing described later. The storage unit 12 is a storage medium such as a magnetic disk or a semiconductor memory. The storage unit 12 is an example of the storage unit in the present invention.

The control unit 13 executes various types of information processing according to a program to control the operation of the entire image pickup apparatus 10. The control unit 13 includes, for example, a processor such as a CPU, MPU (Micro Processing Unit) or DSP (Digital Signal Processor), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) and a semiconductor. It is composed of a memory.

The imaging unit 14 images the shelves Sc and the products displayed on the shelves Sc in the store Sh, and generates an captured image in which the shelves Sc and the products are captured. The image pickup unit 14 is composed of, for example, a lens group, a shutter, an image pickup element 141, an A / D converter, and the like. The captured image generated by the imaging unit 14 is specified as a specific captured image by the image acquisition unit 17, and is stored in the storage unit 12.

The image pickup device 141 is a known image plane phase difference type image pickup device. That is, for example, the incident light passing through the lens group is divided into light in two directions and received by the photoelectric conversion unit of the image sensor 141. The photoelectric conversion unit generates and outputs a pair of image signals. The amount of image shift (defocus amount) can be calculated from the phase difference of the image signal. Further, the image pickup unit 14 can generate an image pickup image from the output signal (pair of image signals) of the image pickup device 141 by adding a pair of image signals. In other words, the image pickup device 141 can generate a captured image.

The image pickup device in the present invention may output a pair of image signals from the photoelectric conversion units corresponding to all the pixels included in the image pickup element, or a pair of photoelectric conversion units corresponding to some pixels. The image signal may be output.

The distance information acquisition unit 15 acquires distance information indicating the distance between the image pickup unit 14 and the subject to be captured in the captured image, based on the output signals (a pair of image signals) from the image sensor 141. The distance information acquisition unit 15 is a known arithmetic circuit capable of calculating distance information from the phase difference of an image signal.

The "distance information" is information on the distance between each subject and the image pickup device 10 (image sensor 141). The distance information is, for example, a defocus amount.

The moving object determination unit 16 determines the presence or absence of a moving object moving in the space between the imaging unit 14 and the shelf Sc (commodity) based on the distance information. The moving body determination unit 16 is a known arithmetic circuit capable of determining the presence or absence of a moving body based on the distance information.

The image acquisition unit 17 acquires an captured image in which a moving object is not captured as a specific captured image based on the determination result of the moving object determination unit 16. The image acquisition unit 17 is a known arithmetic circuit capable of acquiring an image captured from the output signal of the image sensor 141. The specific captured image is stored in the storage unit 12.

Each of the distance information acquisition unit 15, the moving object determination unit 16, and the image acquisition unit 17 is composed of, for example, a common IC (Integrated Circuit), a semiconductor memory, or the like.

The distance information acquisition unit, the moving object determination unit, and the image acquisition unit in the present invention may each be configured by an IC or a semiconductor memory dedicated to information processing.

● Server configuration FIG. 6 is a functional block diagram of the server 20.

The server 20 includes a communication unit 21, a storage unit 22, a control unit 23, a shortage determination unit 24, and a shortage information generation unit 25.

The communication unit 21 communicates with the image pickup device 10 and the information processing terminal 30 via the network N. The communication unit 21 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 22 stores information necessary for the server 20 to execute information processing described later. The storage unit 22 is composed of, for example, a recording device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), a semiconductor memory element such as a RAM or a flash memory, and the like. The storage unit 22 is another example of the storage unit in the present invention.

The control unit 23 executes various information processing according to the program and controls the operation of the entire server 20. The specific configuration and specific operation of the control unit 23 will be described later.

The shortage determination unit 24 determines whether or not the product is out of stock based on the specific captured image stored in the storage unit 22. The specific configuration and specific operation of the shortage determination unit 24 will be described later.

The shortage information generation unit 25 generates out-of-stock information when it is determined by the shortage determination unit 24 that there is a shortage. The specific configuration and specific operation of the shortage information generation unit 25 will be described later.

The "out of stock information" is, for example, information on the out of stock such as the type of the product with the out of stock and the display position in the store Sh.

The control unit 23, the shortage determination unit 24, and the shortage information generation unit 25 are composed of, for example, a processor such as a CPU, MPU, or DSP, an integrated circuit such as an ASIC or FPGA, and a semiconductor memory.

The control unit, the determination unit, and the shortage information generation unit in the present invention may be composed of a common processor or integrated circuit and a semiconductor memory, or may be composed of an individual processor or an integrated circuit and a semiconductor memory. May be done.

● Configuration of Information Processing Terminal FIG. 7 is a functional block diagram of the information processing terminal 30.

The information processing terminal 30 includes a communication unit 31, a storage unit 32, a control unit 33, a display unit 34, and an operation unit 35.

The communication unit 31 communicates with the image pickup apparatus 10 and the server 20 via the network N. The communication unit 31 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 32 stores information (for example, out-of-stock information) transmitted from the server 20. The storage unit 32 is composed of, for example, a recording device such as an HDD or SSD, a semiconductor memory element such as a RAM or a flash memory, or the like.

The control unit 33 executes various types of information processing according to a program to control the operation of the entire information processing terminal 30. The control unit 33 is composed of, for example, a processor such as a CPU, MPU, or DSP, or an integrated circuit such as an ASIC or FPGA, and a semiconductor memory such as a RAM, ROM, or flash memory.

The display unit 34 displays information on products with missing items based on the missing item information. The operation unit 35 receives the operation of the user. The display unit 34 and the operation unit 35 are, for example, a touch panel type liquid crystal display. The display unit 34 is an example of the display unit in the present invention.

● Operation of information processing system Next, the operation of this system 1 will be described. In the following description, the system 1 acquires distance information based on the output signal from the image sensor 141 of the image plane phase difference method, and determines the presence or absence of a moving object based on the distance information.

FIG. 8 is a flowchart of information processing of the system 1.

First, the image pickup device 10 detects a state (zoom, focus, aperture, etc.) of a lens group (not shown) included in the image pickup device 10, reads out the shooting conditions from the storage unit 12, and sets the shooting conditions (S1).

The "shooting condition" is a condition for imaging a product on the shelf Sc corresponding to the imaging device 10, such as a state of a lens group, an exposure amount, and an image quality. In the present embodiment, the imaging conditions are fixed for each imaging device 10, and are stored in advance in the storage unit 12, for example, by the user.

Next, the image pickup device 10 drives the image pickup device 141 to acquire an output signal (image signal) from the image pickup device 141 (S2). That is, the image pickup device 10 (imaging element 141) generates a captured image of the corresponding shelf Sc and the product. The image pickup device 10 generates a captured image at each first time interval. The first time interval is counted by, for example, a counter included in the control unit 13.

Next, the image pickup apparatus 10 (for example, the control unit 13) recognizes the pattern of each subject image based on the output signal (S3).

Next, the imaging device 10 (distance information acquisition unit 15) identifies a main subject to be focused on and another subject (background, moving object, etc.) based on the recognition result of the process (S3) (S4). .. In the present embodiment, for convenience of explanation, the main subjects are the shelf Sc and the products displayed on the shelf Sc.

Next, the image pickup apparatus 10 (distance information acquisition unit 15) calculates (acquires) the distance information of each of the main subject image and the other subject image based on the output signal (S5). That is, for example, the image pickup apparatus 10 (distance information acquisition unit 15) calculates the image shift amount of each of the main subject image and the other subject image from the pair of image signals, and uses a known correlation calculation process to defocus the amount. Calculate (distance information).

The image pickup apparatus in the present invention may specify the subject having the smallest amount of image shift (in focus) as the main subject based on the distance information.

Next, the system 1 executes the moving object determination process (S6).

FIG. 9 is a flowchart of the moving body determination process (S6).

The "moving object determination process (S6)" determines the presence or absence of a moving object moving in the space (passage) between the imaging device 10 (imaging unit 14) and the main subject (shelf Sc, product) based on the distance information. It is a process. The moving object determination process (S6) is executed for each captured image generated by the image sensor 141.

The image pickup apparatus 10 (moving object determination unit 16) determines the presence or absence of a moving object based on the amount of change in distance information in each pixel of the area to be occupied by the main subject image (hereinafter referred to as “main area”). That is, for example, the distance between the image pickup device 10 and the shelf Sc is fixed. Therefore, when the moving object is not captured in the captured image, the distance information of the main region is substantially constant. Therefore, when a moving object enters the passage, a part of the area of the main subject image becomes a shadow of the moving object image, and the distance information in the same area changes (shortens) due to the presence of the moving object. As a result, the imaging device 10 (moving object determination unit 16) can determine the presence or absence of a moving object based on the change in the distance information of the main region.

First, the image pickup apparatus 10 (moving object determination unit 16) compares the distance information with the first threshold value α1 for each pixel in the main region (S601). The “first threshold value α1” is a threshold value used for determining whether or not the distance information has changed. The first threshold value α1 is stored in advance in a storage means (semiconductor memory), a storage unit 12, or the like of the moving object determination unit 16.

When the distance information of the pixels is equal to or greater than the first threshold value α1 (“Yes” in S601), the imaging device 10 (moving object determination unit 16) sets the subject image corresponding to the same pixel as a moving object image candidate (hereinafter referred to as “candidate image”). ) Is specified (S602).

On the other hand, when the distance information of the pixels is less than the first threshold value α1 (“No” in S601), the image pickup apparatus 10 (moving object determination unit 16) specifies the subject image corresponding to the pixels as the main subject image (S603). ..

The image pickup apparatus 10 (moving object determination unit 16) identifies the candidate image and the main subject image for all the pixels in the main region (S604).

When each of the candidate image and the main subject image is specified for all the pixels in the main area (“Yes” in S604), the image pickup apparatus 10 (moving object determination unit 16) is in a continuous candidate image area (hereinafter, “candidate”). The area of "image area") is calculated (S605). The candidate image region is an example of a moving body image region in the present invention.

Next, the image pickup apparatus 10 (moving object determination unit 16) compares the calculated area with the second threshold value α2 (S606). The "second threshold value α2" is a threshold value used for determining the presence or absence of a moving object. The second threshold value α2 is stored in advance in a storage means (semiconductor memory), a storage unit 12, or the like of the moving object determination unit 16.

When the calculated area is equal to or less than the second threshold value α2 (“No” in S606), the imaging device 10 (moving object determination unit 16) determines that the candidate image region is not a moving object image (no moving object) (S607).

Next, when the moving body determination unit 16 determines that there is no moving object (S607), the imaging device 10 (image acquisition unit 17) acquires an image captured by the moving object determination unit 16 as a specific captured image. (S608). The specific captured image is stored in the storage unit 12 (S609).

On the other hand, when the calculated area is larger than the second threshold value α2 (“Yes” in S606), the image pickup apparatus 10 (moving object determination unit 16) determines that the candidate image region is a moving object image (there is a moving object) (S610). ).

Next, when the imaging device 10 (moving object determination unit 16) determines that there is a moving object (S610), the imaging device 10 (moving object determination unit 16) generates moving object information (S611). At this time, the image pickup device 10 (image acquisition unit 17) does not acquire the captured image.

The "moving object information" is information indicating that there is a moving object between the imaging unit 14 and the shelf Sc (commodity). That is, the moving object information is information indicating that the captured image includes the moving object image. The moving object information is stored in the storage unit 12 in association with the information regarding the time when the captured image is generated (s612).

Next, the image pickup device 10 (communication unit 11) transmits each of the specific captured image and the moving object information to the server 20 (S613). The server 20 stores each of the received specific captured image and moving object information in the storage unit 22 (S614).

The imaging device in the present invention may transmit the specific captured image to the server each time it acquires (generates moving object information), or the specific captured image (generated) acquired at each second time interval described later. You may send only the moving object information) to the server.

Return to FIG.
Next, the system 1 executes the shortage determination process (S7).

FIG. 10 is a flowchart of the shortage determination process (S7).

The “out-of-stock determination process (S7)” is a process for determining the presence or absence of a missing item based on a specific captured image (an image taken in which a moving object is not captured) acquired at every second time interval.

The "second time interval" is a time interval defined for extracting a specific captured image used by the shortage determination unit 24 to determine the presence or absence of a missing item from the specific captured images. That is, for example, when the second time interval is 5 minutes, the shortage determination unit 24 determines whether or not there is a shortage in a specific captured image (first specific captured image), and then the first specific captured image. It is determined whether or not there is a shortage in the specific captured image (second specific captured image) acquired 5 minutes after the acquisition. In this case, the second specific captured image is a specific captured image acquired by the image acquisition unit 17 when 5 minutes have passed from the time when the first specially captured image was acquired by the image acquisition unit 17. The second time interval is counted by, for example, a counter provided in the control unit 23.

The second time interval is, for example, product information (type, sales history, sales forecast, etc.), product advertisement, external factors (season, weather, temperature, humidity, etc.), and a time zone in which the imaging unit 14 images the product. It is determined by the user Us based on (time zone when customer Cu is high, time zone when customer Cu is low, etc.). In other words, the second time interval is determined according to the time interval in which the product is expected to be out of stock (the product is purchased). That is, the second time interval is determined based on the product.

First, the server 20 (out of stock determination unit 24) reads out the information received from the image pickup apparatus 10 and stored in the storage unit 22 (S701). Here, when the information transmitted from the image pickup apparatus 10 is the information acquired / generated at each second time interval, the server 20 reads out the information stored in the storage unit 22 in chronological order. On the other hand, when the information transmitted from the image pickup apparatus 10 is the information acquired / generated at each first time interval, the server 20 acquires the information stored in the storage unit 22 at each second time interval. -Read the generated information in chronological order.

Next, the server 20 (out of stock determination unit 24) identifies whether the read information is the specific captured image or the moving object information (S702).

When the read information is a specific captured image (“specific captured image” in S702), the server 20 (out of stock determination unit 24) performs a known image recognition process (for example, SSD: Single Shot MultiBox Detector) on the specific captured image. This is executed to determine whether or not the product captured in the specific captured image is out of stock (S703). In the process (S703), the specific captured image used for determining the presence or absence of a product shortage is the second specific captured image in the present invention.

When the shortage determination unit 24 determines that there is a shortage (“Yes” in S703), the server 20 (outage information generation unit 25) generates outage information (S704).

FIG. 11 is a schematic diagram showing an example of out-of-stock information.

"Out of stock information" is information on product shortage. The figure shows that "product name", "package information", "location information", and "time information" are associated as out-of-stock information. The "product name" is the name of the product that is out of stock. "Package information" is information on the package of a product that is out of stock (packaging photo, product appearance photo, etc.). The "position information" is information regarding the position where the missing product is displayed (the number / position of the shelf Sc, the display position of the product on the shelf, etc.). The "time information" is information regarding the time when the specific captured image determined to be out of stock was acquired.

Return to FIG.
In the process (S702), when the read information is moving object information (“moving object information” in S702), the server 20 (out of stock information generation unit 25) determines whether or not there is a missing item because there is a moving object moving in the passage. Information indicating that the item is not out of stock (out of stock undetermined information) is generated (S705).

Next, the server 20 (out-of-stock information generation unit 25) stores the generated out-of-stock information or out-of-stock undetermined information in the storage unit 22 (S706).

Next, the server 20 (communication unit 21) transmits the shortage information or the shortage undetermined information to the information processing terminal 30 (S707).

The information processing terminal 30 stores the received shortage information or shortage undetermined information in the storage unit 32 (S708). The information processing terminal 30 (display unit 34) displays information on products that are out of stock based on out-of-stock information, and provides information indicating that the presence or absence of out-of-stock is not determined based on out-of-stock undetermined information. Display (S709).

FIG. 12 is a schematic diagram showing an example of information displayed on the information processing terminal 30.
In the figure, the information processing terminal 30 has information on the display unit 34 (at time "T1", there is a shortage in the product "A" package information "F001" of "x1, y1" on the shelf "001". T2 ”indicates that it is not determined whether or not the shelf“ 001 ”is out of stock.).

Return to FIG.
On the other hand, when the shortage determination unit 24 determines that there is no shortage (“None” in S703), the server 20 ends the shortage determination process (S7).

As described above, in the present system 1, the information processing program according to the present invention (hereinafter referred to as “the program”) operates, and the present program is used as the hardware resources of the image pickup apparatus 10, the server 20, and the information processing terminal 30. In collaboration with each other, the information processing method according to the present invention (hereinafter referred to as "the present method") is realized. That is, this program causes the image pickup apparatus 10, the server 20, and the information processing terminal 30 to function as the system 1.

● Summary According to the embodiment described above, the system 1 identifies an captured image in which a moving object is not captured as a specific captured image, and determines whether or not a product is out of stock based on the specific captured image. Therefore, this system 1 does not require a moving object removal process that requires a huge processing time and processing cost unlike a conventional information processing system. In addition, the system 1 acquires only a specific captured image from among a large number of captured images, and determines whether or not the product is out of stock. Therefore, the present system 1 can reduce the processing time and the processing cost as compared with the conventional information processing system that acquires a large number of captured images and performs image processing on each captured image. As a result, the system 1 can reliably determine the presence or absence of a product shortage while suppressing the processing time and processing cost based on the specific captured image in which the product image is not hidden by the moving object image. That is, the system 1 can reduce the processing time and the processing cost in the information processing for determining the shortage (out of stock) of the product.

Further, according to the embodiment described above, the system 1 acquires distance information based on the image signal from the image sensor 141 of the image plane phase difference method. Therefore, the system 1 has finished determining the presence or absence of a moving object based on the distance information when acquiring the specific captured image. That is, the system 1 makes it possible to determine the presence or absence of a moving object by itself for each captured image generated. That is, unlike the conventional information processing system, this system 1 does not require a plurality of captured images for determining the presence or absence of a moving object. As a result, the system 1 can reduce the processing time and the processing cost in the pre-stage of information processing for determining the shortage of the product.

Further, according to the embodiment described above, when it is determined that there is a moving object, the system 1 generates moving object information and does not acquire a captured image. Therefore, when it is determined that there is a moving object, the system 1 does not execute the image recognition process (S703) for determining the presence or absence of the missing item in the missing item determination process (S7). As a result, the system 1 does not execute the calculation on the captured image (non-specific captured image) that may not be able to determine the missing item because the moving object is imaged. That is, the system 1 does not execute unnecessary information processing on the non-specific captured image, and can suppress the power consumption (consumption cost) of the shortage determination unit 24. Further, the system 1 can notify the user Us of information to that effect when it is not possible to determine the presence or absence of a missing item due to the presence of a moving object by generating moving object information.

Furthermore, according to the embodiment described above, in the system 1, a second time interval has elapsed from the time when the specific captured image (first specified captured image) for which the presence or absence of the missing item was previously determined was acquired. Based on the specific captured image (second specific captured image) acquired at the time of the operation, the presence or absence of a missing item is determined. That is, the system 1 determines whether or not there is a shortage based on the specific captured images acquired at each second time interval instead of all the captured images. Therefore, this system 1 can suppress the number of specific captured images used for determining the presence or absence of missing items. As a result, the system 1 can reduce the processing time and the processing cost in the information processing for determining the shortage of the product.

Furthermore, according to the embodiment described above, the second time interval is determined based on the product and the time zone in which the imaging unit 14 images the product. That is, for example, the second time interval of the product that matches the season (the number of sales is large) is determined to be shorter than the second time interval of the product that does not match the season (the number of sales is small). As a result, the system 1 can increase the chance of determining that there is a shortage in the shortage determination process (S7) and reduce the chance of determining that there is no shortage. That is, the system 1 can efficiently determine the presence or absence of a missing item for each specific captured image. That is, the system 1 can reduce the processing time and the processing cost in the information processing for determining the shortage of the product.

Furthermore, according to the embodiment described above, the second time interval is longer than the first time interval. That is, the system 1 uses only the specific captured images acquired at each second time interval among the large number of specific captured images that can be generated and acquired at the first time interval to determine the presence or absence of missing items. .. That is, the system 1 uses a part of the specific captured image acquired at the first time interval at the shortest to determine the presence or absence of a missing item. As a result, the system 1 can reduce the processing time and the processing cost in the information processing for determining the shortage of the product.

Furthermore, according to the embodiment described above, the system 1 identifies a moving body image region of a moving body image based on a change in distance information, and determines the presence or absence of a moving body based on the area of the moving body image region. To do. Specifically, the system 1 determines that there is a moving body when the area of the moving body image region is larger than the second threshold value α2. As a result, for example, the system 1 can effectively use an captured image in which an object (for example, an insect, a human hand, etc.) having a size that does not hide the entire product is captured as a specific captured image. Yes, the captured image in which an object (cart C, customer Cu, etc.) of a size that hides the entire product is captured is not acquired (used). In other words, the system 1 uses only the captured image capable of determining whether or not the product is out of stock as the specific captured image. As a result, the system 1 can reduce the processing time and the processing cost in the information processing for determining the shortage of the product.

Furthermore, according to the embodiment described above, when it is determined that there is a shortage, the system 1 generates out-of-stock information and displays it on the information processing terminal 30. As a result, the user Us who uses the information processing terminal 30 can grasp the presence or absence of a shortage in real time to some extent, and can replenish or rearrange the products in a timely manner.

In the embodiment described above, the distance information acquisition unit 15 acquires distance information based on the image signal from the image sensor 141 of the image plane phase difference method. Instead of this, the distance information acquisition unit in the present invention outputs from a sensor (for example, an ultrasonic sensor, a radio wave sensor, etc.) or a device (for example, a three-dimensional camera or a stereovision type camera) different from the image sensor. Distance information may be acquired based on the signal. That is, for example, the distance information acquisition unit may be an ultrasonic sensor or a radio wave sensor that acquires distance information based on the phase difference between the transmitted signal and the received signal. In this case, the image acquisition unit does not acquire the captured image when the sensor detects a moving object (when the moving object determination unit determines that there is a moving object), and when the sensor does not detect the moving object (moving object determination). (When it is determined by the unit that there is no moving object) The captured image is acquired as a specific captured image.

In addition, this system executes a known image recognition process on the captured image generated by the imaging unit, calculates the area occupied by the moving object image in the captured image, and among the captured images generated within a predetermined time interval. The presence or absence of a product shortage may be determined based on the captured image having the smallest area of the same region. In this case, the moving body determination unit in the present invention executes a known image recognition process on the captured image generated by the imaging unit, calculates the area occupied by the moving body image in the captured image, and based on the area of the same area, The presence or absence of a moving body may be determined. That is, for example, when the area of the same region is equal to or less than the second threshold value, the moving body determination unit may determine that there is no moving body.

Further, the shortage determination unit in the present invention may determine the presence or absence of a shortage based on the specific captured image acquired after the time when the moving object information stored in the storage unit is generated. In other words, the shortage determination unit may determine the presence or absence of a shortage based on the specific captured image acquired after the moving object (customer) moves from the front of the shelf. Missing items can only occur after the moving object has moved from the front of the shelf. Therefore, this system can efficiently determine the presence or absence of a missing item based on a specific captured image in which the product is likely to be out of stock.

Furthermore, the image acquisition unit in the present invention acquires a non-specific captured image in which the moving body is captured when the moving body determination unit determines that the moving body is present, and the moving body image region and the product image in the non-specific captured image. The area may be specified. The “moving body image region” is a region occupied by a moving body image of a moving body in a non-specific captured image. That is, the moving body image region is a region in which it is impossible to determine whether or not a product is out of stock due to the presence of the moving body. The "product image area" is an area occupied by the product image of the product in the non-specific captured image. That is, the product image area is an area in which it is possible to determine whether or not a product is out of stock. In this case, the shortage determination unit may determine the presence or absence of a shortage based on the region image corresponding to the product image region in the non-specific captured image. In other words, the shortage determination unit may determine the presence or absence of a shortage by using a region image in which a moving object is not captured among the non-specific captured images. According to this configuration, the system can determine whether or not a product is out of stock based on the area image even when the specific captured image cannot be acquired at each second time interval.

Furthermore, the shortage determination unit may determine the presence or absence of shortages based on a plurality of region images corresponding to different product image regions. According to this configuration, the system can efficiently use the area image even when the specific captured image cannot be acquired, and can reduce the processing time and the processing cost in the information processing for determining the product shortage. Is.

Furthermore, the shortage determination unit in the present invention is used during the third time interval when the specific captured image is not acquired between the time when the second time interval elapses and the time when the third time interval elapses. The presence or absence of a product shortage may be determined based on the acquired region image of the non-specific captured image. The "third time interval" is a threshold value for the shortage determination unit to determine whether or not to use the region image. According to this configuration, the shortage determination unit can efficiently use each of the specific captured image and the region image with the third time interval as a threshold value. Therefore, this system can reduce the processing time and the processing cost in the information processing for determining the shortage of the product.

Furthermore, the shortage determination unit in the present invention is among the non-specific captured images acquired during the third time interval when the specific captured image is not acquired until the third time interval elapses. The presence or absence of a product shortage may be determined based on the region image of the non-specific captured image having the largest region image. The size of the region image may be calculated based on the output signal from the image sensor, or may be calculated based on known image processing. According to this configuration, the shortage determination unit can determine the presence or absence of shortage of as many products as possible based on the area image.

Furthermore, in the embodiment described above, the server 20 includes the shortage determination unit 24. Instead of this, the image pickup apparatus in the present invention may include a shortage determination unit. In other words, the image pickup apparatus may function as the information processing apparatus according to the present invention. In this case, the present program may operate in the image pickup apparatus, and the present program may cooperate with the hardware resources of the image pickup apparatus to realize the present method. That is, the present program may cause the image pickup apparatus to function as the information processing apparatus according to the present invention.

Furthermore, the relationship between the displayed product and the imaging device is not limited to the present embodiment. That is, for example, the products may be displayed in a state of being stacked on the table, and the image pickup device may image the products on the table from above.

Furthermore, the embodiments described above do not unreasonably limit the content of the invention described in the claims. Moreover, not all of the configurations described in the above-described embodiments are essential configuration requirements of the present disclosure.

1 Information processing system 10 Imaging device 12 Storage unit 14 Imaging unit 15 Distance information acquisition unit 16 Moving object determination unit 17 Image acquisition unit 20 Server 22 Storage unit 24 Out of stock determination unit 25 Out of stock information generation unit 30 Information processing terminal 34 Display unit

Claims (22)

It is an information processing system that determines the shortage of products displayed in the store.
An imaging unit that generates an captured image in which the product is captured, and an imaging unit.
A distance information acquisition unit that acquires distance information indicating a distance between the image pickup unit and a subject imaged in the captured image.
Based on the distance information, a moving object determination unit that determines the presence or absence of a moving object moving in the space between the imaging unit and the product, and
An image acquisition unit that acquires the captured image in which the moving body is not captured as a specific captured image based on the determination result of the moving body determination unit.
A storage unit that stores the specific captured image and
A shortage determination unit that determines the presence or absence of the shortage based on the specific captured image,
Have
An information processing system characterized by this. The imaging unit
Image sensor that generates image signals,
With
The distance information acquisition unit acquires the distance information based on the image signal.
The information processing system according to claim 1. The image sensor is
Image plane phase difference type image sensor,
Including
The distance information acquisition unit acquires the distance information based on the image signal from the image sensor of the image plane phase difference method.
The information processing system according to claim 2. The distance information acquisition unit acquires the distance information based on the phase difference between the transmitted signal and the received signal.
When the moving body determination unit determines that there is no moving body, the image acquisition unit acquires the captured image as the specific captured image.
The information processing system according to claim 1. When the moving body determination unit determines that the moving body is present, the moving body determining unit generates moving body information indicating that the moving body is present.
The image acquisition unit does not acquire the captured image when the moving object determination unit determines that the moving object is present.
The information processing system according to claim 1. The storage unit stores the moving object information and stores the moving body information.
The shortage determination unit determines the presence or absence of the shortage based on the specific captured image acquired after the time when the moving object information stored in the storage unit is generated.
The information processing system according to claim 5. The imaging unit generates the captured image at each first time interval, and generates the captured image.
The specific captured image is
The first specific captured image and
The second specific captured image acquired after the second time interval has elapsed from the time when the first specific captured image was acquired, and the second specific captured image.
Including
The shortage determination unit determines the presence or absence of the shortage based on the second specific captured image.
The information processing system according to claim 1. The second time interval is determined based on the goods.
The information processing system according to claim 7. The second time interval is determined based on the time zone in which the imaging unit images the product.
The information processing system according to claim 7. The second time interval is longer than the first time interval.
The information processing system according to claim 7. When the image acquisition unit determines that the moving object is present by the moving object determination unit,
While acquiring a non-specific captured image in which the moving object is captured,
In the non-specific captured image, a moving body image region, which is a region of the moving body image of the moving body, and a product image region, which is a region of the product image of the product, are specified.
The storage unit stores the non-specific captured image and stores the non-specific captured image.
The shortage determination unit determines the presence or absence of the shortage based on the region image corresponding to the product image region in the non-specific captured image.
The information processing system according to claim 7. When the second specific captured image is not generated between the time when the second time interval elapses and the time when the predetermined third time interval elapses.
The shortage determination unit determines the presence or absence of the shortage based on the region image of the non-specific captured image acquired during the third time interval.
The information processing system according to claim 11. The shortage determination unit is based on the region image of the non-specific captured image in which the region image is the largest among the non-specific captured images acquired during the third time interval. Judge the presence or absence,
The information processing system according to claim 11. The shortage determination unit determines the presence or absence of the shortage based on the plurality of the area images.
The product image regions corresponding to the plurality of region images are different.
The information processing system according to any one of claims 11 to 13. The moving body determination unit determines the presence or absence of the moving body based on the change in the distance information.
The information processing system according to claim 1. The moving body determination unit
Based on the distance information, the moving body image region of the moving body image of the moving body in the captured image is specified.
The presence or absence of the moving body is determined based on the moving body image region.
The information processing system according to claim 15. The moving body determination unit compares the area of the moving body image region with a predetermined threshold value, and when the area is larger than the threshold value, determines that the moving body is present.
The information processing system according to claim 16. When the shortage determination unit determines that there is a shortage, the shortage information generation unit that generates the shortage information, which is information about the shortage,
Based on the shortage information, a display unit that displays information on the product with the shortage, and a display unit.
Have
The information processing system according to claim 1. When the moving body determination unit determines that the moving body is present, the moving body determining unit generates moving body information indicating that the moving body is present.
Based on the moving body information, the display unit displays information indicating that the presence or absence of the missing item is not determined due to the presence of the moving body.
The information processing system according to claim 18. An information processing device that determines the shortage of products displayed in a store.
An imaging unit that generates an captured image in which the product is captured, and an imaging unit.
A distance information acquisition unit that acquires distance information indicating a distance between the image pickup unit and a subject imaged in the captured image.
Based on the distance information, a moving object determination unit that determines the presence or absence of a moving object moving in the space between the imaging unit and the product, and
An image acquisition unit that acquires the captured image in which the moving body is not captured as a specific captured image based on the determination result of the moving body determination unit.
A storage unit that stores the specific captured image and
A shortage determination unit that determines the presence or absence of the shortage based on the specific captured image,
Have
An information processing device characterized by this. The computer functions as the information processing device according to claim 20.
An information processing program characterized by this. An information processing method executed by an information processing system that determines the shortage of products displayed in a store.
The information processing system
An imaging unit that generates an captured image in which the product is captured, and an imaging unit.
A storage unit that stores the captured image as a specific captured image in which a moving object moving in the space between the imaging unit and the product is not captured.
With
The information processing system
A process of acquiring distance information indicating a distance between the image pickup unit and a subject captured in the captured image.
A process of determining the presence or absence of the moving object moving in the space between the imaging unit and the product based on the distance information.
Based on the determination result of the presence or absence of the moving body, the process of acquiring the captured image in which the moving body is not captured as the specific captured image, and
A process of determining the presence or absence of the missing item based on the specific captured image,
Have
An information processing method characterized by this.

Images