JP2022040557A - Information processing device, information processing system, information processing program, and information processing method - Google Patents

Abstract

To provide an information processing device in which accuracy of merchandise management is improved.SOLUTION: An information processing device includes: a storage unit 22 which stores a plurality of focus position patterns for specifying combinations of individual areas in which focus positions are set among a plurality of individual areas; an acquisition unit 231 which acquires a photographed image as a reference image obtained by photographing a merchandise shelf; an information specifying unit 237 which specifies merchandise information corresponding to a merchandise code on the basis of a plurality of focus change images; and a photographing device control unit. The photographing device control unit is composed of an area specifying unit 232 which specifies an area of the merchandise shelf, a detection unit 233 which detects a plurality of specific areas, a selection unit 234 which selects one focus position pattern as a selection pattern according to a detection result of the detection unit, a focus position setting unit 235 which sets a focus position for each of the plurality of individual areas contained in the selection pattern on the basis of the specific areas, and a signal generation unit 236 which generates a control signal for allowing a photographing device to photograph a plurality of focus change images in which respective focus positions are focused.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing apparatus, an information processing system, an information processing program, and an information processing method.

If there is a shortage (out of stock) of displayed products in sales floors such as supermarkets and convenience stores (convenience stores), sales opportunities for the products will be lost and sales in the sales floor will decrease. Therefore, the clerk at the sales floor must regularly monitor the products, and if the products are in short supply, promptly perform the work of replenishing the shortage of products and eliminating the shortage of products.

In addition, when displaying products on the product shelves at the same sales floor, the allocation of products to the product shelves (shelf allocation) is an important factor that influences the sales of the products. Therefore, shelving allocation management such as creating shelf allocations at the sales floor, monitoring whether products are displayed according to the shelving allocation, and changing the shelving allocation according to sales and seasons is often performed at the sales floor. ..

In product management such as product shortages and product shelving allocation, a technique for managing products using an image taken from an image of a product shelf on which products are displayed has been proposed (for example, Patent Document 1). reference).

The technology disclosed in Patent Document 1 includes product identification information of products displayed on the product shelf and product identification information of the price tag based on an image taken by imaging a product shelf with a price tag (product tag). , And compare the two product identification information to determine the display status of the product (for example, whether the product is on display or out of stock). As a result, the technology can accurately determine the display status of the products displayed on the product shelves.

However, the technique presupposes that all the products and the price tag are captured in the captured image in a state where the product identification information can be acquired. Therefore, when the price tag of some products cannot be detected in the captured image, the technique generates erroneous shelving allocation information and / or out-of-stock information. That is, the technique can generate inaccurate shelving allocation information and / or out-of-stock information. As a result, product management using this technology may lack accuracy.

Further, the focal point of the captured image captured by a general image pickup device tends to shift as it approaches the outer edge portion of the captured image, and the image at the outer edge portion tends to be unclear and have low contrast. Further, in the product shelf, the face surface of the product is arranged so as to be aligned with the front of the shelf board (price rail) of the product shelf. Therefore, the front surface of the product shelf tends to have a uniform texture. As a result, even if focusing (focusing) by the autofocus function is used, it is difficult to focus on the product and the product tag in all areas of the captured image.

As described above, in the captured image of the product shelf, it is extremely unlikely that all the price tags are captured in an identifiable state. Therefore, a highly accurate product management technique that improves the recognition accuracy of the price tag captured in the captured image is desired.

Japanese Unexamined Patent Publication No. 2019-185648

An object of the present invention is to provide an information processing apparatus, an information processing system, an information processing program, and an information processing method capable of improving the accuracy of product management.

The information processing device according to the present invention is an information processing device that specifies product information of a product based on a plurality of captured images captured by an image pickup device that captures an image of the product shelf on which the product is displayed. , A display tag displaying the product code corresponding to the product information of the product is attached, and the area of the product shelf captured in the captured image includes a plurality of individual areas, and the plurality of captured images have a plurality of different focal positions. A storage unit that stores a plurality of focus position patterns that specify a combination of individual areas in which a focus position is set, and an image captured by an image pickup device that captures a product shelf, including a focus change image of the above. The product code is based on a plurality of focus change images captured by an acquisition unit acquired as a reference image, an image pickup device control unit that controls the operation of the image pickup device, and an image pickup device whose operation is controlled based on a control signal. It has an information specifying unit that specifies the corresponding product information, and the image pickup device control unit is imaged in the area specifying unit that specifies the area of the product shelf captured in the reference image and the area of the product shelf. Among the detection unit that detects a plurality of specific areas including the product code and the plurality of focal position patterns stored in the storage unit, one focal position pattern according to the detection result of the detection unit is selected as the selection pattern. Based on the selection unit to be selected and the specific area detected by the detection unit, the focus position setting unit that sets the focus position for each of the plurality of individual regions included in the selection pattern and the focus positions are focused on each of the plurality of focus positions. It is characterized by including a signal generation unit for generating a control signal for causing an image pickup apparatus to capture a plurality of focus change images.

According to the present invention, the accuracy of product management can be improved.

It is a network block diagram which shows the embodiment of the information processing system which concerns on this invention. It is a functional block diagram of the image pickup apparatus provided in the information processing system of FIG. It is a schematic diagram which shows the example of the image captured by the image pickup apparatus of FIG. It is a schematic diagram which shows the display example of the display tag which is imaged in the captured image of FIG. It is a functional block diagram which shows the embodiment of the information processing apparatus which concerns on this invention. It is a flowchart which shows the embodiment of the information processing method which concerns on this invention. It is a flowchart of the determination process included in the information processing method of FIG. It is a schematic diagram which shows the example of the reference image after the process of specifying the product shelf area included in the determination process of FIG. 7. It is a schematic diagram which shows the example of the reference image after the process of detecting the display tag included in the determination process of FIG. 7. It is a schematic diagram which shows the example of the reference image after the process of detecting the display tag area included in the determination process of FIG. 7. It is a schematic diagram which shows the example of the reference image after the process which detects the set of the product code included in the determination process of FIG. 7. It is a flowchart of the control signal generation processing included in the information processing method of FIG. It is a schematic diagram which shows the example of the focal position pattern selected in the control signal generation processing of FIG. It is a schematic diagram which shows the example of the reference image after the process of selecting the selection pattern included in the control signal generation process of FIG. It is a flowchart of the 1st focus position setting process included in the control signal generation process of FIG. It is a flowchart of the 2nd focal point position setting process included in the control signal generation process of FIG. It is a flowchart of the 3rd focal point position setting process included in the control signal generation process of FIG. It is a flowchart of the focus change image imaging process included in the information processing method of FIG. It is a flowchart of the product information integration processing included in the information processing method of FIG.

An embodiment of the information processing apparatus, the information processing system, the information processing program, and the information processing method according to the present invention will be described with reference to the following embodiments and drawings.

● Information processing system ●
The information processing system according to the present invention (hereinafter referred to as "the present system") specifies the product information of the product based on the captured image of the product shelf on which the product is displayed. Specifically, the system selects a focal position pattern based on the captured image, sets the focal position corresponding to the selected focal position pattern, controls the operation of the image pickup device, and changes a plurality of focal points. Get an image. This system identifies the product information corresponding to the product code attached to the product shelf based on the focus change image.

A "commodity" is an item that is displayed in a closed space and purchased by a customer. The products are displayed in a state of being arranged (or stacked) according to the types of products on the product shelves installed in the closed space.

The "closed space" is a three-dimensional space partitioned by an arbitrary range, and is a space in which the product information of the product can be specified by this system. The closed space includes a space that is physically partitioned and a space that is not physically partitioned. That is, for example, a closed space includes an internal space of a sales floor such as a supermarket or a convenience store, an internal space of each sales floor of a shopping mall or an airport duty-free shop where many sales floors are connected, and an outdoor space such as an outdoor sales floor of a sales floor. ..

A "product shelf" is an object that has a display space for displaying products and is installed in a sales floor. A display tag, which will be described later, is attached to the product shelf, and products corresponding to the display tag are displayed. Product shelves include display shelves that display products on horizontal shelves, display shelves that display products on top plates, work shelves (work tables) for working on top plates, cardboard, etc. Includes simple shelves, multi-tiered and flat refrigerated / frozen showcases, etc. The product shelf includes an outer frame composed of a base, a support, a top plate, a back plate, and the like, and a shelf plate arranged in an area surrounded by the outer frame.

The "captured image" is image information obtained by capturing a product shelf, a product displayed on the product shelf, and a display tag corresponding to the product. That is, the captured image includes the area of the product shelf.

The “product shelf area” is an area occupied by one or a group of product shelves (area surrounded by an outer frame or a shelf board of the product shelf) among all the areas included in the captured image. The product shelf area (hereinafter referred to as “product shelf area”) includes a plurality of individual areas.

The "individual area" is an area occupied by each of the small areas when the product shelf area is divided into a plurality of small areas. The individual area includes at least a part of the product shelf area, and may include a part of the area other than the product shelf (adjacent product shelf captured in the captured image, aisle, ceiling / wall area). The size (area) of each individual area may be the same or different. In the present embodiment, the number of individual areas included in the product shelf area is "9".

The "display tag" displays the product code and the information required by the customer (hereinafter referred to as "customer product information") among the product information. The display tag is, for example, a paper price tag on which a product code and customer product information are printed. The display tag is the place where the price tag is generally attached, such as the front of the product shelf (for example, the front of the shelf board, the price rail attached to the front of the shelf, the upper part of the suspended product, etc.) for each display range of the corresponding product. Is placed in). That is, the display tag exists for each product displayed on the product shelf.

The display tag may be, for example, an electronic price tag (electronic shelf tag) composed of an information display medium such as an electronic paper, a liquid crystal display, or an organic EL (Electro-Luminescence) display.

The "product code" is a one-dimensional or two-dimensional identifier defined for each product in order to identify the product. The product code includes, for example, a one-dimensional code such as a bar code and a two-dimensional code such as a QR code (registered trademark).

"Product information" is information defined for each product such as product name (product name), price, manufacturer name, size (capacity), weight, date of manufacture, expiration date, etc., and is JAN. (Japanese Article Number) Includes product identification code such as code.

The "focus position pattern" is information that identifies a combination of individual areas in which a focus position is set among a plurality of individual areas. The focal position pattern includes a combination of all individual regions and a combination of some individual regions among a plurality of individual regions.

The "focus position" is a position within the angle of view of the image pickup device that is focused when the focus change image is captured. The focal position is set for each individual region constituting the focal position pattern. That is, the focal position is set to the coordinates of the pixels constituting the individual region, which correspond to the positions to be focused at the time of capturing the focus change image.

The "focus change image" is an image taken by focusing on each focus position for each individual region based on the focus change pattern in which the focus position is set. That is, the captured image includes a plurality of focus change images. Also, the focal position of each focus change image is different.

● Configuration of Information Processing System FIG. 1 is a network configuration diagram showing an embodiment of this system.

The system 1 includes a plurality of image pickup devices 10 (1), ..., 10 (n) (n is an integer) and an information processing device 20. In the following description, the plurality of image pickup devices 10 (1), ..., 10 (n) are referred to as "imaging" unless the respective image pickup devices 10 (1), ..., 10 (n) are particularly distinguished. Notated as "device 10". The information processing apparatus 20 is an information processing apparatus according to the present invention (hereinafter referred to as “the present apparatus”). In the following description, the information processing apparatus 20 is also referred to as the present apparatus 20.

The image pickup apparatus 10 and the present apparatus 20 are connected via a network N (communication line) that uses a wired communication method or a wireless communication method. The network N is, for example, a communication network such as the Internet, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), and a Wi-Fi (registered trademark).

● Configuration of the image pickup device FIG. 2 is a functional block diagram of the image pickup device 10.

The image pickup device 10 takes an image of the product shelf S (see FIG. 3; the same applies hereinafter) to generate an image. The image pickup device 10 is, for example, a camera unit that captures a still image of a product shelf S in a sales floor at predetermined time intervals. Each image pickup device 10 is, for example, in a position (for example, the ceiling of the sales floor) where the entire image of the front portion (front surface) of the corresponding product shelf S and the products displayed on the product shelf S can be imaged in the sales floor. It is placed on the surface or wall surface). In the present embodiment, the image pickup apparatus 10 captures the captured image by aligning the focal position with the center of the angle of view, except when capturing the focus change image. The image pickup apparatus 10 includes a communication unit 11, a storage unit 12, a control unit 13, and an image pickup unit 14.

The "predetermined time interval" is a time interval in which the image pickup apparatus 10 periodically images the product shelf S, and is, for example, 15 seconds.

The image pickup device may be a terminal device having a camera function including an image pickup unit. That is, for example, the image pickup device may be a smart phone having a camera function, a mobile phone, a tablet terminal, a laptop computer, or a digital camera having a communication function.

Further, the image pickup device may be a camera unit that captures a moving image, or may be a camera unit that captures a still image and a moving image.

FIG. 3 is a schematic diagram showing an example of a captured image.
The figure shows a state in which the product shelf S is imaged from the front. The figure shows that 20 kinds of products are displayed on the product shelf S, and 20 display tags P (1 to 20) corresponding to each product are attached. The figure shows that the display tag P (1 to 20) is arranged at the lower right (lower left in the front view) of the display range of the corresponding product. In the following description, the display tags P (1 to 20) are referred to as "display tags P" unless the display tags P (1 to 20) are particularly distinguished.

The direction in which the product shelf is imaged by the captured image is not limited to the front (front). That is, for example, when the product shelf is imaged by an image pickup device arranged on the ceiling of the sales floor, the same direction may be forward and diagonally upward, or diagonally forward and diagonally upward.

FIG. 4 is a schematic diagram showing a display example of the display tag P.
The figure shows that the customer product information (“product name” and “amount”) and two product codes are displayed on the display tag P. The figure shows that the product code is displayed in the area to the right and the customer product information is displayed in the remaining area in the front view. In addition, the figure shows that a two-dimensional code in which a Hamming code is stylized is displayed as a product code.

The product code is not limited to this embodiment. That is, for example, the product code may be a two-dimensional code in which the Reed-Solomon code is stylized, a two-dimensional code in which the two-dimensional parity (Parity) is stylized, or a QR code (registered trademark). ..

Further, the two product codes may be composed of the same type of two-dimensional code, or may be composed of different types of two-dimensional codes, one-dimensional code and two-dimensional code.

Further, the display tag may display only one product code.

Return to FIG.
The communication unit 11 communicates with the apparatus 20 via the network N. The communication unit 11 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 12 stores an image captured by the image pickup unit 14 and information necessary for the operation of the image pickup device 10 (for example, a control signal described later). The storage unit 12 is composed of, for example, a recording device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or the like.

The control unit 13 controls the operation of the entire image pickup apparatus 10. The control unit 13 is, for example, a processor such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like. It is composed of an integrated circuit and a semiconductor memory.

The image pickup unit 14 images the product shelf S, the products displayed on the product shelf S, and the display tag P attached to the product shelf S in the sales floor, and captures the product shelf S, the product, and the display tag P. And to generate an image taken. The image pickup unit 14 is composed of, for example, a lens group, a shutter, an image pickup element, an A / D converter, and the like. The captured image generated by the imaging unit 14 is stored in the storage unit 12.

● Configuration of Information Processing Device FIG. 5 is a functional block diagram showing an embodiment of the device 20.

The apparatus 20 selects a focal position pattern based on the captured image, sets the focal position corresponding to the selected focal position pattern, controls the operation of the imaging device 10, and acquires a plurality of focus change images. do. Further, the present device 20 specifies the product information of the product based on the plurality of focus change images. The present device 20 is, for example, a personal computer such as a server. The apparatus 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

In the present device 20, the information processing program according to the present invention (hereinafter referred to as “the present program”) operates, and the present program cooperates with the hardware resources of the present device 20 to realize the present method described later. .. That is, this program causes the computer to function as the present device 20.

The communication unit 21 communicates with the image pickup apparatus 10 via the network N. The communication unit 21 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 22 stores information necessary for the operation of the apparatus 20 (for example, a reference image, a focus change image, a focus position pattern, etc., which will be described later). The storage unit 12 is composed of, for example, a recording device such as an HDD or an SSD, a semiconductor memory element such as a RAM or a flash memory, or the like.

The control unit 23 executes information processing described later and controls the operation of the entire apparatus 20. The control unit 23 is composed of, for example, a processor such as a CPU, MPU, or DSP, an integrated circuit such as an ASIC or FPGA, and a semiconductor memory. The control unit 23 includes an acquisition unit 231, an area identification unit 232, a detection unit 233, a selection unit 234, a focus position setting unit 235, a signal generation unit 236, and an information identification unit 237.

The acquisition unit 231 acquires an image captured by the image pickup device 10 as a reference image. The specific operation of the acquisition unit 231 and the reference image will be described later.

The area specifying unit 232 identifies the product shelf area captured in the reference image. The specific operation of the area specifying unit 232 will be described later.

The detection unit 233 detects a plurality of specific areas imaged in the product shelf area. The specific operation of the detection unit 233 will be described later.

The "specific area" is all or a part of the area in which the display tag P can be imaged (hereinafter referred to as "display tag area") in the product shelf area. That is, the specific area is all or a part of the area to which the display tag P can be added in the product shelf S. In other words, the specific area is an area of the product shelf area corresponding to the front surface of the shelf board of the product shelf S or the price rail attached to the shelf board. The specific area includes the display tag P and the product code displayed on the display tag P. In the present embodiment, the area treated as a specific area differs depending on the setting method described later.

The selection unit 234 selects one focal position pattern according to the detection result of the detection unit 233 as the selection pattern from the plurality of focal position patterns stored in the storage unit 22. The specific operation of the selection unit 234 will be described later.

The focal position setting unit 235 sets the focal position for each of a plurality of individual regions included in the selection pattern based on the specific region (product code) detected by the detection unit 233. The specific operation of the focal position setting unit 235 will be described later.

The signal generation unit 236 generates a control signal based on the selection pattern and the focal position set in the selection pattern. The specific operation of the signal generation unit 236 will be described later.

The "control signal" is a signal for causing the image pickup apparatus 10 to capture a plurality of focus change images focused on each of the plurality of focus positions set in the selection pattern.

The area specifying unit 232, the detection unit 233, the selection unit 234, the focal position setting unit 235, and the signal generation unit 236 constitute the image pickup device control unit in the present invention. That is, the image pickup device control unit includes a region specifying unit 232, a detection unit 233, a selection unit 234, a focal position setting unit 235, and a signal generation unit 236. The image pickup device control unit controls the operation of the image pickup device 10 by using the control signal.

The information specifying unit 237 identifies the product information corresponding to the product code based on the plurality of focus change images captured by the image pickup device 10 whose operation is controlled based on the control signal. The specific operation of the information specifying unit 237 will be described later.

The apparatus may include a display unit (for example, a monitor) that displays various information to the user of the system, and an operation unit (for example, a keyboard) that is operated by the same user.

● Information processing method ●
Next, the information processing method (hereinafter referred to as “the present method”) according to the present invention executed by the present system 1 (the present apparatus 20) will be described with reference to FIG.

FIG. 6 is a flowchart showing an embodiment of the present method.

This method executes a determination process (S1), a control signal generation process (S2), a focus change image imaging process (S6), and a product information integration process (S7).

● Judgment process FIG. 7 is a flowchart of the determination process (S1).
The "determination process (S1)" is a process for determining whether or not it is necessary to execute the control signal generation process (S2), the focus change image imaging process (S6), and the product information integration process (S7) by the apparatus 20. be.

First, the present device 20 sets the next processing execution flag to “OFF” (S101). The "next process execution flag" is a flag that determines whether or not the control signal generation process (S2), the focus change image imaging process (S6), and the product information integration process (S7) need to be executed in this method.

Next, the acquisition unit 231 acquires the captured image as a reference image from the image pickup device 10 via the communication unit 21 (S102). The process (S102) is an example of the acquisition step in the present invention. At this time, the acquisition unit 231 also acquires, for example, information (for example, ID information) for identifying the image pickup device 10 that has captured the reference image. The reference image is stored in the storage unit 22 in association with the information that identifies the image pickup device 10 that has captured the reference image.

The "reference image" is an image used by the apparatus 20 to execute the determination process (S1) among the captured images, and is it necessary to execute the control signal generation process and the focus change image imaging process by the apparatus 20? It is a captured image that serves as a reference for determining whether or not to use it. In the following description, it is assumed that the acquisition unit 231 acquires the captured image shown in FIG. 3 as a reference image.

Next, the area specifying unit 232 identifies the product shelf area captured in the reference image by using, for example, a known image recognition process (S103). The process (S103) is an example of the region specifying step in the present invention. The "known image recognition process" is, for example, a process of detecting an object by using deep learning image recognition (for example, SSD: Single Shot Multibox Detector model). That is, for example, the area specifying unit 232 has a sign (described later) attached to the front surface of the shelf frame of the product shelf S (a member that is the outline of the product shelf S such as a support column or a top plate) captured in the captured image. To detect. Next, the area specifying unit 232 connects the signs arranged in the vertical direction (vertical direction in the product shelf S) among the detected signs, and detects them as the area of the support column (outer frame). Next, the area specifying unit 232 connects the detected signs that are lined up in the left-right direction (horizontal direction in the product shelf S) at the upper end and the lower end, and detects the area of the top plate / base (outer frame). Next, the area specifying unit 232 connects the detected columns, top plate, and base area to specify a rectangular product shelf area. The specified product shelf area is associated with the reference image and stored in the storage unit 22.

A "sign" is, for example, a sticker indicating the position of a strut. The signs are attached to the front of the column, for example, side by side in the vertical direction.

The sign is not limited to the seal. That is, for example, the sign may be a magnet arranged on the support column, a paint applied to the support column, or an unevenness / hole formed on the support column.

Further, the area specifying unit may specify the product shelf area by using the reference image as it is, or the reference image is subjected to a known fluoroscopic conversion process to correct the product shelf to the front view state. The product shelf area may be specified using an image.

Further, the area specifying unit displays, for example, a reference image on a display unit such as a monitor, causes the user of the system 1 to select four points in the product shelf area, and sets the area connecting the selected four points as the product shelf. It may be specified as an area. That is, the area specifying unit may specify the manually selected area as the product shelf area.

Furthermore, the area specifying portion may connect the signs arranged in the left-right direction to specify the area of the shelf board (display tag area).

FIG. 8 is a schematic diagram showing an example of a reference image after the process of specifying the product shelf area (S103). In the figure, the frame showing the product shelf area is shown by a thick line. The figure shows that the product shelf area is a rectangular area.

Return to FIG.
Next, the detection unit 233 detects the display tag P in the product shelf area specified by the area identification unit 232 using, for example, a known image recognition process (S104). The detected display tag P is stored in the storage unit 22 in association with the reference image, for example, together with the ID and position information of the display tag P.

FIG. 9 is a schematic diagram showing an example of a reference image after the process of detecting the display tag P (S104). In the figure, the frame showing the detected display tag P is shown by a thick line, and the product shelf area is shown by a broken line. The figure shows that all the display tags P were detected in the product shelf area.

Return to FIG.
Next, the detection unit 233 determines the presence / absence of the undetected display tag P (S105). At this time, for example, the detection unit 233 refers to the number of types of products displayed on the product shelf S, past detection results (history), an arrangement table of display tags P set for each product shelf S, and the like. Then, it is determined whether or not there is an undetected display tag P. The number of types of products, past detection results, and sequence listings are stored in, for example, in the storage unit 22 in advance.

When there is an undetected display tag P (“Y” in S105), the apparatus 20 sets the next processing execution flag to “ON” (S106). On the other hand, when there is no undetected display tag P (“N” in S105), the present device 20 maintains the next processing execution flag in “OFF” (S107).

Next, the detection unit 233 detects the display tag area based on the detected display tag P (S108). That is, for example, the detection unit 233 extracts a group of display tags P arranged in the X direction of the reference image (horizontal direction of the product shelf S) from the detected display tags P. Next, the detection unit 233 connects the display tags P constituting the group for each extracted group, and detects a linear band-shaped area extending to the support column (end of the product shelf area) as the display tag area. The display tag area has substantially the same width (length in the vertical direction) as the display tag P and substantially the same length (length in the left-right direction) as between the columns. Processing (S108) is an example of a detection step in the present invention. The display tag area is stored in the storage unit 22 in association with the reference image, for example, together with the position information of the display tag area.

FIG. 10 is a schematic diagram showing an example of a reference image after the process of detecting the display tag area (S108). In the figure, the frame showing the detected display tag area is shown by a thick line, and the product shelf area is shown by a broken line. The figure shows that four display tag areas were detected in the product shelf area. For convenience of explanation, the figure shows a display tag P of 1, a product code displayed on the display tag P, another display tag P excluding the display tag P, and a product code displayed on the other display tag P. Illustration is omitted. It is shown that the display tag P and the product code displayed on the display tag P are included in the display tag area.

Return to FIG.
Next, the detection unit 233 detects a set of product codes in all the detected display tags P by using a known image recognition process (S109). At this time, the detection unit 233 acquires the sharpness (contrast) of each product code and specifies the position information thereof. The detected product code is, for example, stored in the storage unit 22 in association with the reference image together with the number assigned for each product code position information, the product code position information, and the sharpness.

The detection unit may perform known preprocessing such as gamma correction and bilateral filter on the reference image before detecting the product code.

Here, the sharpness is calculated using, for example, a known image processing method. That is, for example, the sharpness is calculated by converting the reference image into grayscale, extracting the edges of the product code by the Canny method, and calculating the variance of all the luminance values of the edge image.

Next, the detection unit 233 determines whether or not there is a set of undetected product codes (S110). At this time, for example, the detection unit 233 determines whether or not there is a set of undetected product codes based on the number of display tags P detected in the process (S104).

FIG. 11 is a schematic diagram showing an example of a reference image after the process (S109) for detecting a set of product codes. In the figure, the illustration of the product is omitted for convenience of explanation. In the figure, the detected product code is indicated by "■", and the undetected product code is indicated by "□". The figure shows that the set of detection codes surrounded by the thick line frame has not been detected.

Return to FIG.
When there is an undetected set of product codes (“Y” in S110), the apparatus 20 sets the next processing execution flag to “ON” (S111) and ends the determination processing (S1).

On the other hand, when there is no undetected product code set (“N” in S110), the apparatus 20 ends the determination process (S1) while maintaining the next process execution flag in “OFF” (S112).

● Control signal generation processing FIG. 12 is a flowchart of the control signal generation processing (S2).
In the "control signal generation process (S2)", the apparatus 20 selects a focal position pattern, sets the focal position in an individual region included in the focal position pattern, and sets the selected focal position pattern and the set focal position. It is a process of generating a control signal based on. The control signal generation process (S2) is an example of the signal generation step in the present invention.

First, the present device 20 confirms the state (“ON” or “OFF”) of the next processing execution flag (S201).

When the next processing execution flag is “ON” (“Y” in S201), the selection unit 234 selects one of the plurality of focal position patterns stored in the storage unit 22 according to the detection result of the detection unit 233. The focal position pattern is selected as the selection pattern (S202). That is, for example, the selection unit 234 identifies an individual region in which a set of undetected product codes (or display tag P) exists, and selects a focal position pattern including the individual region. When there are a plurality of focal position patterns as candidates for the selection pattern, the selection unit 234 selects, for example, the selection pattern based on the number of individual regions (maximum number or minimum number) included in the focus position pattern, or A focus position pattern candidate may be shown to the user of the system 1 and the focus position pattern selected by the user may be selected as the selection pattern. The process (S202) is an example of a selection step in the present invention. The selection pattern is stored in the storage unit 22 in association with the reference image, for example. On the other hand, when the next processing execution flag is “ON” (“Y” in S201), the apparatus 20 ends the control signal generation processing (S2). In this case, the apparatus 20 specifies the product information of all the product codes based on, for example, one reference image.

The method by which the selection unit in the present invention selects a selection pattern is not limited to the present embodiment. That is, for example, the selection unit in the present invention may select a selection pattern based on a past selection history. Alternatively, a focal position pattern including an individual area in which an undetected set of product codes (or display tags) is expected to exist is predetermined, and the selection unit selects the determined focal position pattern. You may. Alternatively, the selection unit may select a selection pattern based on the pixel information (position information, sharpness) of the product code.

13 (a) to 13 (f) are schematic views showing an example of a focal position pattern stored in the storage unit 22. In the figure, the individual areas are shown by solid lines, and the individual areas included in the focal position pattern are shown by thick lines. The figure shows that the individual area of “9” is included in the product shelf area. FIG. (A) shows an example of a focal position pattern including the individual region of “9”, FIG. 2B shows an example of a focal position pattern including the individual region of “7”, and FIG. 3C shows an example of the focal position pattern. An example of a focal position pattern including the individual area of "5" is shown, FIG. 6D shows an example of a focal position pattern including the individual area of "4", and FIG. 6E shows an example of the individual area of "6". An example of a focal position pattern including is shown, and FIG. 3F shows an example of a focal position pattern including individual regions having different sizes of “2”. The numbers in the figure are examples of unique numbers for individual areas.

A unique number is assigned to each individual area included in each focal position pattern. In the present embodiment, as shown in FIG. 12, the unique numbers are "2", "3", "4", "5", and "6" clockwise from the upper left individual area, with the central individual area as "1". "7" "8" "9" are assigned.

FIG. 14 is a schematic diagram showing an example of a reference image after the process of selecting a selection pattern (S202).
In the figure, individual areas are indicated by thick dashed lines. The figure shows that a focal position pattern containing "9" individual regions of the same size is selected as the selection pattern.

Return to FIG.
Next, the focus position setting unit 235 selects one setting method from the plurality of setting methods stored in the storage unit 22 (S203). The focal position setting unit 235 selects a setting method based on, for example, the pixel information (for example, sharpness) of the product code acquired in the process (S109). That is, for example, when the average value of the sharpness of the product code is higher than the predetermined threshold value “V1”, the focal position setting unit 235 selects the first setting method described later. On the other hand, when the average value is equal to or less than the predetermined threshold value “V1”, the focal position setting unit 235 selects the second setting method or the third setting method described later. The "predetermined threshold value" V1 "" is set so that, for example, in the process (S303) described later, the sharpness of a sufficient number of product codes is equal to or higher than the threshold value "V2" of the process (S303). The threshold value “V1” is set in advance by the user, for example, and is stored in the storage unit 22.

The "setting method" is a method used by the focus position setting unit 235 to set the focus position for each individual region included in the selection pattern. The setting method includes a first setting method, a second setting method, and a third setting method. The "first setting method" is a method of setting the focal position based on the sharpness of each product code of the reference image detected by the detection unit 233. The "second setting method" is a method of setting the focal position based on the position information of each product code of the reference image detected by the detection unit 233. The "third setting method" is a method of setting the focal position based on the positional relationship of the display tag addition area detected by the detection unit 233 with respect to the product shelf area. In the present embodiment, it is assumed that the storage unit 22 stores the first setting method, the second setting method, and the third setting method in advance.

The focal position setting unit may select, for example, a setting method based on information that can be acquired based on a specific area, and the selection of the focal position setting unit is not limited to the present embodiment. That is, for example, the focal position setting unit may select a setting method based on the number of product code sets or position information detected in the process (S109). Further, for example, the focal position setting unit may select a setting method based on the number of display tags or position information detected in the process (S104). Further, for example, the focal position setting unit may select a setting method preset by the user of the system 1.

When the focus position setting unit 235 selects the first setting method (“first setting method” in S203), the apparatus 20 executes the first focus position setting process (S3). On the other hand, when the focus position setting unit 235 selects the second setting method (“second setting method” in S203), the apparatus 20 executes the second focus position setting process (S4). Further, when the focus position setting unit 235 selects the third setting method (“third setting method” in S203), the apparatus 20 executes the third focus position setting process (S5).

FIG. 15 is a flowchart of the first focal position setting process (S3).
The "first focal position setting process (S3)" is a process of setting the focal position of each individual region included in the selection pattern based on the sharpness of the product code. In the first focal position setting process (S3), the product code is treated as a specific area. The first focal position setting process (S3) is an example of the focal position setting step in the present invention.

First, the focal position setting unit 235 sets the unique number “n” of the setting area for setting the focal position to “1” (S301).

Next, the focal position setting unit 235 reads out all the product codes detected by the detection unit 233 and their sharpness from the storage unit 22 in the individual area “1” (S302).

Next, the focal position setting unit 235 selects, among the product codes, the product code having the sharpness of the predetermined threshold value “V2” or more and the minimum as the specific product code (S303). The “threshold value“ V2 ”” is, for example, a lower limit value of sharpness that allows the information specifying unit 237 to specify product information based on a product code, or a value obtained by adding a slight margin to the lower limit value. In other words, the threshold value "V2" is a value set based on the sharpness by which the apparatus 20 can decode the product code. The threshold value “V2” is set in advance by the user, for example, and is stored in the storage unit 22.

Next, the focal position setting unit 235 reads the position information of the product code selected as the specific product code from the storage unit 22, and sets the center coordinates thereof as the focal position of the individual area “1” (S304). The focal position is stored in the storage unit 22 in association with the corresponding individual area and the selection pattern.

Next, the focus position setting unit 235 determines the presence or absence of an individual region in which the focus position is not set (S305).

When there is an individual area in which the focal position is not set (“Y” in S305), the focal position setting unit 235 adds 1 to the unique number “n” of the individual area in which the focal position is set (S306). , Processing (S302 to S305) is executed.

On the other hand, when there is no individual area in which the focal position is not set (“N” in S305), the focal position setting unit 235 ends the first focal position setting process (S3).

As described above, in the first focus position setting process (S3), the focus position setting unit 235 has a sharpness of the threshold value V1 or more and a sharpness of the plurality of product codes detected in each of the individual regions included in the selection pattern. Select the minimum specific product code and set the focal position based on the position information of the specific product code. In other words, the focal position setting unit 235 sets the focal position to the product code having the smallest sharpness among the decodable product codes in each individual area.

FIG. 16 is a flowchart of the second focal position setting process (S4).
The "second focal position setting process (S4)" is a process of setting the focal position of each individual region included in the selection pattern based on the positional relationship between the product codes. In the second focal position setting process (S4), the product code is treated as a specific area. The second focal position setting process (S4) is an example of the focal position setting step in the present invention.

First, the focal position setting unit 235 reads out the product shelf area corresponding to the reference image from the storage unit 22 (S401).

Next, the focal position setting unit 235 sets the unique number “n” of the setting area for setting the focal position to “1” (S402).

Next, the focal position setting unit 235 reads out all the display tags P detected by the detection unit 233 and their position information (coordinates) from the storage unit 22 in the individual area “1” (S403).

Next, the focal position setting unit 235 specifies the coordinates of the center of gravity of the product shelf area, and specifies the display tag P having the position information closest to the coordinates as the center display tag (S404). That is, for example, the focal position setting unit 235 specifies the coordinates of the center of gravity of each display tag P, and specifies the display tag P having the shortest distance between the coordinates and the coordinates of the center of gravity of the reference image as the center display tag. ..

The focal position setting unit may specify the coordinates of the center position of the reference image instead of the center of gravity of the product shelf area, and may specify the display tag having the position information closest to the coordinates as the center display tag.

Next, the focus position setting unit 235 identifies the product code arranged on the upper side as the center product code among the two product codes corresponding to the center display tag (S405).

The focus position setting unit may specify the product code arranged on the lower side as the central product code.

Next, the focal position setting unit 235 reads the position information of the central product code from the storage unit 22, and obtains the position information (coordinates of the pixels corresponding to the center) corresponding to the center of the central product code to the focal position of the individual area “1”. Is set as (S406). The focal position is stored in the storage unit 22 in association with the corresponding individual area and the selection pattern.

Next, the focus position setting unit 235 adds 1 to the unique number “n” of the individual area for setting the focus position (S407).

Next, the focal position setting unit 235 determines whether the unique number of the individual region is an even number (“2,4,6,8”) or an odd number (“3,5,7,9”) (S408).

When the unique number of the individual area is an even number (“even number” in S408), the focal position setting unit 235 identifies the display tag P closest to the corner of the product shelf area as the end display tag in the individual area (“even number” in S408). S409). In other words, the focal position setting unit 235 identifies the display tag P that is close to the end of the product shelf area and is the furthest from the central product code as the end display tag in the individual area. That is, for example, the focal position setting unit 235 specifies the coordinates of the center of gravity of each display tag P, and the end display tag P is the display tag P having the shortest distance between the coordinates and the coordinates of the corner of the product shelf area. Specify as. Here, the "corner portion of the product shelf area" is a corner portion corresponding to each individual area. That is, for example, in the present embodiment, the upper left corner of the product shelf area corresponds to the individual area "2", the upper right corner of the product shelf area corresponds to the individual area "4", and the product shelf area. The lower right corner corresponds to the individual area "6", and the lower left corner of the product shelf area corresponds to the individual area "8".

On the other hand, when the unique number of the individual area is an odd number (“odd number” in S408), the focal position setting unit 235 displays the display tag P closest to the midpoint of each side of the product shelf area at the end of the individual area. Specify as a tag (S410). In other words, the focal position setting unit 235 specifies the display tag P closest to the central product code among the display tags P near the end of the product shelf area as the end display tag in the individual area. That is, for example, the focal position setting unit 235 specifies the coordinates of the center of gravity of each display tag P, and ends the display tag P having the shortest distance between the coordinates and the coordinates of the midpoint of each side of the product shelf area. Specify as a department display tag. Here, "each side of the product shelf area" is a side corresponding to each individual area. That is, for example, in the present embodiment, the upper side of the product shelf area corresponds to the individual area "3", the right side of the product shelf area corresponds to the individual area "5", and the lower side of the product shelf area corresponds to the individual area "7". , And the left side of the product shelf area corresponds to the individual area “9”.

Next, the focus position setting unit 235 identifies the product code arranged on the upper side as the end product code among the two product codes corresponding to the end display tags (S411).

The focus position setting unit may specify the product code arranged on the lower side as the end product code.

Next, the focus position setting unit 235 reads the position information of the end product code from the storage unit 22, and uses the position information (coordinates of the pixels corresponding to the center) corresponding to the center of the end product code on the same end display tag. It is set as the focal position of the corresponding individual area (S412). The focal position is stored in the storage unit 22 in association with the corresponding individual area and the selection pattern.

Next, the focus position setting unit 235 determines the presence or absence of an individual region in which the focus position is not set (S413).

When there is an individual area in which the focal position is not set (“Y” in S413), the focal position setting unit 235 executes the process (S407 to S413).

On the other hand, when there is no individual area in which the focal position is not set (“N” in S413), the focal position setting unit 235 ends the second focal position setting process (S4).

As described above, in the second focus position setting process (S4), the focus position setting unit 235 is the corner portion or each side of the product shelf area among the display tags P detected in each of the individual areas included in the selection pattern. The display tag P closest to the midpoint is specified as the end display tag. In other words, the focal position setting unit 235 is the display tag P that is close to the end of the product shelf area and is the furthest from the central product code, or the display tag P that is closest to the end of the product shelf area in the individual area. Of these, the display tag P closest to the central product code is specified as the end display tag.

Further, the focus position setting unit 235 specifies the product code arranged on the upper side as the end product code among the two product codes corresponding to the end display tags. In other words, the focal position setting unit 235 is the product code located near the end of the product shelf area and the farthest from the central product code, or the end of the product shelf area among the product codes arranged on the upper side. Among the product codes close to, the product code closest to the central product code is specified as the end product code. That is, the focal position setting unit 235 is detected in each of the individual regions based on the positional relationship between each product code excluding the central product code and the central product code among the product codes detected by the detection unit 233. One product code is selected from the product codes, and the focal position is set based on the position information.

FIG. 17 is a flowchart of the third focal position setting process (S5).
The "third focal position setting process (S5)" is a process of setting the focal position of each individual area included in the selection pattern based on the positional relationship of the specific area (display tag area) with respect to the product shelf area. In the third focus position setting process (S5), the display tag area is treated as a specific area. The third focal position setting process (S5) is an example of the focal position setting step in the present invention.

First, the focal position setting unit 235 reads out the product shelf area corresponding to the reference image, the display tag area, and the position information thereof from the storage unit 22 (S501).

Next, the focal position setting unit 235 sets the unique number “n” of the setting area for setting the focal position to “1” (S502).

Next, the focus position setting unit 235 specifies all the display tag areas detected by the detection unit 233 and the position information (coordinates) thereof in the individual area "1" (S503).

Next, the focal position setting unit 235 specifies the coordinates of the center of gravity of the product shelf area, and specifies the display tag area having the position information closest to the coordinates (S504).

The focal position setting unit may specify the coordinates of the center position of the reference image instead of the center of gravity of the product shelf area, and may specify the display tag area having the position information closest to the coordinates.

Next, the focal position setting unit 235 sets the coordinates of the center of gravity of the display tag area specified in the individual area "1" as the focal position of the individual area "1" (S505). The focal position is stored in the storage unit 22 in association with the corresponding individual area and the selection pattern.

Next, the focus position setting unit 235 identifies the individual area having an even number (“2, 4, 6, 8”) among the individual areas for setting the focus position as the specific individual area, and the unique number is assigned. An odd number (“3, 5, 7, 9”) individual area is specified as a non-specific individual area (S506). The "specific individual area" is an individual area including a corner portion of the product shelf area among the individual areas. The "non-specific individual area" is an individual area located between the specific individual areas among the individual areas. That is, the individual area includes a specific individual area and a non-specific individual area.

Next, the focus position setting unit 235 selects the specific individual area having the smallest unique number among the specific individual areas in which the focus position has not been set (S507).

Next, the focal position setting unit 235 identifies the display tag area closest to the corner of the product shelf area in the specific individual area (S508).

Next, the focus position setting unit 235 is the coordinates of a point located on the center line of the specified display tag area and inside the predetermined length X direction from the end of the product shelf area adjacent to the display tag area. Is set as the focal position of the corresponding specific individual region (S509). The focal position is stored in the storage unit 22 in association with the corresponding individual area (specific individual area) and the selection pattern.

Next, the focal position setting unit 235 determines the presence or absence of a specific individual region in which the focal position has not been set (S510). When there is a specific individual area in which the focal position has not been set (“Y” in S510), the focal position setting unit 235 executes processing (S507 to S509).

On the other hand, when there is no specific individual area for which the focal position has not been set (“N” in S510), the non-specific individual area having the smallest unique number is selected from the non-specific individual areas for which the focal position has not been set (S511). ).

Next, the focal position setting unit 235 generates a line segment connecting the focal positions set in the two specific individual regions adjacent to the selected non-specific individual region, and the coordinates of the midpoint of the same line segment, or The intersection of the same line segment and the center line of the display tag area closest to the midpoint of the same line segment is set as the focal position of the non-specific individual area (S512). That is, for example, the focal position of the non-specific individual area "3" is the coordinates of the midpoint of the line segment connecting the focal positions of the specific individual areas "2" and "4". The focal position is stored in the storage unit 22 in association with the corresponding individual area (non-specific individual area) and the selection pattern.

Next, the focus position setting unit 235 determines the presence or absence of a non-specific individual region in which the focus position has not been set (S513). When there is a non-specific individual area in which the focal position has not been set (“Y” in S513), the focal position setting unit 235 repeats the process (S511 to S512).

On the other hand, when there is no non-specific individual area in which the focal position has not been set (“N” in S513), the focal position setting unit 235 ends the third focal position setting process (S5).

As described above, in the third focal position setting process (S5), the focal position setting unit 235 sets the focal position corresponding to the specific individual area based on the positional relationship of the specific individual area with respect to the product shelf area, and sets the specific individual area. Based on the focal position set in the area, the focal position corresponding to the non-specific individual area is set.

Return to FIG.
Next, the signal generation unit 236 reads out the selection pattern and the focal position set in the individual area included in the selection pattern from the storage unit 22 (S204).

Next, the signal generation unit 236 generates a control signal for the image pickup device 10 to capture a plurality of focus change images focused on each focus position of the selection pattern based on the read selection pattern and the focus position. (S205). The control signal is stored in the storage unit 22.

● Focus-changing image imaging process FIG. 18 is a flowchart of the focus-changing image imaging process (S6).
The “focus change image imaging process (S6)” is a process in which the image pickup apparatus 10 acquires a focus change image based on a control signal.

First, the control unit 23 of the present device 20 reads out the control signal from the storage unit 22 (S601).

Next, the control unit 23 transmits the control signal to the image pickup device 10 that has captured the reference image that is the basis of the generation of the control signal via the communication unit 21 (S602).

Next, the control unit 13 of the image pickup apparatus 10 receives the control signal from the apparatus 20 via the communication unit 11 (S603). The control signal is transmitted to the control unit 13.

Next, the control unit 13 captures a focus change image focused on each focal position based on the control signal (S604). The number of focus change images is captured as many as the number of focus positions.

Next, the control unit 13 determines whether or not there is a focus shift (so-called out-of-focus) for each focus change image (S605).

When the focus is appropriate (“Y” in S605), the control unit 13 uses a known image recognition process to determine whether or not a person is included in the focus change image (S606).

When the focus is inappropriate (“N” in S605) or when a person is included in the focus change image (“Y” in S606), the control unit 13 controls the focus position corresponding to the corresponding focus change image. Is read from the storage unit 12 (S607), and the process (S604) is executed.

On the other hand, when the focus change image does not include a person (“N” in S606), the control unit 13 associates each focus change image with the corresponding focus position and associates the focus change images with each other. It is stored in the storage unit 12 as a set of focus change image groups (S608).

Next, the control unit 13 reads out the focus change image group from the storage unit 12 and transmits it to the apparatus 20 via the communication unit 11 (S609).

Next, the control unit 23 (acquisition unit 231) of the apparatus 20 receives (acquires) the focus change image group from the image pickup apparatus 10 via the communication unit 21 (S610). At this time, the control unit 23 (acquisition unit 231) also receives (acquires), for example, information (for example, ID information) for identifying the image pickup device 10 that has captured the focus change image group. The focus change image group is stored in the storage unit 22 in association with the information that identifies the image pickup device 10 that has captured the focus change image group.

● Product information integration process FIG. 19 is a flowchart of the product information integration process (S7).
In the "product information integration process (S7)", the apparatus 20 acquires and integrates the product information for each focus change image based on the focus change image, and obtains the product information corresponding to the product code attached to the product shelf S. It is a process to acquire. The product information integration process (S7) is an example of the information specifying step in the present invention. In the following description, the focus change image captured at the focus position set in the individual area “n” is referred to as “focus change image“ n ””. That is, for example, the focus change image captured at the focus position set in the individual area "1" is referred to as the focus change image "1".

First, the detection unit 233 reads out the focus change image group from the storage unit 22 (S701).

Next, the detection unit 233 detects the product code for each focus change image (S702). At this time, the detection unit 233 detects the display tag P, detects the product code corresponding to the detected display tag P, and specifies the position information thereof, for example, as in the process (S104, S109). The detected product code is, for example, stored in the storage unit 22 in association with the focus change image together with the number assigned for each product code position information and the product code position information.

Next, the information specifying unit 237 specifies the product information corresponding to the product code for each focus change image (S703). That is, the information specifying unit 237 decodes (identifies) the product code for each focus change image. The product information is stored in the storage unit 22 in association with the position information of the corresponding product code and the corresponding focus change image.

Next, the information specifying unit 237 compares the position information of each product code between the focus change images (S704). That is, for example, the information specifying unit 237 compares the position information of each product code of the focus change image "1" with the position information of each product code of the focus change images "2" to [9].

Next, the information specifying unit 237 identifies the product code located within the common distance range “L1” between the focus change images as a product code group corresponding to the same product code between the focus change images (S705). That is, the information specifying unit 237 estimates that the product codes imaged at substantially the same position in each focus change image are the same product code, and these product codes are a set of the same product code, which is a product code group. Specify as. The “distance range“ L1 ”” is a range for specifying the product code of the focus change images as the product code attached to the same position on the product shelf S over the plurality of focus change images. That is, for example, when the distance between the product code "α" having the focus change image "1" and the product code "β" having the focus change image "2" is within the distance range "L1", the product code. The "α" and the product code "β" are specified as product codes (that is, the same product code) attached to the same position on the product shelf S.

Next, the information specifying unit 237 compares the product information corresponding to the product code constituting the product code group for each product code group, and identifies a single product information corresponding to the product code group (S706). That is, for example, the information specifying unit 237 specifies the product information having the largest number for each product code group as a single product information corresponding to the product code group. That is, the information specifying unit 237 integrates the product information for each product code group and identifies a single product information corresponding to the product code group. The single product information is associated with the corresponding product code group and stored in the storage unit 22.

The information specifying unit may delete duplicate product information for each product code group and specify the remaining product information as a single product information corresponding to the product code group.

In this way, the present device 20 identifies the product information corresponding to the product code detected in each focus change image, compares the position information of each product code between the focus change images, and identifies the product code group. A single product information corresponding to a product code group is specified based on the product information corresponding to a plurality of product codes included in the product code group. That is, the present device 20 does not synthesize the focus change images captured by the so-called multi-focus points, but integrates the product information specified in the same focus change image and captures the product shelf in the focus change image. Identify a single product information for S.

Further, the present device 20 identifies the focal position of each focus change image based on one captured image by the above-mentioned method, and corresponds to the image pickup device based on the product code detected in the focus change image. It is possible to more accurately identify the product information of the products displayed on the product shelves. Then, the present device 20 can execute product management such as shelf allocation creation and shortage detection for each product shelf with high accuracy based on the specified product information.

● Summary According to the embodiment described above, the system 1 (device 20) detects a storage unit 22 that stores a plurality of focal position patterns and a plurality of specific areas imaged in the product shelf area. It has a detection unit 233 and a selection unit 234 that selects one focal position pattern according to the detection result of the detection unit 233 as a selection pattern among a plurality of focal position patterns stored in the storage unit 22. Become. According to this configuration, the system 1 (device 20) simply selects the focal position pattern of 1 according to the detection result of the detection unit 233 as the selection pattern, and sets the focal position of the so-called multi-focus point. The combination of (individual areas) can be specified.

Further, according to the embodiment described above, the system 1 (device 20) sets the focal position for each of a plurality of individual areas included in the selection pattern based on the specific area detected by the detection unit 233. It has a focal position setting unit 235 to be set. According to this configuration, the system 1 (device 20) can easily set the focal position for each individual region included in each selection pattern for each captured image acquired as a reference image by using the information of the specific region. .. In other words, the system 1 (device 20) can set the focal position for each captured image according to the imaging conditions (number of pixels, angle of view, etc.) and imaging environment (brightness, etc.) of the imaging device 10.

Further, according to the embodiment described above, the system 1 (device 20) generates a control signal for causing the image pickup device 10 to capture a plurality of focus change images focused on each of the plurality of focus positions. It has a generation unit 236 and an information identification unit 237 that specifies product information corresponding to a product code based on a plurality of focus change images captured by an image pickup device 10 whose operation is controlled based on a control signal. It becomes. According to this configuration, the system 1 (device 20) identifies the product code from each of the focus change images captured at the so-called multi-focus points for each captured image acquired as the reference image, and corresponds to the product code. You can specify the product information to be used.

As described above, the system 1 (device 20) can capture all the product codes in the captured image of the product shelf S in a distinguishable state. Therefore, the recognition accuracy of the product code captured in the captured image is improved. As a result, the system 1 (device 20) can be utilized for highly accurate product management (such as shelving allocation creation and shortage detection).

Furthermore, according to the embodiment described above, the selection unit 234 selects a selection pattern based on the pixel information (position information, sharpness) of the plurality of product codes detected by the detection unit 233. According to this configuration, the system 1 (device 20) can select a selection pattern based on the position information or sharpness of the product code that requires accurate detection in order to decode the product information. As a result, the system 1 (device 20) can select a selection pattern that can accurately detect the product code. As a result, the system 1 (device 20) can be utilized for highly accurate product management.

Furthermore, according to the embodiment described above, the focal position setting unit 235 has the product code having a sharpness of a predetermined threshold value V1 or more and the minimum among the plurality of product codes detected in each of the individual regions. Select and set the focal position based on the position information of the selected product code. That is, the focal position setting unit 235 sets the focal position based on the sharpness of each product code detected by the detection unit 233. Usually, when a product code is decoded from a captured image, the sharpness (contrast) of the product code is a factor that greatly affects the decoding (identification) of the product code. According to this configuration, the focal position setting unit 235 can set the focal position according to the product code having a sharpness close to the identifiable lower limit value among the product codes detected in the individual regions. That is, the system 1 (device 20) can reliably identify the product code in the focal position change image regardless of the position of the product code in the individual area, and can identify other product codes in the corresponding individual area. You can set the focus position on the product code that can be improved. As a result, the system 1 (device 20) can be utilized for highly accurate product management.

Furthermore, according to the embodiment described above, the focal position setting unit 235 is the product code or the end portion which is close to the end portion of the product shelf area and is the farthest from the central product code for each individual area. Among the product codes close to, select one of the product codes closest to the central product code, and set the focal position based on the position information of the selected product code. That is, the focus position setting unit 235 sets the focus position based on the positional relationship between the product codes detected by the detection unit 233. Normally, the image pickup apparatus 10 aligns the focal position with the vicinity of the center of the angle of view and captures the captured image. Then, in the captured image, the focal point tends to shift as the distance from the focal position at the time of imaging increases. According to this configuration, the focal position setting unit 235 can set the focal position according to the product code (the product code that is out of focus) that is easily out of focus among the product codes detected in the individual areas. That is, even if the position of the product code in the individual area is changed due to, for example, a change in shelving allocation, the system 1 (device 20) becomes a product code that is easily out of focus (a product code that is out of focus). You can also set the focal position. As a result, the system 1 (device 20) can be utilized for highly accurate product management.

Furthermore, according to the embodiment described above, the detection unit 233 corresponds to the product code located within the common distance range “L1” between the focus change images and the same product code between the focus change images. Specify as a product code group. The information specifying unit 237 identifies the product information corresponding to the product code group based on the product information corresponding to the plurality of product codes included in the product code group. According to this configuration, the system 1 (device 20) identifies a group of product codes (product code group) detected at substantially the same positions among a plurality of focus change images as the same product code, and the same. A single product information corresponding to a group can be identified. As a result, the system 1 (device 20) can be utilized for highly accurate product management.

Furthermore, according to the embodiment described above, the information specifying unit 237 integrates the product information specified for each focus change image and corresponds to the product code displayed by each display tag attached to the product shelf. Specify the product information to be used. As a result, the information specifying unit 237 can identify the product information having the largest number for each product code group as a single product information corresponding to the product code group as a result of the integration. Further, the information specifying unit can delete duplicate product information for each product code group and specify the remaining product information as a single product information corresponding to the product code group. As a result, the system 1 (device 20) can be utilized for highly accurate product management.

The number of individual areas included in the product shelf area is not limited to "9". That is, for example, the number of individual areas included in the product shelf area may be "25" or may be an even number of "4" and "6". When the number of individual areas included in the product shelf area is even, the focal position setting unit in the present invention uses the coordinates of the center position of the reference image or the center of gravity of the product shelf area in place of the center product code in the control signal generation process. The focal position may be set based on the positional relationship between the product code of each individual area and the product code of each individual area.

Further, the detection unit in the present invention does not have to detect the display tag in the determination process. That is, for example, the detection unit in the present invention may detect only the product code as a specific area in the determination process.

Further, the detection unit in the present invention may detect only the upper product code (only the lower product code) instead of the set of product codes in the determination process.

Furthermore, the number of product codes displayed on the display tag may be "1". In this case, the detection unit in the present invention detects a single product code for each display tag in the determination process.

Furthermore, the detection unit in the present invention does not have to acquire the sharpness of the product code in the determination process. In this case, the sharpness may be acquired, for example, in the control signal generation process (first focus position setting process).

Furthermore, the detection unit in the present invention does not have to detect the display tag area. In this case, the system 1 (device 20) does not have to execute the third focal position setting process.

Furthermore, the focal position setting unit in the present invention may select at least one setting method from the first to third setting methods in the control signal generation process. That is, for example, the focal position setting unit in the present invention may be configured to select only one of the first setting method and the second setting method in the control signal generation processing, or only the third setting method. May be configured to select. In other words, the storage unit in the present invention may store only one or two of the first to third setting methods, or the first to third settings. A setting method other than the method may be stored.

Furthermore, the focal position setting unit in the present invention may select the product code having the minimum sharpness as the specific product code in the first focal position setting process.

Furthermore, when only one product code is displayed on the display tag, the focus position setting unit in the present invention uses the product code displayed on the display tag as the center product code or the end in the second focus position setting process. It may be specified as a department product code.

Furthermore, the focal position setting unit in the present invention need only set the focal position in the individual region corresponding to the unique number of the individual region included in the selection pattern in the control signal generation process. That is, for example, the focal position setting unit in the present invention has individual areas "1" and "2" when the unique numbers of the individual areas included in the selection pattern are "1", "2", "4", "6", and "8". It is only necessary to set the focal position in "4", "6", and "8".

Furthermore, the selection unit in the present invention has a focus position pattern different from the selection pattern based on the position information of the product code for which the product information is not specified when the information identification unit cannot specify the product information corresponding to the product code. May be selected as a new selection pattern. According to this configuration, this system (this device) can acquire product information more accurately.

Furthermore, the image pickup apparatus in the present invention may function as the present apparatus.

Furthermore, the acquisition unit, the area identification unit, the detection unit, the selection unit, the focal position setting unit, the signal generation unit, and the information identification unit in the present invention are composed of a common processor and a semiconductor memory. It may be composed of a plurality of processors and a semiconductor memory.

Furthermore, according to the embodiment described above, the present device 20 is composed of one computer. Instead of this, the present device may be composed of a plurality of computers. That is, for example, the present device may be composed of a plurality of computers that function as the present device. Specifically, for example, the present apparatus (computer group) may be composed of a computer including a storage unit and a computer including a control unit for executing the present method. Further, for example, a plurality of computers may provide the acquisition unit, the area identification unit, the detection unit, the selection unit, and the functions of the focus position setting unit, the signal generation unit, and the information identification unit in a distributed manner. In this case, the plurality of computers may send and receive information through a communication line, or may transfer information using a portable storage medium.

Furthermore, in the present method described above, each treatment may be limited to the present embodiment as long as the present invention can be carried out. That is, for example, this method may be executed for a captured image in which all product codes can be detected without performing a determination process, or a product shelf area, a display tag area, and a set of product codes. Detection may be performed in the control signal generation process.

Furthermore, the embodiments described above do not unreasonably limit the content of the invention described in the claims. Moreover, not all of the configurations described in the embodiments described above are essential constituent requirements of the present disclosure.

1 Information processing system 10 Image pickup device 20 Information processing device 22 Storage unit 231 Acquisition unit 232 Area identification unit 233 Detection unit 234 Selection unit 235 Focus position setting unit 236 Signal generation unit 237 Information specification unit

Claims (18)

An information processing device that identifies product information of the product based on a plurality of captured images captured by an image pickup device that captures an image of the product shelf on which the product is displayed.
A display tag displaying a product code corresponding to the product information is attached to the product shelf.
The area of the product shelf captured in the captured image includes a plurality of individual areas.
The plurality of captured images include a plurality of focus change images having different focal positions.
A storage unit that stores a plurality of focal position patterns that specify a combination of the individual areas in which the focal position is set among the plurality of individual areas.
An acquisition unit that acquires the captured image of the product shelf as a reference image by the imaging device, and
An image pickup device control unit that controls the operation of the image pickup device, and
An information specifying unit that identifies the product information corresponding to the product code based on a plurality of focus change images captured by the image pickup device whose operation is controlled based on the control signal.
Have
The image pickup device control unit is
An area specifying part that specifies the area of the product shelf captured in the reference image, and
A detection unit that detects a plurality of specific areas including the product code imaged in the area of the product shelf, and a detection unit.
A selection unit that selects one of the focal position patterns stored in the storage unit according to the detection result of the detection unit as a selection pattern.
A focal position setting unit that sets the focal position for each of the plurality of individual regions included in the selection pattern based on the specific region detected by the detection unit.
A signal generation unit that generates the control signal for causing the image pickup apparatus to image a plurality of the focus change images focused on each of the plurality of focal positions.
To prepare
An information processing device characterized by this. The specific area is the product code.
The information processing apparatus according to claim 1. The selection unit selects the selection pattern based on the pixel information of the plurality of product codes detected by the detection unit.
The information processing apparatus according to claim 2. The focal position setting unit sets the focal position based on the sharpness of each of the product codes detected by the detection unit.
The information processing apparatus according to claim 2. The focal position setting unit is
From the plurality of product codes detected in each of the individual regions, the product code having the sharpness equal to or higher than a predetermined threshold and the minimum is selected.
The focal position is set based on the position information of the selected product code.
The information processing apparatus according to claim 4. The focal position setting unit sets the focal position based on the positional relationship between the product codes detected by the detection unit.
The information processing apparatus according to claim 2. The focal position setting unit is
In the reference image, the center of gravity of the area of the product shelf is specified.
The product code closest to the center of gravity is specified as the central product code, and the product code is specified.
Among the product codes detected by the detection unit, the product code detected in each of the individual regions based on the positional relationship between each of the product codes excluding the central product code and the central product code. Of these, select the product code of 1 and select
The focal position is set based on the position information of the selected product code.
The information processing apparatus according to claim 6. The focal position setting unit is the product code of the product code closest to the end of the region of the product shelf and farthest from the center product code, or the product code closest to the end of the product shelf for each individual area. Among them, the product code of any of the product codes closest to the central product code is selected.
The information processing apparatus according to claim 7. The storage unit stores a plurality of setting methods in which the focal position setting unit sets the focal position.
The setting method is
A first setting method for setting the focal position based on the sharpness of each of the product codes detected by the detection unit of the reference image, and
A second setting method for setting the focal position based on the position information of each of the product codes detected by the detection unit of the reference image, and
Including
The focal position setting unit selects either the first setting method or the second setting method based on the pixel information of each of the product codes detected by the detection unit of the reference image.
The information processing apparatus according to claim 2. The detector is
For each focus change image, a plurality of the product codes in the area of the product shelf are detected.
The position information of each of the product codes detected for each focus change image is specified, and the position information is specified.
The position information of each of the product codes is compared between the focus change images, and the position information is compared.
The product code located within a common distance range between the focus change images is specified as a product code group corresponding to the same product code between the focus change images.
The information processing apparatus according to claim 1. The information specifying unit identifies the product information corresponding to the product code group based on the product information corresponding to the plurality of product codes included in the product code group.
The information processing apparatus according to claim 10. The information specifying part is
For each focus change image, the product information is specified based on the product code.
The product information specified for each focus change image is integrated to specify the product information corresponding to the product code displayed by each of the display tags attached to the product shelves.
The information processing apparatus according to claim 9 or 10. When the information specifying unit cannot specify the product information corresponding to the product code.
The selection unit selects the focal position pattern different from the selection pattern as a new selection pattern based on the position information of the product code for which the product information is not specified.
The information processing apparatus according to claim 1. The focal position setting unit sets the focal position based on the positional relationship of the specific area with respect to the area of the product shelf.
The specific area is an area in the area of the product shelf on which the display tag can be imaged.
The information processing apparatus according to claim 3. The individual area includes a specific individual area and a non-specific individual area.
The focal position setting unit is
Based on the positional relationship of the specific area with respect to the area of the product shelf, the focal position corresponding to the specific individual area is set.
Based on the focal position set in the specific individual area, the focal position corresponding to the non-specific individual area is set.
The information processing apparatus according to claim 14. The computer functions as the information processing device according to claim 1.
An information processing program characterized by this. An information processing system that identifies product information of the product based on a plurality of captured images of the product shelves on which the product is displayed.
An image pickup device that captures the captured image and
An information processing device that identifies the product information and
Have
The information processing device is the information processing device according to claim 1.
An information processing system characterized by this. An information processing method executed by an information processing device that specifies product information of the product based on a plurality of captured images captured by an image pickup device that captures an image of the product shelf on which the product is displayed.
A display tag indicating a product code corresponding to the product information of the product is attached to the product shelf.
The area of the product shelf captured in the captured image includes a plurality of individual areas.
The plurality of captured images include a plurality of focus change images having different focal positions.
The information processing device is
A storage unit that stores a plurality of focal position patterns that specify a combination of the individual areas in which the focal position is set among the plurality of individual areas.
Equipped with
The information processing device
An acquisition step in which the image pickup device acquires the captured image as a reference image obtained by capturing the image of the product shelf, and the acquisition step.
An area specifying step for specifying the area of the product shelf captured in the reference image, and
A signal generation step for generating a control signal for controlling the operation of the image pickup device, and
An information identification step for specifying the product information corresponding to the product code based on a plurality of focus change images captured by the image pickup device whose operation is controlled based on the control signal.
Have
The signal generation step is
A detection step for detecting a specific area including the product code imaged in the area of the product shelf, and a detection step.
A selection step of selecting one of the focal position patterns stored in the storage unit according to the detection result of the detection unit as a selection pattern.
A focal position setting step for setting the focal position for each of the plurality of individual regions included in the selection pattern based on the specific region detected by the detection step.
Including
The control signal is a signal for causing the image pickup apparatus to capture a plurality of the focus change images focused on each of the plurality of focal positions.
An information processing method characterized by that.

Images