JP7401129B1 - Product display shelf - Google Patents

Abstract

【assignment】
Rather than processing a huge amount of image information using multiple cameras to observe customer consumption behavior and fraudulent behavior, or to make stores unmanned, we can use simple processing to monitor consumer behavior, prevent fraud, and create unmanned stores. We provide product display shelves that make this possible.
[Solution]
A camera using a linear sensor is installed on the product display shelf to monitor the front plane of the product display shelf, and when work is required to move products in and out of the product display shelf.To provide a product display shelf that greatly reduces the amount of information associated with picking up products and realizes an unmanned store by outputting only images of areas where work is occurring when products are taken in and out.

[Selection diagram] Figure 11

Description

The present invention relates to the provision of a device for understanding customer purchasing behavior, in which a sensor is installed on a product display shelf to determine the purchasing behavior of customers in a retail store, and the information is utilized to determine the purchased product and price. In addition to contributing to unmanned and labor-saving stores, product display also makes it possible to identify and analyze returned products and contribute to sales promotion measures, as well as to prevent shoplifting. Regarding shelves.

In retail stores, understanding the purchasing behavior of customers and providing products that match customer preferences in a timely manner leads to increased sales. Currently, this largely relies on shopkeepers' intuition based on their long experience and observations of customer purchasing behavior on a daily basis, which has become a kind of know-how.
Additionally, POS systems analyze sales information at the cash register and determine which products are best sellers, and do not measure customer interest in products by picking them up and returning them.

On the other hand, attempts are being made to collect such customer purchasing behavior information using advanced technology and use it to improve purchasing performance.
For example, Patent Document 1 discloses that customer movement information is collected by calculating customer position information from image information in a store, and product movement information is calculated by calculating product position information from a transmitting/receiving device attached to the product. A means is disclosed for collecting product selection behavior of a specific customer from the moment he/she enters the store to the time he/she leaves the store, based on customer movement information, product movement information, and purchased product information.

Furthermore, Patent Document 2 discloses that an IC chip for individual identification is attached to a product, and information on whether a customer has picked up the product, information on the customer's movement, and information on the customer's residence time are acquired together. discloses an invention for identifying and analyzing products that a customer obtained but did not end up purchasing.

Furthermore, in Patent Document 3, IC chips (IC tags) are attached to products, and non-contact IC chip reading devices placed throughout the store read information held by nearby IC tags, and the results are sent to the store. A technique for collecting information using an information terminal device has been disclosed.

On the other hand, Patent Document 4 discloses that a camera composed of an area sensor is installed on the upper part of a product display shelf, and a part of a customer's body enters the shelf and remains at each product position for a predetermined period of time or more. A system is disclosed in which it is assumed that an object stored on a step has been accessed.

Although Patent Document 5 is different from purchasing behavior information, it describes a method of detecting a touched position using a plurality of light receiving elements based on the principle of an optical touch panel.

However, the following problems and unsatisfactory points have been pointed out in the conventional methods and systems as described above.
In Patent Document 1, it is necessary to attach a transmitter/receiver device to the product, which increases costs and takes time and effort, and it is only possible to collect information on a specific customer, so it is hard to say that it is a collection of truly useful information.

The means described in Patent Document 2 also requires each customer to carry an IC card specific to the customer in order to detect movement information and residence time, and it is impossible to collect information for customers who do not have an IC card. However, it was only a means of analyzing customer purchasing behavior within a limited range.

In Patent Document 3, an IC chip (IC tag) is attached, and non-contact IC chip reading devices placed at various locations in the store read the information held by the nearby IC tag, and the results are sent to an information terminal device in the store. Since the products are collected at a local store, it is necessary to attach IC tags to the products, which inevitably increases costs due to equipment costs and man-hours.

The invention described in Patent Document 4 is closest to the present application, but it requires image processing to determine the position and part of the human body from the camera image information of the area sensor, and it requires image processing to determine the position and part of the human body from the camera image information of the area sensor. Similar to cameras, since area sensor cameras are installed so that they look down, they are an eyesore and require a complex system to process image information to be installed on the product display shelves. In addition, it is necessary to constantly take images, which puts stress on customers as they are being watched by cameras on each shelf.
The invention described in Patent Document 6 has nothing to do with purchasing behavior, but only has a function of detecting a touch position.

The present invention has been made in view of such problems, and does not require the use of IC tags or the like, does not require analysis of images taken with a camera of customer behavior in a retail store, and does not require the use of a part of the human body. It also eliminates the need for image processing. By monitoring the flat surface in front of the display shelf with a linear sensor camera installed on the product display shelf, it is possible to know what items the customer has picked up and how many, and whether they have put them back.

It is possible to provide a product display shelf that is useful for a customer behavior analysis system that can accurately and automatically store the purchasing behavior of an unspecified number of customers as a database. In addition, by combining surveillance cameras with customer flow lines, it becomes possible to improve the accuracy of shoplifting prevention, and contribute to the realization of unmanned stores in the future.
In addition, based on the customer behavior information obtained from this product display shelf, it becomes possible to collect detailed data such as the number of customers who came into contact with the shelf or the product, the number of products displayed on the shelf, etc. It can provide sales promotion support information such as product placement and store layout that are useful for significant labor savings and sales promotion.

JP 11-175597 JP2004-348681 JP2006-318191 Patent No. 4972491 Patent No. 5025552

In the conventional methods of understanding customer purchasing behavior shown in Patent Documents 1, 2, and 3, it was necessary to attach an IC chip or an IC tag to the product, which required extra work. Further, in Patent Document 4, it is necessary to perform image processing on customer behavior, which increases the calculation scale and increases power consumption.
The labor shortage has become serious due to the declining birthrate and aging population, and despite increasing the number of foreigners in the technical intern training program, the number of people has decreased due to the coronavirus pandemic and the weak yen, and the labor shortage has not been resolved. In addition, unmanned retail stores such as unmanned stores have been experimentally introduced, but image processing using cameras and AI cameras is the main focus.In order to eliminate blind spots, many cameras are installed and a huge number of This required extensive image processing.

Current retail stores generally use POS systems to grasp sales trends at the time of checkout.If a customer picks up a product from a product display shelf and puts it back, current POS systems cannot detect it. Can not. In order to understand a customer's level of interest in a product, it was necessary to observe the customer at the store using a researcher or camera, or to have the customer wear a device called eye tracking to track their line of sight.

In order to solve the above problem, we installed a camera using a linear image sensor (hereinafter abbreviated as linear sensor) on the product display shelf to monitor the front plane of the product display shelf, and monitor the work associated with the movement of products into and out of the product display shelf. To provide a product display shelf for acquiring information regarding products when a problem occurs. By using a linear sensor, there is no need to use a circular lens like a normal camera using an area sensor, and a linear sensor camera using a flat lens (called a cylindrical lens) Free yourself from the stress of being watched.

Another good thing about linear sensor cameras is that stationary objects are not captured in the image. In other words, with the linear sensor camera proposed in this proposal, which can only see the flat surface in front of the product display shelf and only monitor the movement of the customer's hands, images cannot be taken even if the camera is located below the product display shelf and is positioned to look upwards. It doesn't give you the stress of being watched.

With conventional camera systems using area sensors, it was necessary to extract the customer's movements in real time from a huge amount of image information, and to understand which products the customer grabbed based on the movements in their hands. However, the product display shelf of the present invention does not handle a huge amount of image information, and the computational load is significantly reduced. In addition, it is possible to understand the process of picking up a product and returning it to its original location, making it easy to conduct a survey of customer purchase intentions, which is not possible with current POS systems.

Since it is possible to know when a product is picked up from a product display shelf and placed in a cart inside the store, checking the information with the information at the time of checkout helps prevent shoplifting and facilitates the realization of unmanned stores.
The feature is that it uses a linear sensor camera, which can significantly reduce the amount of information for image processing by acquiring information only when a product is picked up from the shelf or returned to the shelf. It is. This makes it possible to build unmanned stores with a simple system, contributing to the declining birthrate and aging population.

According to the present invention, when work associated with moving products in and out of a product display shelf occurs, by narrowing down the information and processing it, the computational load can be significantly reduced. Additionally, the process of picking up the product and returning it to its original location can be grasped, making it easy to conduct a survey of customer purchase intentions.

By monitoring the flat surface in front of product display shelves, blind spots are eliminated, and by combining this with the customer's movement route within the store, it is possible to provide a simple system to prevent shoplifting, making it easier to realize unmanned stores.

FIG. 1 is a diagram showing the configuration of a product display shelf using a linear sensor camera according to a first embodiment of the present invention. FIG. 2 is a diagram showing the position of the hand performing the work, captured at each timing by the linear sensor camera in FIG. 1; 3 is a diagram showing the pixel signal output of the linear sensor and the reproduced image at each timing in FIG. 2. FIG. FIG. 7 is a configuration diagram for acquiring product information from a shelf position where a task occurs in the second embodiment of the present invention. FIG. 5 is an explanatory diagram of the positional relationship between the linear sensor camera on the product display shelf in FIG. 4 and the hand that is performing the work. FIG. 5 is a diagram showing the position of the hand performing the work, captured at each timing by the linear sensor camera in FIG. 4; 7 is a diagram showing pixel signal outputs of a plurality of linear sensor cameras at each timing in FIG. 6. FIG. FIG. 6 is a diagram showing the relationship between the pixel signal outputs of the plurality of linear sensor cameras in FIG. 5 and the position of the hand. 4 is a diagram showing actual pixel signal output of a linear sensor with background light at each timing in FIG. 3. FIG. FIG. 10 is an explanatory diagram of a product display shelf showing one method of reducing the influence of background light in FIG. 9 in the third embodiment of the present invention. FIG. 10 is a configuration diagram in which a masking signal is generated based on a pixel difference absolute value signal acquired using line recording in a fourth embodiment of the present invention, and only the timing at which work occurs and a work image are grasped. 12 is an explanatory diagram of the pixel signal output of FIG. 11 and the difference mechanism using line storage, the pixel difference absolute value output, the masking signal output, and the masking pixel output at the excerpt timing of FIG. 9; FIG. 12 is an explanatory diagram of pixel signal output, pixel difference absolute value output, and masking signal output in FIG. 11 at each timing in FIG. 9; FIG. FIG. 7 is a configuration diagram for acquiring a pixel difference absolute value signal using a line memory in a fifth embodiment of the present invention. 15 is an explanatory diagram of the pixel signal output of FIG. 14, the difference mechanism using the line memory, and the pixel difference absolute value output at the excerpt timing of FIG. 3; FIG. 15 is an explanatory diagram of pixel signal output and pixel difference absolute value output in FIG. 14 at the excerpt timing of FIG. 9; FIG. 15 is an explanatory diagram of pixel signal output and pixel difference absolute value output in FIG. 14 at each timing in FIG. 9; FIG. 15 is a configuration diagram in which a masking signal is generated based on the pixel difference absolute value signal acquired using the line memory of FIG. 14 to grasp only the timing at which work occurs and the work image in the fifth embodiment of the present invention; FIG. At the excerpt timing shown in FIG. 9, a motion contour edge information output is generated based on the pixel difference absolute value output in FIG. 18, and an explanatory diagram of the masking signal output and masking pixel output generated based on the motion contour edge information output. . 19 is an explanatory diagram of pixel difference absolute value output and masking signal output in FIG. 18 at each timing in FIG. 9; FIG. 19 is an explanatory diagram of pixel signal output and masking pixel signal output in FIG. 18 at each timing in FIG. 9; FIG. 19 is an explanatory diagram of the pixel difference absolute value, masking signal, and masking pixel signal output in FIG. 18 at each timing in FIG. 9. FIG. FIG. 5 is an explanatory diagram of a method of detecting the position of the shelf where work has occurred using the arrangement of the plurality of linear sensor cameras shown in FIG. 4, acquiring product image data, and acquiring product information in the sixth embodiment of the present invention. FIG. 7 is an explanatory diagram of an actual sales floor according to the seventh embodiment of the present invention, where a plurality of product display shelves according to the sixth embodiment are arranged, a cash register is arranged, and a plurality of customers are on the sales floor. An explanatory diagram of the structure of an in-store cart according to the eighth embodiment of the present invention, which shows an example of a method for acquiring in-store cart information in addition to the seventh embodiment of the present invention. FIG. 7 is an explanatory diagram of an operation in which a CCD register is used as a line memory in a fifth embodiment of the present invention. FIG. 7 is an explanatory diagram of an operation in which a CMOS linear sensor is used as a line memory in a fifth embodiment of the present invention.

Several selection cases are possible for embodiments of the invention. The cases include: 1) A case of a product display shelf that uses a linear sensor camera to monitor the work involved in entering and exiting products. 2) A case where multiple linear sensor cameras are used to identify the shelf position where work has occurred and obtain product information corresponding to this shelf position. 3) A case where a non-reflective plate is installed on the edge of the product display shelf corresponding to the background in order to suppress the output of the background when monitoring the work occurring with a linear sensor camera. 4) A case in which the line pixel signal output of the linear sensor is compared when no work occurs and when work occurs, and only the timing at which work occurs and the work image are determined. 5) A case in which the signal outputs of adjacent line pixels of the linear sensor are compared to grasp only the timing at which work has occurred and the work image. 6) A case where multiple linear sensor cameras are used to identify the shelf position where work has occurred and product information is obtained from product image data. 7) Cases in which product information obtained at the time of work is linked, information on customers circulating around the sales floor, and checkout information of the customers at the cash register; 8) A mechanism for monitoring work associated with the entry and exit of goods into in-store carts. In some cases, this in-store cart information is also added to the product information acquired at the time of the work, information on customers circulating the sales floor, and transaction information of the customer at the cash register.

Since the product display shelf according to the embodiment of the present invention can be used in several cases as described above, each case will be explained with reference to the drawings. The claims also apply to each case, and the cases applicable to each claim and the corresponding drawings will be described later.

In the following explanation, a case of a product display shelf in which the linear sensor camera of Embodiment 1 is installed will be used. We will explain the hands involved in the work and the images reproduced by the linear sensor.
In the following description, the same parts will be given the same reference numerals and process names, a detailed explanation will be given first, and a redundant description of the same parts will be omitted.

<Embodiment 1 of product display shelf>
FIG. 1(a) shows an example of the overall configuration of a product display shelf according to Embodiment 1 of the present invention. The product display shelf 1 is composed of a plurality of display shelves 2, 2', 2", and 2"' with different heights, and products 3, 3', 3", and 3"' are arranged on the shelves. A linear sensor camera 4 for observing the front surface of the product display shelf 1 is installed at the upper edge of the front surface of the product display shelf 1. On the front of the product display shelf 1, a customer's hand 5 is shown performing work associated with putting in and taking out products.

FIG. 1(b) shows the structure of the linear sensor camera 4. The linear sensor camera 4 has a linear sensor 7 disposed on a substrate on the bottom surface of a camera housing 6, and a lens 8 attached to the front surface of the camera housing 6 forms an image of a subject on the linear sensor 7. The linear sensor 7 is electrically coupled to an electrode wiring portion (not shown) of the substrate by a metal wire 9.

FIG. 1(c) shows the structure of a linear sensor 7 that is a CCD linear sensor. The linear sensor 7 is constructed on a semiconductor substrate, and has pixels 10, which are photoelectric conversion units, arranged in a straight line in the center, and CCD registers 11, 11' arranged on both sides of the pixel column. Shift electrodes 12 and 12' are provided between the pixel 10 and the CCD registers 11 and 11', respectively, for controlling the transfer of signal charges generated by photoelectric conversion in the pixel to the CCD registers 11 and 11'. . The signal charges transferred to the CCD registers 11, 11' are transferred in the direction of the output circuit 13, where the signal charges are converted into a signal voltage and output from the electrodes 14 to the outside as a pixel signal output. This electrode 14 is electrically coupled to an electrode wiring portion (not shown) of the substrate by a metal wire 9.

In FIG. 1(c), the linear sensor 7 is explained in which the CCD registers 11 and 11' are arranged on both sides of the pixel column, but of course a structure with only one CCD register on one side may also be used. Instead of a sensor, a CMOS linear sensor may be used.

FIG. 2 shows a side view of the product display shelf 1 shown in FIG. 1(a). A linear sensor camera 4 for observing the front surface of the product display shelf 1 is installed at the upper edge of the front surface of the product display shelf 1. On the front of the product display shelf 1, a customer's hand 5 is shown performing work associated with putting in and taking out products. Since the pixels 10 of the linear sensor camera 4 are arranged in a straight line, the range that can be photographed is limited to the plane in front of the product display shelf 1.

This plane is shown by a broken line extending from the linear sensor camera 4 at timing t= _T0 . At the timing of t=T ₀ , the customer's hand 5 has not yet reached this plane. The customer's hand 5 approaches to pick up the product on the product display shelf 1, but for the sake of explanation here, we will focus on which part of the customer's hand 5 crosses the broken line extending from the linear sensor camera 4 (linear sensor The regions that can be photographed with the camera 4 are indicated by timings t=T ₀ , t=T ₁ , t=T ₂ , t=T ₃ , and t=T ₄ .

In FIG. 2, timings t=T ₀ , t=T ₁ , t=T ₂ , t=T ₃ , and t=T ₄ indicate which part of the customer's hand 5 can be photographed by the linear sensor camera 4. . Pixel output signals from the linear sensor camera 4 at each timing are shown in FIGS. 3(a), (b), (c), (d), and (e). Here, the pixel signal output is the output of the pixel row of the linear sensor 7 in FIG. 1(c), the horizontal axis shows the pixel position, and the vertical axis shows the pixel signal output at each pixel position.

The pixel signal output represents a signal output corresponding to one line of a linear sensor in which all pixels of the linear sensor 7 are read out and there is idle feeding before and after, and each line is synchronized. The pixel difference that will appear in the following explanation is a synchronized difference between these lines, and means a difference between pixels located at the same position.

A feature of the linear sensor camera 4 is that when there is no movement, the background is a constant output, which is set to zero here for convenience (see FIG. 3(a)). When there is movement (here, movement of the customer's hand 5), a change occurs in the output of the linear sensor camera 4. Corresponding to the movement of the hand, an image is taken on a plane indicated by a broken line extending from the linear sensor camera 4 at timing t=T ₀ , and the pixel output waveform changes depending on the part of the customer's hand 5 that crosses this plane. occurs. (See FIGS. 3(b), (c), (d), and (e)).

If the pixel signal outputs of the linear sensor camera 4 are arranged in chronological order, a reproduced image of the customer's hand 5 can be obtained. FIG. 3 shows a reproduced image obtained from the pixel output waveform corresponding to the part of the customer's hand 5. Here, the reproduced image expands or contracts depending on the speed of movement of the customer's hand 5, but since the aspect ratio (length and width) of the human hand is approximately constant, the reproduced image can be easily corrected. Furthermore, even if the customer 5's hand grabs the product 3 on the product display shelf 1 and moves away from the product display shelf 1, the reproduced image can be easily obtained, and the reproduced image of the product can also be obtained. If the aspect ratio of the customer's hand is corrected, the distortion of the product's external shape due to the speed of movement of the customer's hand 5 can be easily corrected since the hand 5 grabs the product 3 and comes out together (at the same speed). It can be corrected to

<Embodiment 2 of product display shelf>
FIG. 4 shows an example of the overall configuration of a product display shelf according to Embodiment 2 of the present invention. Similarly to FIG. 1(a), the product display shelf 1 is composed of a plurality of display shelves 2, 2', 2", 2"' with different heights, and products 3, 3', 3" are placed on each shelf. , 3''' are arranged, and on the front side of the product display shelf 1, a customer's hand 5 is shown performing work associated with putting in and taking out products.
The difference from FIG. 1A is that a plurality of linear sensor cameras 4 for observing the front surface of the product display shelf 1 are installed on the front edge of the product display shelf 1. On the front of the product display shelf 1, a customer's hand 5 is shown performing work associated with putting in and taking out products. FIG. 4 shows an example in which four linear sensor cameras 4 (indicated by cameras 1 to 4) are arranged at four corners of the product display shelf 1.

A front view, a top view, and a side view of the product display shelf 1 according to the second embodiment shown in FIG. 4 are shown in FIGS. 5(a), (b), and (c), respectively. Display shelves 2, 2', 2", 2"', products 3, 3', 3", 3"', and linear sensor cameras 4 (cameras 1 to 4) are also shown in the same manner as in FIG.
In each case, the customer's hand 5, which performs the work associated with taking in and out the product, is shown, but as shown by the broken line in FIG. The customer's hand 5 is photographed in different directions with respect to the linear direction.

In the side view of the product display shelf 1 shown in FIG. 5(c), a customer's hand 5 is shown. As shown again in FIG. 6, the linear sensor cameras 4 located at the upper and lower edges of the product display shelf 1 monitor the front plane of the product display shelf 1, and this plane is defined by a broken line extending from the linear sensor camera 4. It is shown in The customer's hand 5 approaches to take the product on the product display shelf 1, but at the timing of t= _T0 shown in FIG. It has not been reached yet. Therefore, the pixel output waveforms of cameras 1 to 4 do not appear.
On the other hand, at the timing of t= _T1 shown in FIG. 7(b), the tip of the customer's hand 5, shown by the solid line, reaches this plane. Therefore, a pixel output corresponding to the tip of the customer's hand 5 appears at a pixel position corresponding to the misalignment between the normal direction of the cameras 1 to 4 and the direction of the tip of the customer's hand 5.

To explain more specifically using FIG. 8(a), the linear sensor cameras 4 located at the four corners of the product display shelf 1 face the direction of the customer's hand 5, as shown by cameras 1 to 4. There is. The angle at which the tip of the customer's hand 5 is viewed differs between cameras 1 to 4. In the figure, the angles formed by the directions of the tips of the customer's hands 5 as seen from the cameras 1 and 2 with the upper edge of the product display shelf 1 are indicated by θ ₁ and θ ₂ , respectively.

Normally, light coming from the normal direction corresponding to the optical axis direction of the linear sensor camera 4 is assembled so as to be focused near the center of the pixel row of the linear sensor 7. For this reason, the pixel output appears on the front side of the pixel row for a subject on the left side from the front of the camera (case of camera 1), and the pixel output appears on the rear side of the pixel column for a subject on the right side from the front of the camera (case of camera 2). will begin to appear. However, if you reverse the direction of the linear sensor, the front and back will also be reversed.
FIG. 8(b) shows the pixel signal output waveforms of cameras 1 to 4. The appearance positions of the pixel outputs corresponding to the tip of the customer's hand 5 correspond to the front side and the rear side, and for camera 1, L ₁ , For camera 2, L ₂ and L ₁ <L ₂ .

The appearance positions of the pixel outputs corresponding to the tip of the customer's hand 5, which determine the angles θ ₁ and θ ₂ in FIG. 8(a), are L ₁ and L of the cameras 1 and 2, respectively, shown in FIG. 8(b). It is found corresponding to ₂ . The position of the tip of the customer's hand 5 is uniquely determined from the angles θ ₁ and θ ₂ using the principle of triangulation. By utilizing the cameras 3 and 4, the positional accuracy of the tip portion is further increased.

Based on the uniquely determined position of the tip of the customer's hand 5 using the principle of triangulation shown in FIG. It is possible to determine at what height of the display shelf the user reached for (display shelf 2' in the figure) and which product and what number from the end the user tried to grab (3' in the figure).
Here, in order to identify the product from its position, it is necessary to know in advance (for example, when displaying the product) what product 3 is displayed at what position from the end of the display shelf 2 and at what height. It needs to be tied.

As shown in FIG. 3, when the pixel signal outputs of the linear sensor camera 4 are arranged in chronological order, a reproduced image of the customer's hand 5 is obtained. When the customer's hand 5 grabs the product 3 from the product display shelf 1 and comes out, the product 3 can be read, but the product label cannot be seen due to the way the customer's hand 5 grips the product 3. Judgment may be difficult in some cases, so it is desirable to link the location and product to increase accuracy.
On the other hand, the number of products can be determined from the reproduced image, but depending on how the product 3 is held, it may be difficult to see the number, so the reproduction images from a plurality of linear sensor cameras 4 improve accuracy. To further improve accuracy, a weight sensor can be placed on each display shelf 2 or product display shelf 1 to determine the product name and number of products 3 based on changes in weight, thereby increasing the accuracy.

<Embodiment 3 of product display shelf>
In the explanation of FIG. 3, if there is no movement, the background will be a constant output, and the background has been set to zero for convenience (see FIG. 3(a)). In an actual sales floor, the background does not become zero due to the illumination light inside the store, and in reality, the waveform of the pixel signal output of the background light from the linear sensor camera 4 is as shown in FIG. 9(a).
However, the background is stationary, and in the short period in which the customer's hand movements are captured, it can be considered to be a fixed pattern. Therefore, an image is taken on a plane indicated by a broken line extending from the linear sensor camera 4 at timing t=T ₀ in FIG. The output signals have output waveforms shown in FIGS. 9(b), (c), (d), and (e).

What should be noted here is that the magnitude of the pixel signal output at the position corresponding to the customer's hand 5 is not superimposed on the output waveform of the stationary background, but rather the magnitude of the pixel signal output at the position corresponding to the customer's hand 5 is The pixel signal output of the customer's hand 5 is blocked, the pixel signal output of the background corresponding to the hand becomes zero, and the pixel signal output corresponding to the part of the customer's hand 5 appears. This is expressed using dotted lines in FIGS. 9(b), (c), (d), and (e).

A device for suppressing the waveform of the pixel signal output of the background light of the linear sensor camera 4 as shown in FIG. 9(a) is shown in FIG. Give an example. Taking the product display shelf 1 in FIG. 4 as an example, there are multiple display shelves 2, 2', 2", 2"' with different heights, and products 3, 3', 3", 3"' on each shelf. The linear sensor camera 4 (cameras 1 to 4) and the customer's hand 5 are the same as in FIG.
The difference from FIG. 4 is that a non-reflective plate is placed on the flat surface of the front surface of the product display shelf 1 monitored by cameras 1 to 4, and at the edge of the front surface of the product display shelf 1 so as to face the camera. 15. In FIG. 10, the non-reflective plate is expressed as 15, and the back surface of the non-reflective plate is expressed as 15'.

In the product display shelf 1 of FIG. 10, the waveform of the pixel signal output of the background light of the linear sensor camera 4 (cameras 1 to 4) is such that even if there is illumination light inside the store, this non-reflective plate 15 is photographed. , the background light where there is no customer's hand 5 can be suppressed, and the background can be made zero for convenience, as shown in FIG. This configuration is referred to as a product display shelf 1 according to Embodiment 3 of the present invention.

<Embodiment 4 of product display shelf>
In the case of the product display shelf 1 according to the first embodiment of the present invention shown in FIG. 1(a), in which there is only one linear sensor camera 4, the influence of the background can be eliminated without using the non-reflective plate 15. The method will be explained with reference to FIG.
FIG. 11 shows a signal processing circuit that processes signals from the output of the linear sensor camera 4, extracts only the portion corresponding to the customer's hand 5, and makes the background output zero. A product display shelf 1 using the signal processing method of the linear sensor camera 4 according to this method is referred to as a product display shelf 1 according to Embodiment 4 of the present invention.

The signal processing circuit shown in FIG. 11 forms an image of the subject on a linear sensor in the camera housing 6 using a lens 8, and outputs the obtained pixel signal output 16 from the linear sensor camera 4. From the output pixel signal output 16, a pixel difference absolute value 22 is obtained in the line difference processing section 17 (more precisely, it is a pixel difference absolute value signal output, but the signal output is omitted in the following explanation).

Next, the configuration of the line difference processing section 17 will be explained. A portion of the pixel signal output 16 is taken into the line storage section 19 via the switch circuit 18. Some of the pixel signal outputs 16 (referred to as line storage outputs) coming out of the line storage section 19 are subtracted from the pixel signal output 16 by a difference circuit 20, and then passed through an absolute value circuit 21 to obtain the pixel difference absolute value. 22 is obtained.

Here, the part of the pixel signal output 16 (line storage section output) is the waveform of the pixel signal output of the background light of the linear sensor camera 4 at the timing t= _T0 in FIG. 9(a). In the switch circuit 18, when storing the pixel signal output 16 of the background light in which the output of the customer's hand 5 does not appear, the switch is turned ON and the pixel signal output 16 is stored in the line storage section 19, and the output of the customer's hand 5 is stored. The pixel signal output 16 that is present is turned off and is not taken into the line storage section 19. In this way, the difference circuit 20 can remove background light.
The difference circuit 20 has a positive or negative output depending on the magnitude relationship of the pixel signal outputs between the lines to be compared. Therefore, the output of the difference circuit is input to an absolute value circuit 21 to obtain a pixel difference absolute value 22.

The pixel difference absolute value 22 obtained by the line difference processing section 17 is input to the masking signal generation section 23. The masking signal generator 23 is a circuit that generates a masking signal 26 for extracting the timing at which the customer's hand 5 moves and the pixel output timing corresponding thereto.

Next, the configuration of the masking signal generator 23 will be explained. The pixel difference absolute value 22 is compared with a threshold value 24 in a masking signal generation circuit 25 and binarized to obtain a masking signal 26. The masking signal 26 that is the output of the masking signal generator 23 is further processed by the masking circuit 27 from the pixel signal output 16 that is the output from the linear sensor camera 4 to the masking pixel corresponding only to the movement of the customer's hand 5. We get output 28.

At this time, if a problem occurs in extracting only the movement of the customer's hand 5 from the pixel signal output 16 due to the calculation processing time of the line difference processing unit 17 and the masking signal generation unit 23, the calculation time will be The pixel signal output 16 may be delayed by the delay circuit 29 by a time corresponding to , and the masking pixel output may be obtained by the masking circuit 27.

The line difference and masking method shown in FIG. 11 will be explained using FIG. 12 based on each signal waveform. The timing of the position of the customer's hand 5 is the case of t=T ₀ , t=T ₁ , and t=T ₃ shown in FIGS. 9(a), (b), and (d).
First, at the timing of t= _T0 , which corresponds to the background image, as shown in FIG. Since the switch circuit is ON, the signal is captured in the line storage section 19. This line storage section output is an output at the timing of t=T ₀ shown in FIGS. 12(b) and 12(d). Since this waveform is stored in the line storage section 19, it appears as a background image at other timings as well. In FIGS. 12(b) and 12(d), this is expressed as line storage unit output.

Next, when the customer's hand 5 comes at timing t= _T1 in FIG. 2, the waveform becomes as shown in FIG. 9(b), and FIG. 12(b') shows the pixel signal output. At this timing t= _T1 , the difference circuit 20 subtracts this pixel signal output from the line storage section output corresponding to the background image shown in FIG. 12(b) at the timing t= _T0 , After passing through the absolute value circuit 21, the pixel difference absolute value 22 at the timing t= _T1 shown in FIG. 12(b'') is obtained.
If the customer's hand 5 is not recognized after t=T ₀ and before t=T ₁ , the pixel signal output has the same waveform as in FIG. The absolute difference value 22 remains zero.

At this timing t= _T1 , the pixel difference absolute value 22 is compared with the threshold value 24 in the masking signal generation unit 23 and binarized, and if the pixel difference absolute value output value exceeds the threshold value 24, the pixel difference absolute value 22 is , otherwise 0 is assigned and a masking signal is obtained. FIG. 12(b"') shows the masking signal at the timing of t= _T1 .

The period during which this masking signal 26 is 1 corresponds to the timing at which the movement of the customer's hand 5 occurs and the pixel output timing corresponding to the position of the customer's hand 5, and masking is performed from the pixel signal output 16. This is the appropriate period.
That is, the pixel signal output 16 at the timing when the masking signal 26 is generated (becomes 1) corresponds to seeing only the movement of the customer's hand 5, and is shown as the masking pixel output 28 in FIG. 12(b). ""). This is the signal of the customer's hand 5 at the timing t= _T1 in the case where there is no background, as shown in FIG. 3(b).

In a similar sequence, the customer's hand 5 outputs a pixel signal as shown in FIG. 12(d') at the timing t= _T3 in FIG. 2, similar to FIG. 9(d). At this timing t=T ₃ , the difference circuit 20 subtracts this pixel signal output from the line storage section output corresponding to the background image in FIG. 12(d) at timing t=T ₀ , and the absolute value After passing through the circuit 21, the pixel difference absolute value 22 at the timing of t= _T3 shown in FIG. 12(d") is obtained. Similarly, the masking signal is shown in FIG. 12(d"') and the masking pixel The output 28 is shown in FIG. 12(d"") and is the signal of the customer's hand 5 at timing t= _T3 in the absence of background, as shown in FIG. 3(d).

The above description has been made regarding timings t=T ₀ , t=T ₁ , t=T ₃ , but other timings are also included, such as timings t=T ₀ , t=T ₁ , t=T _{2 , and t=T 2} shown in FIG. For the waveforms of pixel signal outputs shown in FIGS. 9(a), (b), (c), (d), and (e) corresponding to t=T ₃ and t=T ₄ , the pixel difference absolute value and masking are Figures 13(a), (b), (c), (d), and (e) show how the signals look.

<Embodiment 5 of product display shelf>
In the case of the product display shelf 1 according to the first embodiment of the present invention shown in FIG. 1(a), in which there is only one linear sensor camera 4, the influence of the background can be eliminated without using the non-reflective plate 15. Another method will be explained with reference to FIG.
FIG. 14 shows a signal processing circuit that processes signals from the output of the linear sensor camera 4, extracts only the portion corresponding to the customer's hand 5, and makes the background output zero. A product display shelf 1 using the signal processing method of the linear sensor camera 4 according to this method will be referred to as a product display shelf 1 according to Embodiment 5 of the present invention.

The signal processing circuit shown in FIG. 14 forms an image of the subject on a linear sensor in the camera housing 6 using a lens 8, and outputs the obtained pixel signal output 16 from the linear sensor camera 4. A pixel difference absolute value 22 is obtained from the output pixel signal output 16 in a line difference processing section 17 .

The difference from FIG. 11 is the configuration of the line difference processing section 17. Pixel signal output 16 is taken into line memory 30. The pixel signal output 16 coming out from the line memory 30 is the line output of the previous line, and the difference between it and the current pixel signal output 16 is taken in the difference circuit 20, and the pixel difference absolute value 22 is obtained through the absolute value circuit 21. can get. The time corresponding to one line is the cycle time (time taken to read one line) of the linear sensor 7, and is denoted as Δt in the subsequent drawings. That is, the difference circuit 20 calculates the difference between adjacent lines.
The line difference processing section 17 in FIG. 14 has a line memory 30 instead of the line storage section 19, and does not have the switch circuit 18 that was in FIG.
Since the pixel signal output from the line memory 30 is the line output of the previous line, the line memory 30 has the function of an analog delay circuit that has the function of delaying the pixel signal by a time corresponding to one line. For this reason, it is called a line memory, but the pixel signal output 16 of the background light is stored as one line output. Although the name is different from the line storage section 19, the function is the same in terms of storing one line output. For this reason, the line memory 30 can also be referred to as the line storage section 19.

Of course, the line memory 30 does not have to be a line memory that corresponds to a delay of one line, but may generate a delay equivalent to two lines and take the difference by skipping one line. The amount of line delay is determined by the balance between the speed at which the customer's hand 5 moves in and out and the cycle time of the linear sensor 7. In order to simplify the system, the delay time of the line memory is generally set to an integral multiple of the cycle time, but in the following, the case of the difference between adjacent lines corresponding to the delay (Δt) of one line will be explained.

The line difference method shown in FIG. 14 will be explained using FIG. 15 based on each signal waveform. For the sake of simplicity, we will first consider the timing of the position of the customer's hand 5 in the case where there is no background light, t=T ₀ , t=T as shown in FIGS. 3(a), (b), and (d). ₁ , t= _T3 .
First, at the timing of t= _T0 , as shown in FIG. 15(a), there is no output from the customer's hand 5 in the waveform of the pixel signal output 16 of the linear sensor camera 4. At this time, the waveform of the pixel signal output of the previous line (T _{0 -Δt} ) that comes out from the line memory 30 also has no output from the customer's hand 5, as shown in FIG. 15(a'). Even if the difference between the two is taken by the difference circuit 20, it is zero, and even after passing through the absolute value circuit 21, the pixel difference absolute value 22 remains zero as shown in FIG. 15(a'').

Next, at timing t= _T1 , the output of the customer's hand 5 appears in the waveform of the pixel signal output 16, as shown in FIG. 15(b). At that time, the output of the customer's hand 5 also appears in the waveform of the pixel signal output of the previous line (T _{0 -Δt} ), which is output from the line memory 30, as shown in FIG. 15(b'). The waveforms of both are slightly different depending on the insertion position of the customer's hand 5, and a difference generally appears in the outline of the output of the hand. Therefore, when the difference between the two is taken by the difference circuit 20, an output like a differential waveform appears in this contour portion. After passing through the absolute value circuit 21, the pixel difference absolute value 22 becomes as shown in FIG. 15(b''). This has a shape similar to the differential waveform of the pixel signal output.

In a similar sequence, when the customer's hand 5 comes at the timing t= _T3 in FIG. 2, the output of the entire customer's hand 5 appears as shown in FIG. 15(d). At that time, the waveform of the previous line (T _{3 - Δt} ) that comes out from the line memory 30 is also the output of the entire customer's hand 5, as shown in FIG. 15(d'), and the two are slightly different. . When the difference between the two is taken by the difference circuit 20, in this case, in addition to the outline, the area between the fingers also appears subtly. After passing through the absolute value circuit 21, the pixel difference absolute value 22 becomes as shown in FIG. 15(d'').

In Fig. 15, the case where there is no background light was explained for simplicity, but next, the case with background light at t=T ₁ and t=T ₃ as shown in Figs. 9(b) and (d) will be explained. .
At the timing of t= _T1 , the output of the customer's hand 5 appears in the waveform of the pixel signal output 16, as shown in FIG. 16(b). At that time, the output of the customer's hand 5 also appears in the waveform of the pixel signal output of the previous line (T _{0 - Δt} ), which comes out from the line memory 30, as shown in FIG. 16(b'), and both of them are The waveforms are slightly different. Therefore, when the difference between the two is taken by the difference circuit 20, the background light disappears because the area is the same, and an output like a differential waveform appears in the outline of the hand, and after passing through the absolute value circuit 21, the pixel difference absolute value 22 becomes The result is as shown in FIG. 16(b"). This is similar to the case without background light shown in FIG. 15(b").

In a similar sequence, at the timing of t=T ₃ , the entire hand appears as shown in _FIG . As shown in FIG. 16(d'), the output of the entire hand is slightly different between the two. When the difference between the two is taken by the difference circuit 20, in addition to the outline part, the part between the fingers also appears subtly, and the pixel difference absolute value 22 becomes as shown in FIG. 16(d").FIG. 15(d") ) is similar to the case without background light.

The above description has been made regarding the timings t=T ₁ and t=T ₃ , but other timings are also included, such as the timings t=T ₀ , t=T ₁ , t=T ₂ , t=T _{3 , and t=T 3} shown in FIG. What is the pixel difference absolute value 22 for the waveform of the pixel signal output 16 shown in FIGS. 9(a), (b), (c), (d), and (e) corresponding to t=T ₄ ? This is shown in FIGS. 17(a), (b), (c), (d), and (e).

FIG. 18 shows a signal processing circuit that performs signal processing on the output of the linear sensor camera 4 of the product display shelf 1 according to Embodiment 5 of the present invention, extracts only the part corresponding to the customer's hand 5, and makes the background output zero. show. Since the pixel difference processing section 17 has already been explained with reference to FIG. 14, the masking signal generation section 23 will be explained.

In the pixel difference processing unit 17, the pixel difference absolute value 22 is obtained by taking the difference between the timing at which the movement of the customer's hand 5 occurs and the timing one line before. Based on this pixel difference absolute value 22, the masking signal generation unit 23 determines the timing at which the movement of the customer's hand 5 occurs (the meaning of both the timing of the movement occurrence and the output timing of the pixel corresponding to the location of the movement). A masking signal 26 is generated for extracting the pixel output timing corresponding to the pixel output timing.

Next, the configuration of the masking signal generator 23 will be explained. From the pixel difference absolute value 22, a motion contour end determining section 31 extracts the outer contour of the customer's hand 5. As explained with reference to FIG. 16(d''), in addition to the outline of the hand, the area between the fingers also appears from the difference circuit 20, and the pixel difference absolute value 22 has a complex waveform.
The motion contour end determining section 31 can extract the outer contour of the customer's hand 5 by following the progress of the pixel difference absolute value waveform.

The method will be explained, but the premise is that the time from when the customer's hand 5 enters the flat surface of the front surface of the product display shelf 1 monitored by the linear sensor camera 4 until it is pulled out is captured as a series of movements. . Generally speaking, in order to grasp product 3 on product display shelf 1, a customer's hand inserts the hand, grasps the product, and then pulls it out as a series of movements, so this premise holds. It doesn't stop inside the display shelf, and you don't insert anything other than your hands (head, feet, etc.)

The tip of the customer's hand 5 is captured by the linear sensor camera 4, and gradually expands to the width of the hand. As shown in Fig. 3(c), the part between the fingers also appears, but it remains within the width of the hand, and the outline of the outside of the hand can be followed. As shown in FIG. 3(d), there is a temporary gap between the thumb and the hand, making it impossible to distinguish it from a new moving object, but as shown in FIG. 3(e), they are connected and can be recognized as a single body.
Since the size of the customer's hand 5 is smaller than the size of the product display shelf 1 observed by the linear sensor camera 4, a new moving object (fingertip) that appears near the hand is attributed to the hand. If a criterion is provided, it becomes easier to determine the outermost contour of the hand (motion contour end determination).

The motion contour edge information 32 corresponding to the outer contour of the customer's hand 5 obtained by the motion contour edge determination section 31 in this way is processed by the masking signal generation circuit 25 to a threshold value 33 (the threshold value in FIG. (discriminated) to obtain a masking signal 26. The masking signal 26 that is the output of the masking signal generator 23 is further processed by the masking circuit 27 from the pixel signal output 16 that is the output from the linear sensor camera 4 to the masking pixel corresponding only to the movement of the customer's hand 5. We get output 28.

The pixel difference processing and masking method shown in FIG. 18 will be explained using FIG. 19 based on each signal waveform. The timing of the position of the customer's hand 5 is the case of t=T ₁ and t=T ₃ shown in FIGS. 9(b) and (d), and the case shown in FIG. 17(b) and (d) Corresponds to pixel signal output and pixel difference absolute value.

When the customer's hand 5 comes at timing t= _T1 , the pixel difference absolute value waveform becomes as shown in FIG. 19(b). The motion contour end information 32 at this time is as shown in FIG. 19(b'). The motion contour edge information 32 is the edge of the contour of one moving object, two of which form a pair, and the region of one moving object is determined by the two pairs.
The motion contour edge information 32 is compared with a threshold value 33 in a masking signal generation circuit 25, and masking area setting and binarization are performed. This masking area is an area of one moving object. If the motion contour edge information 32 exceeds the threshold value 33 and is recognized as a region to be masked, 1 is assigned, otherwise 0 is assigned, and a masking signal 26 is obtained. The masking signal 26 obtained based on the motion contour edge information 32 of FIG. 19(b') becomes as shown in FIG. 19(b"').

The period during which this masking signal 26 is 1 corresponds to the timing at which the movement of the customer's hand 5 occurs and the pixel output timing corresponding to the position of the customer's hand 5, and masking is performed from the pixel signal output 16. This is the appropriate period. That is, the pixel signal output 16 at the timing when the masking signal 26 is generated (becomes 1) corresponds to seeing only the movement of the customer's hand 5.
The masking pixel output 28 obtained by the masking circuit 27 from the masking signal 26 and the pixel signal output 16 shown in FIG. 19(b'') is shown in FIG. 19(b''). This corresponds to FIG. 3(b), which is the pixel signal output at the timing of = _T1 .

In a similar sequence, when the customer's hand 5 is at the timing t= _T3 in FIG. 2, the pixel difference absolute value of FIG. , the part between the fingers is determined not to be an end and is deleted, resulting in a result as shown in FIG. 19(d'). At this time, the gap between the thumb and other fingers is not yet determined to be between the fingers, and the thumb is determined to be a new moving object. Both are determined to be the same moving object based on the timing of the base of the thumb. The masking signal 26 obtained based on the motion contour edge information 32 in FIG. 19(d') becomes as shown in FIG. 19(d"').
The masking pixel output 28 obtained by the masking circuit 27 from the masking signal 26 and the pixel signal output 16 shown in FIG. 19(d'') is shown in FIG. 19(d''). This corresponds to FIG. ₃ (d), which is the pixel signal output at the timing of =T3.

Here, the judgment of the gap between the fingers is based on the initial edge information determined from the fingertip at the tip of the customer's hand 5, and a new moving object that occurs in the vicinity of this edge corresponding to the thickness of the finger is , when defined as other fingertips, it is recognized as one hand. If we expand this further and set a criterion that a new moving object (fingertip) that appears in the vicinity equivalent to the width of the hand belongs to the hand, we can determine that a new moving object has appeared in Figure 19 (d"'). Even if the thumb is moved, it is not determined that a new moving object has appeared, and the outline of the outermost part of the hand can be easily determined.This is the criterion mentioned in paragraph number 76.

The above description has been made regarding the timings t=T ₁ and t=T ₃ , but other timings are also included, such as the timings t=T ₀ , t=T ₁ , t=T ₂ , t=T _{3 , and t=T 3} shown in FIG. What is the pixel difference absolute value 22 for the waveform of the pixel signal output 16 shown in FIGS. 9(a), (b), (c), (d), and (e) corresponding to t=T ₄ ? This is shown in FIGS. 17(a), (b), (c), (d), and (e).Based on this, how will the motion contour edge information 32 and the masking signal 26 become? This is shown in FIGS. 20(a), (b), (c), (d), and (e).

9(a), (b), (c), (d) corresponding to the timings t=T ₀ , t=T ₁ , t=T ₂ , t=T ₃ , t=T ₄ shown in FIG. 2, Based on the waveform of the pixel signal output 16 shown in (e) and the waveform of the masking signal 26 shown in FIGS. The masked pixel outputs 28 thus obtained are shown in FIGS. 21(a), (b), (c), (d), and (e) at respective timings. The waveform of the original pixel signal output 16 is also included in the figure.
Furthermore, timings t=T ₀ , t=T ₁ , t=T ₂ , t=T ₃ , and t=T ₄ correspond to the parts of the customer's hand 5 indicated by broken lines.

In paragraph number 82, a case will be described with reference to FIG. 22 in which a criterion is set that a new moving object (fingertip) that occurs in the vicinity corresponding to the width of the hand belongs to the hand. 9(a), (b), (c), and (d) corresponding to timings t=T ₀ , t=T ₁ , t=T ₂ , t=T ₃ , and t=T ₄ shown in FIG. 2 , 17(a), (b), (c), (d), and (e) show how the pixel difference absolute value 22 becomes for the waveform of the pixel signal output 16 shown in , (e). This is shown again in FIGS. 22(a), (b), (c), (d), and (e). The waveforms of the motion contour edge information 32 determined by the motion contour edge determination unit 31 to be an edge based on the new determination criteria based on the waveform of the pixel difference absolute value 22 are as shown in FIGS. 22(a) and 22(b) at each timing. , (c), (d), and (e), which are the differential waveforms at both ends of several differential waveforms of the absolute value of the pixel difference shown on the left side.

Based on the waveform of the motion contour edge information 32 obtained by the motion contour edge determination section 31 based on the new determination criteria, the masking signal generating circuit 25 determines that the masking signal is 1 from the time when the threshold value 33 is exceeded. 26 is the masking signal waveform shown in the center of FIGS. 22(a), (b), (c), (d), and (e). The masking signal waveform corresponds to the size of the customer's hand 5.

9(a), (b), (c), (d) corresponding to the timings t=T ₀ , t=T ₁ , t=T ₂ , t=T ₃ , t=T ₄ shown in FIG. 2, Based on the waveform of the pixel signal output 16 shown in (e) and the waveform of the masking signal 26 shown in FIGS. The masked pixel outputs 28 thus obtained are shown in FIGS. 22(a), (b), (c), (d), and (e) at respective timings. Comparing this waveform with the waveform of the masking pixel output 28 shown in FIGS. 21(a), (b), (c), (d), and ( _e ), it is found that d) and FIG. 21(d). That is, in the waveform of the masking signal 26 obtained using the new criterion, the area between the fingers is recognized as part of the customer's hand 5 and is adopted as an image, so the masking signal becomes 1. Therefore, the background image appears in the area between the fingers in the masking pixel output 28 in FIG. 22(d). This is different from the masking pixel output 28 shown in FIG. 21(d), which is obtained with zero masking signal between the fingers in FIG. 20(d).

<Embodiment 6 of product display shelf>
Only the part corresponding to the customer's hand 5 is extracted from the image signal output of the linear sensor camera 4 used in the product display shelf 1 according to the fourth embodiment of the present invention shown in FIG. 11, and the background output is set to zero. The present invention provides a product display shelf 1 in which a plurality of cameras using a signal processing method are arranged at the corners of the product display shelf 1 to observe the hands 5 of customers who perform work associated with putting products in and out in front of the product display shelf 1. A product display shelf 1 according to Embodiment 6 of FIG.

Four linear sensor cameras 4 (indicated by cameras 1-4) having the signal processing circuit shown in FIG. 11 are shown in FIG. This linear sensor camera 4 observes the front portion of the product display shelf 1 at the front edge of the product display shelf 1 configured according to the second embodiment of the present invention. The pixel signal output 16 output from each linear sensor camera 4 (cameras 1 to 4) passes through the line difference processing section 17, the masking signal generation section 23, and the masking circuit 27, and the masking pixel output corresponds to the cameras 1 to 4. is output.

A new addition to the signal processing circuit shown in FIG. 23 is that in the masking circuit 27, the pixel signal output 16 is output (denoted as ON) or not (described as OFF) depending on whether the masking signal is 1 or 0. decide. This corresponds to the waveforms explained so far.
Masking here is not used in the commonly used sense of hiding data, but is used in this specification in the sense of hiding other than data and emphasizing and exposing data. The purpose of this application is to hide unnecessary background, eliminate unnecessary data processing, extract essential data (customer's hand 5, product 3), and simplify signal processing.

Also, the explanation was vague about how to switch the switch circuit 18, but when the masking signal is 0, there is no movement of the customer's hand 5 and it corresponds to the background light, so the switch circuit is memorized as an ON line and the masking When the signal is 1, there is a movement of the customer's hand 5, which corresponds to the movement of the customer's hand 5 that should be captured, so the switch circuit is set to OFF and is not stored in line.

The masking signal has a waveform of 1 and 0 within one line, but when we say that the masking signal is 0 or 1 here, it means that the masking signal is always 0 in one line of masking signal. is said to be 0, and if there is even a slight timing at which the masking signal becomes 1, it is said to be 1. Furthermore, since there are few changes in the background light, it is sufficient to perform line storage of the background light periodically (for example, about once every 10 minutes).

In order to distinguish the masking pixel outputs 28 for each camera (cameras 1 to 4), they will be referred to as masking pixel signals 1 to 4 for convenience. From the masking pixel signals 1 to 4, as explained in FIG. 8, display shelves 2 and 2' of product display shelf 1 are determined from the uniquely determined position of the tip of the customer's hand 5 using the principle of triangulation. , 2", 2"', which height of the display shelf the user reached for (display shelf 2' in the figure), and what product number from the end he tried to grab (3' in the figure). At this time, in order to identify the products from their positions, it is necessary to know in advance what products are displayed at which height and at what position from the end. This is called a product position calculation section in FIG. 23.

By arranging the masking pixel signals 1 to 4 in chronological order, a reproduced image of the customer's hand 5 as seen by the cameras 1 to 4 can be obtained. When the customer's hand 5 grabs the product 3 from the product display shelf 1 and comes out, the product 3 can be read, but the product label cannot be seen due to the way the customer's hand 5 grips the product 3. Judgment may be difficult in some cases, so it is desirable to link the location and product to increase accuracy. Furthermore, although the number of products can be determined from the reproduced image, it may be difficult to see the number depending on how the product 3 is gripped, so the accuracy can be improved by using the reproduced images with a plurality of linear sensor cameras 4. This is called a product image processing section in FIG. 23.

<Embodiment 7 of product display shelf>
In an actual store in which a plurality of product display shelves 1 according to the embodiment of the present invention described above are arranged, as explained in FIG. If the customer also has sales information at the cash register at the time of checkout, system accuracy can be improved by combining the customer response data on product display shelf 1, this customer's tracking data, and this customer's sales data. can be raised. This can prevent fraud such as shoplifting and provide an unmanned store system.
The product display shelf 1 that constructs this system is the product display shelf 1 according to Embodiment 7 of the present invention.

In the store shown in FIG. 24, the product display shelves 1 shown in the top view shown in FIG. )) 6 sets (3 sets x 2 columns) are arranged.
Customer E's travel route is tracked with time by a surveillance camera (not shown) installed in the store. Other customers A, B, C, and D are working on products at product display shelves 1 (1, 3), (2, 1), (3, 2), and (4, 1), respectively. The travel route of customer E is shown by the freehand broken line in FIG. Here, it can be seen that the user stopped at product display shelves 1 at (1, 2) and (2, 3) and arrived at the cash register.

In the store shown in Figure 24, customer response data on product display shelf 1, customer position tracking data, and customer sales data are acquired, but what is important here is that these data are synchronized by time. That's true. That is, the customer data acquired by the product display shelf 1 of the present invention includes information on the product display shelf 1, such as when and how many products of what type were taken from which shelf position, and how many of those products were returned. It's just local information. However, by having the time, it is possible to link customers, and by combining the information such as the route the customer took to reach the product display shelf and the sales data of the customer at the cash register, System accuracy can be improved. This can prevent fraud such as shoplifting and provide an unmanned store system.

<Embodiment 8 of product display shelf>
In an actual store where a plurality of product display shelves 1 according to the embodiment of the present invention are arranged, as explained in FIG. can.
Furthermore, in order to improve the accuracy of the system, information corresponding to the products obtained by sensing the arrival and departure of products in the in-store cart will be added. This information is called in-store cart data.
As a method for acquiring this in-store cart data, a normal area sensor camera may be installed in the in-store cart, and in-store cart data corresponding to the product may be obtained through image processing. By combining this in-store cart data with customer response data on the product display shelf 1, this customer's tracking data, and this customer's sales data, the accuracy of the system can be further improved. If the focus is on in-store cart data, customer tracking data is not necessarily necessary. This can prevent fraud such as shoplifting and provide an unmanned store system. The product display shelf 1 that constructs this system is the product display shelf 1 according to the eighth embodiment of the present invention.

As a method for acquiring this in-store cart data, sensing means provided on the product display shelf 1 may be utilized. Specifically, a camera using a linear sensor is installed on the edge of the top of the in-store cart where products are taken out and put in, and the camera monitors the flat surface of the top of the in-store cart, sensing when products come in and out. Add corresponding information.

A front view, a top view, and a side view of the in-store basket 34 used as a set with the product display shelf 1 according to the eighth embodiment shown in FIG. 25 are shown in FIGS. 25(a), (b), and (c), respectively. . The in-store basket 34 is a basket used in a normal store, and a linear sensor camera 4 (referred to as camera 4 (basket)) is installed on its upper edge. Products 3 are placed in the in-store basket 34, and the customer's hand 5, which takes in and out the products 3, is placed above. FIG. 25(d) shows the customer's hand 5 holding the product 3, and if the pixel signal outputs of the linear sensor camera 4 are arranged in chronological order, a reproduced image of the customer's hand 5 and the product 3 can be obtained. The operation principle of the linear sensor camera 4 of the product display shelf 1 is the same as that of the product display shelf 1.

In the side view of the product display shelf 1 shown in FIG. 25(c), the customer's hand 5 is shown. As shown in FIGS. 25(a), 25(b), and 25(c), the linear sensor camera 4 located at the upper edge of the in-store basket 34 detects the plane of the entrance of the in-store basket 34 by the broken line extending from the linear sensor camera 4. I am monitoring it accordingly. Since the linear sensor camera 4 of the in-store cart 34 does not specify the position, only one is sufficient, or two may be provided to avoid blind spots caused by hands.

The in-store cart data obtained by the linear sensor camera 4 of the in-store cart 34 corresponds to the products in the in-store cart, and as mentioned above, data on customer interaction on the product display shelf 1, tracking of this customer By combining this data with this customer's sales data, the accuracy of the system can be further increased. This can prevent fraud such as shoplifting and provide an unmanned store system.

<Other modification example 1>
Although the invention has been described with reference to several cases, further variations are possible. For example, in the signal processing circuit shown in FIG. 14, the pixel signal output 16 output from the linear sensor camera 4 is subjected to line difference with the line output of the previous line using the line memory 30. Even without using the line memory 30, the CCD register 11 can be used as a substitute for the line memory 30 in the linear sensor structure shown in FIG. 1(c).

Next, this method will be explained. When the shift electrode 12 opens, the charges photoelectrically converted and accumulated in the even-numbered pixels 10 in FIG. It is converted into a signal voltage and output to the outside. Similarly, charges photoelectrically converted and accumulated in the odd-numbered pixels 10 are transferred to the CCD register 11' when the shift electrode 12' is opened, transferred from the CCD register 11' to the output circuit 13, and transferred to the output circuit 13. It is converted into a signal voltage and output to the outside in the same way. In a normal linear sensor 7, the CCD registers 11, 11' alternately output pixel outputs according to the arrangement of the pixels 10.

The signal charge of the odd-numbered pixel 10 transferred to the CCD register 11' is stopped only in the CCD register 11', and used as a line memory. The CCD register 11 drives the signal charge of the even-numbered pixel 10 transferred to the CCD register 11 and discards it from the output circuit. In the next line period, the shift electrode 12' is not opened to transfer charge from the pixel to the CCD register 11', and only the shift electrode 12 is opened to transfer the signal charge of the even-numbered pixel 10 to the CCD register 11. After this, when the CCD registers 11 and 11' are driven, even and odd pixel outputs are alternately output from the CCD registers 11 and 11', but the odd pixel is the pixel output of the previous line, The even numbered pixel output is the pixel output of the current line.

Since the odd-numbered and even-numbered pixel outputs are output alternately, the line difference can be easily determined by taking the difference between the two. Strictly speaking, this is not a comparison of pixels at the same position, but since it is a comparison of pixel outputs at adjacent positions, there is no problem in considering it as a line difference.
Furthermore, since the shift electrodes 12' and 12 are not opened periodically, it is necessary to discard them from the output circuit as dummy pixel signals, and it is not possible to continuously obtain line differences. In this case, since the movement of the hand is slow compared to the operation cycle of the linear sensor 7, there is no problem in practical use.

<Other modification example 2>
Another method will be explained using FIGS. 26(a) and 26(b). First, in FIG. 26(a), the photoelectrically converted and accumulated charges are transferred from all pixels 10 to the CCD register 11' when the shift electrode 12' is opened. At this time, since the shift electrode 12 is not opened, the data is not transferred to the CCD register 11. The transferred charges are stopped in the CCD register 11' for a period corresponding to one line. Next, in FIG. 26(b), the charges photoelectrically converted and accumulated within the period of one line are transferred from all the pixels 10 to the CCD register 11 when the shift electrode 12 is opened. At this time, since the shift electrode 12' is not opened, the data is not transferred to the CCD register 11'.

Next, the charges corresponding to all the pixels 10 in each CCD register 11, 11' are transferred toward the output circuit 13, converted into a signal voltage by the output circuit 13, and output to the outside. The CCD registers 11, 11' alternately output pixel outputs according to the arrangement of the pixels 10. Here, the difference from the driving shown in FIG. 1(c) is that the pixel output of the pixel 10 is such that the signal one line before and the current signal correspond to the same pixel position.
Of course, in this case, the CCD registers 11 and 11' will need a sufficient number of transfer stages to transfer all the pixels, but if you set the mode in which two adjacent pixels are combined in the CCD registers 11 and 11', as shown in FIG. The number of transfer stages of the CCD registers 11 and 11' in c) is sufficient.

Since all pixel outputs of two lines with different photoelectric conversion timings are output alternately for one line period, the line difference can be easily determined by taking the difference between the two lines. Since all pixels are read out, it is a pixel comparison at exactly the same position, and is a line difference. Since a one-line delay is performed on one side of the CCD register (both CCD registers 11' in the two examples), the CCD register 11' can be considered as the line memory 30 shown in FIGS. 14 and 18. The term "line memory" in the claims includes this CCD register.
Since the signal charges of all pixels 10 transferred to the CCD register 11 are stopped for one line period, the CCD register 11 is used as a line memory. Since all the pixels 10 are read every time, there is no need to discard them as dummy pixels, so line differences can be taken continuously.

<Other modification example 3>
In Modifications 1 and 2, it has been explained that by using a CCD type sensor as the linear sensor 7 without using the line memory 30, the CCD register 11 can be used as a substitute for the line memory 30.
In the present invention, a CMOS type sensor is used as the linear sensor 7, and the CMOS linear sensor can also be used as a substitute for the line memory 30, which will be explained next.

First, the structure of a CMOS linear sensor will be explained with reference to FIG. A CMOS shift register 35 is arranged along the linearly arranged pixels 10, and the signal charges of the pixels 10 are connected to the signal line 37 by sequentially opening the switching gates 36 connected to each pixel. An output circuit 38 is provided at the end of the signal line 37, and outputs the signal charge of the pixel to the outside in the form of a signal voltage. The photoelectric conversion time and output timing of each pixel are determined by opening and closing of a switching gate 36 associated with the pixel, and this is controlled by a CMOS shift register 35.

Next, a method of performing an alternative operation of the line memory 30 using the CMOS linear sensor shown in FIG. 27 will be explained. The key point is to double the exposure time between odd-numbered pixels and even-numbered pixels in the pixel row 10. For convenience, odd-numbered pixels perform photoelectric conversion corresponding to two lines (photoelectric conversion starts one line earlier than even-numbered pixels), and even-numbered pixels perform photoelectric conversion equivalent to one line. Odd-numbered pixels and even-numbered pixels sequentially come out from the output circuit 38 via the signal line 37, but when an even-numbered pixel is subtracted from an adjacent odd-numbered pixel, the remainder is the output of the odd-numbered pixel one line before. Here, it is assumed that the outputs of adjacent odd-numbered pixels and even-numbered pixels on the same line are the same. That is, the odd-numbered pixels explained in paragraph numbers 104 and 105 shown in FIG. 1(c) are in the same state as the pixel output of one line before.

The output of the odd-numbered pixel one line before, obtained by subtracting the even-numbered pixels from the adjacent odd-numbered pixels, is the same as the output that has passed through the line memory 30 in FIG. A line difference is obtained by taking the difference from the output of an even numbered pixel, which is the pixel output of the current line. The switching gates 36 for odd-numbered pixels and even-numbered pixels can be controlled by controlling what kind of pulses are generated by the CMOS shift register 35. In this way, the line memory 30 can be replaced by a CMOS linear sensor. The term "line memory" in the claims includes this CMOS linear sensor.

<Other modification example 4>
The observation item of the linear sensor camera used in the product display shelf of the present invention is merely observing the movement of the customer's hand 5 when putting in and out of the product on the front plane of the product display shelf. This movement does not occur frequently, and only occurs when the customer picks up the product. Therefore, unlike the operation of a surveillance camera inside a store, the occurrence interval is about 1 to 10 minutes. Therefore, there is no need to constantly drive the linear sensor camera. Furthermore, when a product display shelf is monitored by multiple linear sensor cameras, it is sufficient to normally operate only one camera and activate the other cameras when the product is about to be picked up.

In addition, as shown in Figure 24, when the customer's travel route is known, the linear sensor camera starts driving when the customer approaches the product display shelf or reaches for the product, and is inactive at other times to save energy. You may do so. Such energy-saving measures in stores are common and are not mentioned in the claims.

<Relationship between claims, embodiments, and corresponding drawings>
Claim 1: A first embodiment of the present invention, corresponding to FIG.
Claim 2: A first embodiment of the present invention, corresponding to FIGS. 2 and 3.
Claim 3: A first embodiment of the present invention, corresponding to FIGS. 2 and 3.
Claim 4: A second embodiment of the present invention, corresponding to FIGS. 4 to 8.
Claim 5: A third embodiment of the present invention, corresponding to FIGS. 9 and 10.
Claim 6: A fourth embodiment of the present invention, which corresponds to FIGS. 11, 12, 14, and 18.
Claim 7: A fourth embodiment of the present invention, corresponding to FIGS. 11 and 12.
Claim 8: A fourth embodiment of the present invention, corresponding to FIGS. 11 to 13.
Claim 9: A fifth embodiment of the present invention, corresponding to FIGS. 14 to 17, 26, and 27.
Claim 10: A fifth embodiment of the present invention, corresponding to FIGS. 18 to 22.
Claim 11: A sixth embodiment of the present invention, corresponding to FIG. 23.
Claim 12: A sixth embodiment of the present invention, corresponding to FIG. 23.
Claim 13: Seventh embodiment of the present invention, corresponding to FIG. 24.
Claim 14: Eighth embodiment of the present invention, corresponding to FIG. 25.

1 Product display shelf 2, 2', 2", 2"' Display shelf 3, 3', 3", 3"' Product 4 Linear sensor camera 5 Customer's hand 6 Camera housing 7 Linear sensor 8 Lens 9 Metal wire 10 Pixel 11, 11' CCD register 12, 12' Shift electrode 13 Output circuit 14 Electrode 15, 15' Non-reflective plate 16 Pixel signal output 17 Line difference processing unit 18 Switch circuit 19 Line storage unit 20 Difference circuit 21 Absolute value circuit 22 Pixel Absolute difference 23 Masking signal generation section 24 Threshold 25 Masking signal generation circuit 26 Masking signal 27 Masking circuit 28 Masking pixel output 29 Delay circuit 30 Line memory 31 Motion contour edge determination section 32 Motion contour edge information 33 Threshold 34 In-store cart 35 CMOS shift register 36 Switching gate 37 Signal line 38 Output circuit

Claims (5)

A camera using a linear sensor with pixels arranged in a line is installed on the front edge of the product display shelf where products are displayed, and the front plane of the product display shelf is used as the imaging surface of the camera. sensing means for sensing the ingress and egress of goods passing in front of the
an image acquisition unit that acquires image information of the product by chronologically arranging output signals of a camera using a linear sensor of the sensing unit when the product enters or leaves the product;
A line output signal of the linear sensor of the sensing means at a previous photoelectric conversion timing is temporarily stored in a line storage section, and this stored line output signal and a line output signal output at the current photoelectric conversion timing are combined. , means for comparing differences between pixels corresponding to the same location on the imaging surface;
means for obtaining pixel difference absolute value data by taking the absolute value of the pixel difference data obtained by the difference comparison;
means for obtaining a binary masking signal obtained by comparing the pixel difference absolute value data with a threshold;
From the binary masking signal and the pixel signal output of the linear sensor, only the pixel signal output of the area where the masking signal is 1 is output as an image of the area where work is occurring due to the entry and exit of products. means and
A product display shelf characterized by having.
2. The product display shelf according to claim 1, wherein the line output signal at the timing of the previous photoelectric conversion is a line output signal at a time when no work is occurring due to the entry and exit of the product.
Binary masking signals are used to determine the timing when work is not occurring due to the entry and exit of products, and a line where all masking signals are 0 is determined to be the timing when work is not occurring due to the entry and exit of products. 3. The product display shelf according to claim 2, wherein the line output signal is temporarily stored in a line storage section.
2. The product display shelf according to claim 1, wherein the previous photoelectric conversion timing is the immediately previous line output signal corresponding to the time difference between adjacent lines of the linear sensor.
Claim 4 characterized in that a binarized masking signal is obtained from the pixel difference absolute value, via motion contour edge information corresponding to information on the outer contour of the moving object, and by comparison with a threshold value. Product display shelves listed in.

Images