JP7389997B2 - Marketing system using camera - Google Patents

Description

For example, the present invention associates barcode input information of a product at a register, simple sales information, etc., as one of the bind information, like in a conventional POS cash register system, and identifies the photographed person as a customer of the product. The present invention relates to a marketing system using a camera that records and stores information such as text information about a person being photographed without storing images or feature amounts.

2. Description of the Related Art Conventionally, many POS systems have been introduced in stores such as convenience stores to grasp the status of customers visiting the store. Initially, products were managed by inputting product barcodes at registers and simple sales information, but later POS registers recorded facial images of customers standing in front of the register. There are also suggestions for doing so.

In particular, sales opportunities where a customer visits a sales floor but does not purchase a product are based on stop-in information from a stop-in detection device that detects a customer's visit to a sales floor, and purchase information from a POS terminal that manages customer purchasing information. By measuring the loss, we can estimate the lost sales opportunity and the potential demand for the product based on the lost sales without inconveniencing the consumer and without being misled by false information to determine the true purchase intention of the consumer. A device that can do this has been proposed (for example, see Patent Document 1).

Many POS terminals, including the above-mentioned POS terminal, ignore purchasers' portrait rights and privacy. In addition, the handling of personal information, including facial images, has become stricter, and ultimately POS terminals, where information is entered based on store staff's judgment based on the purchaser's appearance, are still the mainstream. Therefore, the present applicants are able to collect and save attributes such as gender and age of people who visited the store or the venue, while protecting the portrait rights and privacy of the people who visited the store, as well as to save the behavior of the people. proposed a collection system (see Patent Document 2).

Japanese Patent Application Publication No. 2001-331875 Japanese Patent Application Publication No. 2018-165849

By proposing this collection system, it has become possible to obtain attribute information of the person who purchased the product while protecting portrait rights and privacy. In contrast, an object of the present invention is to obtain a marketing system that binds the obtained attribute information with other information to achieve advanced utilization.

That is, the present invention uses the facial image itself of a person to be photographed, such as a product purchaser, taken in an area in front of a cash register counter, an area in front of a vending machine or an automatic ticket vending machine, or the feature amount of the facial image of the person to be photographed. A marketing system using a camera that does not need to be stored, but estimates the attribute information of the person photographed at the same time as the photograph is taken, binds the estimated attribute information with related data related to the person, and stores it in a recording means. The purpose is to obtain.

A marketing system using a camera according to the invention described in claim 1 includes a photographing means for photographing within a specific area;
a person identifying means for detecting a photographed person within an image area photographed by the photographing means and detecting the face of the photographed person;
Attribute acquisition means for estimating attribute data such as gender and age from feature quantities extracted from an image including at least a face of the photographed person identified by the person identification means;
Attribute information extraction means for adding date and time data when the photographed person in the specific area was photographed to the attribute data of the photographed person estimated by the attribute acquisition means and extracting the data as photographed person attribute information;
Related data acquisition means for acquiring related data related to the person to be photographed;
comprising a bind data storage means for binding the photographed person attribute information and the related data acquired by the related data acquisition means and storing the bound data in a recording means,
The face image of the person to be photographed identified by the person identification means is not stored, but the attribute information of the person to be photographed and the related data acquired by the related data acquisition means are stored in the recording means by the bind data storage means. It is
The attribute data estimated by the attribute acquisition means is an image in which each rotation of the three-dimensional vector of the face of the photographed person identified by the person identification means is within a pre-allowed angular range with respect to the front view. It is characterized in that it is estimated from feature quantities extracted from .

A marketing system using a camera according to the invention described in claim 2 is such that the photographing means according to claim 1 photographs an area in front of a checkout counter or an area in front of a vending machine,
The related data is purchase data of the photographed person obtained from a cash register or a vending machine,
The bind data storage means is characterized in that the purchase data and the photographed person attribute information are bound and stored in the recording means at approximately the same time as the photographed date and time data of the photographed person attribute information. It is something.

In the marketing system using a camera according to the invention described in claim 3, the photographing means according to claim 1 or 2 is disguised so that it is difficult for the photographed person to find it. It is characterized by:

A marketing system using a camera according to the invention described in claim 4 includes: the person identifying means according to any one of claims 1 to 3 , the attribute acquisition means, the attribute information extraction means, and the a bind data storage means is provided in a personal computer connected to the photographing means via a wired or wireless data communication line,
The personal computer is characterized in that it includes an accelerator.

The present invention does not store the facial image itself of the photographed person, such as a product purchaser, taken in the area in front of the checkout counter or the area in front of the vending machine, and the attribute information of the photographed person is recorded at the same time as the photograph is taken. The estimated attribute information and related data related to the person to be photographed can be bound and stored in the recording means, and there is an effect that a marketing system that can be used to a high degree can be obtained.

FIG. 1 is an explanatory diagram showing the configuration of a marketing system of the present invention. FIG. 2 is an explanatory diagram illustrating the configuration of an embodiment of the marketing system of the present invention shown in FIG. 1. FIG. 1 is a flowchart of an embodiment of a marketing system using a camera according to the present invention. FIG. 3 is an explanatory diagram showing types of rotation of a three-dimensional vector of a face image of a person to be photographed.

In the present invention, there are provided a photographing means for photographing within a specific area, a person specifying means for detecting a photographed person in an image area photographed by the photographing means and the face of the photographed person, and a person specifying means. an attribute acquisition means for estimating attribute data such as gender and age from feature quantities extracted from an image including at least the face of the photographed person identified in the above; and attribute data of the photographed person estimated by the attribute acquisition means. an attribute information extraction means for adding date and time data of a photographed person in a specific area and extracting the data as photographed person attribute information; a related data acquisition means for acquiring related data related to the photographed person; A bind data storage means is provided which binds the photographed person attribute information and the related data acquired by the related data acquisition means and stores the bound data in the recording means.

In a system equipped with each of these means, the face image of the photographed person identified by the person identification means is not saved, but the bind data storage means records the photographed person's attribute information and related data acquired by the related data acquisition means. It is stored in means.

This eliminates the need to save the facial image or features of the photographed person, such as a product purchaser, taken in the area in front of the cashier counter, vending machine, or automatic ticket vending machine. By estimating the attribute information of the photographed person at the same time as the photograph is taken, and binding the estimated attribute information with related data related to the photographed person and storing it in a recording means, the product purchased by the product purchaser and the product searcher can be saved. etc. can be associated as one of the binding information.

The photographing means of the present invention may be of any type as long as it photographs within a specific area. More specifically, a predetermined area placed in a corner of a commercial facility, such as an area in front of a cash register counter, an area in front of a vending machine such as an automatic ticket vending machine or an in-store coffee server, or an area in front of a predetermined product shelf. It is sufficient to photograph a specific area such as an area in front of a product shelf. More preferably, it is disguised so that it is difficult for the photographed person, such as the purchaser or visitor, to find it.

That is, the facial image of the person photographed in the present invention is discarded without being recorded, thereby protecting portrait rights and privacy. For this reason, if the presence of a camera is known, it may raise unnecessary questions for the person being photographed, and the store operator may be forced to provide an explanation, so it may not be possible to camouflage it. Preferably.

For example, the diameter of the lens, which is the image-receiving part of a camera serving as a photographing means, may be made smaller to make it difficult for buyers and visitors to recognize the image. Furthermore, by making a small hole in the display board of the LED panel or the frame of the panel and placing the image receiving part of the camera through the hole, it is possible to make the image receiving part disappear into the display board of the LED panel or the frame of the panel. All you have to do is disguise it.

Another method of camouflage is to cover the lens part of the camera, which is the image receiving part, with mirror glass, acrylic smoke, or the like, so that it becomes difficult for the person being photographed to see. In addition, in this case, by using the camera's HDR (High Dynamic Range) function, such as combining multiple photos with different brightness, a wider range of brightness (dynamic range) can be expressed.

The person identifying means of the present invention may be any means that detects the face of the person to be photographed by detecting facial feature amounts from a photographed image. Specifically, for captured images, instead of detecting the person being photographed and the features of that person's face from a single still image, we first identify areas of movement in the captured image. It is also possible to determine whether the person is the person being photographed.

Next, for the person to be photographed who has been identified within the moving area, it is detected that the person is a person based on feature points, and a facial contour that is estimated to be a face is detected in a region corresponding to the person's face. Detection of the face contour includes both eyes placed within the face contour, eyebrows placed above each of the eyes, a nose placed below between the eyes, and a nose placed below the nose. Any area in which all or part of a part presumed to be a mouth is placed within an area that can be placed as a human body may be specified as a face.

When it is recognized as a face, estimation begins immediately. That is, attribute data such as gender and age are estimated by the attribute acquisition means from the facial feature amount of the estimated facial contour. Note that, for example, the recognition rate of attribute information estimation by the attribute acquisition means can be increased by making a facial image three-dimensional and recognizing it from an image of a photographed person whose movements are continuous.

In addition, the attribute acquisition means estimates attribute data such as gender and age from the features extracted from an image that includes at least the face of the photographed person. It may also be possible to assist in estimating gender and age based on fleshiness, clothing, etc. Furthermore, if there is a risk that images of hairstyles, fleshiness of the body below the face, etc. may infringe on portrait rights or privacy, these images will also be discarded without being recorded.

Further, when a plurality of people are reflected in the photographed image, the person closest to the camera serving as the photographing means may be detected as the target person to be photographed. For example, if a camera is equipped with an autofocus function, the person closest to many people may be selected as the subject of the photograph, or the The person with the widest area of the facial contour may be selected as the target person to be photographed.

The attribute acquisition means of the present invention may be one that estimates attribute information such as the gender and age of the person to be photographed from the face of the person detected by the above-mentioned person identification means and stores this in the recording means. Bye. For example, by creating personal facial feature information for a wide range of age groups for men and women in advance, and determining the degree of similarity with this information, we can use a function to determine age and gender, This is possible by using a model.

In particular, the attribute acquisition means of the present invention can improve the recognition rate by capturing an image that is nearly directly facing the face of the person to be photographed. That is, the angle of the face of the person to be photographed changes as the person moves through continuous motion. For example, if the eye level position is close to the center of the contour of the face, it is determined that the person is young. This is because in many cases, the personal facial feature information created in advance is trained on a face facing directly.

Due to these changes in the angle of the face, a large amount of inaccurate data is accumulated, and these inaccurate data may become bugs that hide accurate data. Therefore, as the attribute data estimated by the preferable attribute acquisition means of the present invention, each of the rotations of the three-dimensional vector of the face of the person to be photographed identified by the person identification means is determined in advance with respect to the front view. It is estimated from feature amounts extracted from images within the angular range.

Specifically, as for the rotation of the three-dimensional vector of the face of the person being photographed, the roll angle is the rotation on the X-axis of the front and back of the face image facing directly, and the rotation on the Y-axis of the left and right of the face image facing directly. The pitch angle is the pitch angle, and the yaw angle is the rotation around the Z-axis above and below the facing face image. For each of these roll angles, pitch angles, and yaw angles, it is sufficient to avoid adopting images outside the allowable angle ranges.

That is, by discarding images outside the allowed angle range and not using them in the attribute acquisition means, or discarding the attribute data estimated by the attribute acquisition means for images outside the allowed angle range, Attribute data such as gender and age are estimated from the feature values extracted from the image within the specified angle range, and the subsequent attribute information extraction means extracts attribute information of the photographed person, and the bind data storage means extracts attribute data of the photographed person. The person attribute information and the related data acquired by the related data acquisition means may be bound and stored in the recording means.

Specifically, when a face image of a person to be photographed is recognized, feature points such as the face, eyes, nose, mouth, eyebrows, etc. are detected from the image. At this time, based on the positional relationship between both eyes and nose, the person to be photographed is set to be directly facing the photographed person at 0°, and the roll angle, pitch angle, and yaw angle are calculated with respect to this facing position, and the angles are calculated within the allowable range. All you have to do is determine which one is.

More specific images within a pre-allowed angle range include roll angle, pitch angle, and yaw angle, with a face image facing directly at 0°, depending on the resolution of the camera used as a photographing means and the installed lighting, etc. What is necessary is to set the optimum angle for any of the above, and in the embodiment described later, each of the roll angle, pitch angle, and yaw angle is operated at ±9°.

Furthermore, by recognizing a three-dimensional facial image based on continuous motion, the estimation of attribute information is improved. For example, since the height positions of the eyes, nose, mouth, etc. relative to the entire face are clarified, age and gender discrimination is improved.

A preferable attribute acquisition means of the present invention uses artificial intelligence (AI) to estimate attribute information such as the gender and age of the person to be photographed. In particular, by performing deep learning, the accuracy of recognition of attribute acquisition data can be improved. In addition to age and gender, race data may be added to the attribute information estimated by the attribute acquisition means. For example, a racial classification model may be used to divide the race into five categories: white, black, Asian, Indian, and other races.

The attribute information extraction means of the present invention adds date and time data of the photographed person within a specific area to the attribute data of the photographed person estimated by the attribute acquisition means and extracts the data as photographed person attribute information. It is fine as long as it is something. That is, the date and time data of the photograph may be added to attribute data such as gender and age estimated by the attribute acquisition means.

The related data acquisition means may be any device that acquires related data related to the photographed person. Related data refers to checkout counter and product purchase data within a specific area, purchase data from vending machines and automatic ticket vending machines, and the like.

The related data of the present invention may be any data as long as it is related to the person to be photographed. Examples include purchase data of a person to be photographed in a store where a camera is installed as a photographing means, line of sight data of a person to be photographed, or product information estimated from an acquisition operation from a product shelf. Furthermore, if the store is a store that provides food such as ramen noodles and coffee, the temperature and weather inside and outside the store may be bound as related data.

That is, for a product purchaser who is detected in front of the checkout counter and whose attribute information is estimated, the purchase data of the product becomes related data. In this case, the data on the detection date and time of the product purchaser detected at the counter in front of the cash register and the data on the input date and time of the purchase data of the product at the corresponding cash register may be recorded in the recording means while being inquired.

Regarding purchase data for vending machines, if the purchase data cannot be obtained through a data communication line for an existing vending machine, the purchaser can obtain an image of the vending machine's push button operation status. A camera may be placed at the location, and the image taken by the camera may be recorded in the recording means as purchase data of the product purchased by the purchaser's actions. This allows even existing vending machines to be used.

In addition, for visitors whose attribute information has been estimated by being detected in a predetermined area in front of a product shelf placed in a corner of a commercial facility, the information can be estimated from the visitor's line of sight data or the acquisition behavior from the product shelf. The product information obtained becomes related data. Even in this case, if the data on the date and time of detection of a visitor intruding into the area in front of the product shelf and the product information estimated from the visitor's line of sight data or the acquisition operation from the product shelf are recorded together in the recording means. good.

In yet another preferable embodiment, the related data is product information estimated from line-of-sight data of the photographed person or an acquisition operation from a product shelf, and the binding data storage means stores the photographed date and time data of the photographed person attribute information. One example is one in which product information and photographed person attribute information at approximately the same time are bound and stored in the recording means.

The bind data storage means of the present invention may be any device that binds the attribute information of the person to be photographed and the related data related to the person to be photographed and stores the bound data in the recording means. In a preferred embodiment, the related data to be bound is purchase data of the photographed person, and the binding data storage means stores the purchase data and the photographed person attributes at approximately the same time as the photographing date and time data of the photographed person attribute information. One example is one that binds information and stores it in a recording means.

It should be noted that, as for substantially the same time, it is preferable to adjust the binding so that even times separated by about 2 to 5 seconds are bound as the same time. In this case, the attribute information of the person to be photographed and the related data related to the person to be photographed were adjusted so that they would be bound at the same time even if approximately 2 to 5 seconds had passed; It is also possible to adjust other times, and select a preferable time by adjusting the set time.

After the attribute information estimated by the attribute acquisition means is stored in the recording means, the facial image of the photographed person specified by the person identification means is discarded. That is, since the data to be saved is not image data but attribute information obtained by this attribute acquisition means, portrait rights and privacy are protected.

This attribute acquisition means may include not only attribute information but also facial feature amount information and hair styles including hair color within a range that does not infringe on privacy. Further, this facial feature amount information, hair style including hair color, etc. may also be included as one of the related data described later.

The bind data storage means of the present invention may be one that acquires other related data of the attribute information of the photographed person specified by the person identification means and stores it in the recording means. For example, each person to be photographed is assigned an arbitrary number and associated with attribute information each time a person is identified. Thereafter, by further adding related data to this arbitrary number and storing it in the recording means, it becomes possible to clearly distinguish the photographed person from other specified persons.

The person identification means, attribute acquisition means, attribute information extraction means, and bind data storage means of the present invention are installed in a personal computer (hereinafter also referred to as "PC") placed near the photographing means and in this PC. It is required to operate a system that is installed as an accessory and does not transmit at least the image data of the person to be photographed using an Internet line. This is because when transmitting data via the Internet, it becomes difficult to protect the portrait rights and privacy of the person being photographed.

Therefore, images taken from a photographic means are so-called "stand-alone" as long as they are isolated from third parties without using a worldwide Internet connection, and the portrait rights and privacy of the photographed person are protected. The control device may be connected via a wired or wireless data communication line such as a local area network (LAN), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

In a preferred embodiment, the PC includes a person identifying means, the attribute acquiring means, and an attribute information extracting means inside the PC, and is connected to a camera as a photographing means via a wired or wireless data communication line. Alternatively, the bind data storage means may be provided inside the PC or on another PC outside the PC.

The recording means for storing the bound data of the purchase data and the photographed person attribute information by the bind data storage means may be connected inside or outside the PC, and the storage means may be stored in one or more servers via an Internet line. It is also possible to do so.

As this PC, it is recommended to use a PC equipped with a CPU with high processing power, especially when performing deep learning, but a PC with a smaller size, a camera, and a storage device is recommended. In this case, it becomes difficult to install a CPU with high processing power. In that case, image processing speed can be increased by using a PC equipped with a deep learning accelerator such as a GPU (Graphics Processing Unit) that supports high-speed processing by performing parallel calculation processing on a large number of cells. , a high accuracy rate can be obtained.

In addition, as another preferred aspect, attribute information of the photographed person is extracted from the image photographed by the photographing means in a PC placed near the photographing means, and the binding data storage means and related information related to the photographed person are stored in a PC disposed near the photographing means. One example is where the data and the recording means are located in one or more servers communicated via the Internet.

With this system, for example, a comprehensive marketing system can be constructed for a chain store having many branches. P has a PC equipped with a person identification means, an attribute acquisition means, an attribute information extraction means, and a bind data storage means in each branch located near the photographing means, and the person identification means detects information within the image area. The person to be photographed is detected, the face of the person to be photographed is detected, and the attribute acquisition means of the same control device is used to determine the gender, age, etc. from the feature points of the person to be photographed and/or the feature points of the face of the person to be photographed. Estimate attribute data.

Next, the meal ticket purchase data from the automatic ticket vending machine at the same branch is used as the attribute data of the person to be photographed using the attribute information extraction means in the same PC, and the date and time data of the photographed person are taken while standing in front of the automatic ticket vending machine. As the time, food ticket purchase data that matches this time is bound and added and extracted as photographed person attribute information.

Subsequently, using the bind data storage means, the sales data of the automatic ticket vending machine connected to the headquarters at approximately the same time as the photographed person's attribute information is bound as related data and connected to the control device of the headquarters via the Internet line. However, the face image of the photographed person specified by the person identification means is connected to a PC only in the branch and is used by the person identification means and attribute acquisition means, but is not saved.

Note that the attribute data is the meal ticket purchase data from the automatic ticket vending machine at each branch, but if the automatic ticket vending machine itself cannot output the data, the person being photographed can push the ticket while standing in front of the automatic ticket vending machine. It is also possible to use images of buttons. In this case, a PC located at each branch may recognize which button the purchaser has pressed, or only an image of the button may be sent to the head office for analysis.

In addition, related data related to the bound person to be photographed is stored in cloud computing provided via a data communication line such as the Internet, or on a predetermined server connected via a data communication line. You may also use the Furthermore, whether it is stored in a binding data storage means, stored in a storage means in cloud computing provided via a data communication line such as the Internet, or stored in a storage means connected via a data communication line such as the Internet. The information may also be stored in a predetermined storage device such as a server.

Further, the related data to be bound in the present invention may be any data that forms effective information when bound to the attribute data of the person to be photographed in the specific area where the photographing means is installed. For example, if the photographing method is to photograph the area in front of the cashier counter or the area in front of the vending machine, the related data may include purchase data of products purchased by the person being photographed at the cash register or vending machine. . As a result, the bind data storage means binds the purchase data and the photographed person attribute information at approximately the same time as the photographed date and time data of the photographed person attribute information, and stores the bound data in the recording means.

In addition, if the photographing means is to photograph an area in front of a predetermined product shelf located in a corner of a commercial facility, the related data may be the line of sight data of the person being photographed or the acquisition operation from the product shelf. For example, product information estimated from As a result, the bind data storage means binds the product information and the photographed person attribute information at substantially the same time as the photographed date and time data of the photographed person attribute information, and stores the bound data in the recording means. In addition, weather conditions such as temperature and humidity around the commercial facility may also be added and bound as related data.

FIG. 1 is an explanatory diagram showing the configuration of the marketing system of the present invention. FIG. 2 is an explanatory diagram illustrating the configuration of one embodiment of the marketing system of the present invention shown in FIG. FIG. 3 is a flowchart of an embodiment of a marketing system using a camera according to the present invention. FIG. 4 is an explanatory diagram showing the types of rotation of the three-dimensional vector of the face image of the person to be photographed, in which figure a is a perspective view showing the rotation of the three-dimensional vector, and figure b is the face image of the person to be photographed facing directly. , C shows a face image with the roll angle rotated, D shows a face image with the pitch angle rotated, and E shows a face image with the yaw angle rotated.

As shown in the figure, the marketing system of the present embodiment includes a camera 11 as a photographing means for photographing a specific area in front of the register 21 (hereinafter referred to as "in front of the register"). The recognition rate can be improved by preferably placing the camera 11 at a position directly facing the person 20 to be photographed. In addition to the camera 11 that directly faces the person to be photographed 20, an overhead camera 11a that is placed on the ceiling of the store and that views the area in front of the cash register may also be used.

Image data taken by the camera 11 is introduced into a control device 26 connected by wire. The control device 26 uses the image data taken by the camera 11 to estimate attribute data of the person photographed in the image data. Specifically, first, the person specifying means 12 detects the person 20 to be photographed within the image area photographed by the camera 11 and detects the face of the person 20 to be photographed.

Next, the attribute acquisition means 13 estimates attribute data such as gender and age from the feature amount of the person to be photographed 20 specified by the person specifying means 12 and/or the feature amount of the face of the person to be photographed. At this time, regarding the estimation of the attribute information of the person to be photographed by the attribute acquisition means 13, in this embodiment, the attribute information including the gender and age of the person to be photographed is estimated using artificial intelligence (AI). In other words, by creating a certain amount or more of personal facial feature information for a wide range of age groups for each known gender, and determining the degree of similarity with this information, you can use a function to determine age and gender, or This is possible by using an aging model of the face.

Next, the attribute information extraction means 14 adds the photographed date and time data 18 of the photographed person 20 in the area in front of the cash register to the attribute data of the photographed person 20 estimated by the attribute acquisition means 13, and Extract as person attribute information. The bind data storage means 15 binds the obtained photographed person attribute information and the related data 17 related to the photographed person and stores it in the recording means 16. Note that the face image of the photographed person specified by the person specifying means is not stored, but the photographed person attribute information and the related data acquired by the related data acquisition means are stored in the recording means by the bind data storage means.

In one embodiment, as shown in FIG. 2, the person identification means 12, the attribute acquisition means 13, the attribute information extraction means 14, and the bind data storage means 15 are arranged near the photographing means. The recording means 16 is installed attached to this PC 26.

As this PC 26, a smaller PC 26 is adopted because it is placed near the register, but if it is small, it will inevitably become a fragile CPU. For this reason, the PC 26 is equipped with an accelerator (not shown) that performs parallel calculation processing to support high-speed processing when drawing images. This built-in accelerator increases image processing speed and provides high accuracy rates.

As shown in FIGS. 2 and 3, the camera 11 directly facing the photographed person 20 standing in front of the register 21 in the store is disguised so that it is difficult for the photographed person 20 to find it. That is, the diameter of the lens, which is the image receiving part of the camera 11, is made smaller, and a small hole is made in the display board of the panel 22 or the frame of the panel 22, which is placed directly facing the person 20 to be photographed, and the camera is inserted through the hole. By arranging the eleven image receiving sections, the image receiving sections are disguised by being blended into the display board of the panel 22 or the frame of the panel 22.

In the embodiment shown in FIG. 2, as shown in the flowchart of FIG. The person 20 is detected, and furthermore, the facial contour of the person 20 to be photographed is detected. Similarly, the attribute acquisition means 13 in the PC 26 estimates attribute data such as gender and age from the specified feature amount of the person to be photographed and the feature amount of the face of the person to be photographed. When estimating from the feature amount of the person to be photographed and the feature amount of the face of the person to be photographed, high-speed processing can be achieved by performing parallel calculation processing using an accelerator.

Attribute information extraction means 14 adds date and time data 18 when the person to be photographed within a specific area was photographed to the attribute data of the person to be photographed estimated by attribute acquisition means 13, and extracts the data as attribute information of the person to be photographed. At this time, it is determined whether the rotation angle of each three-dimensional vector of the face of the person to be photographed is within an allowable range for facing directly, and those outside the allowable range are bound with related data.

On the other hand, related data 17 as bind data is input with a bar code or the like in the register 21, and date and time data 18 is added to the purchased product data 23 obtained via a data communication line such as the Internet 24. It is extracted by the corresponding register 21.

The bind data storage means 15 in the PC 26 is connected to the PC 26 by binding the photographed person attribute information with related data 17, which is purchase data at approximately the same time as the photographed date and time data of the photographed person attribute information. It is saved in the recording means 16.

That is, regarding the photographic person attribute information extracted by the attribute information extraction means, the extracted related data 17 is bound based on the date and time data 18. Furthermore, the date and time data 18 may be adjusted so that it is bound even when there is a gap of several seconds between the photographed person attribute information and the product purchase time, rather than the photographed person attribute information at exactly the same time. Note that the time interval can be adjusted by the user.

As mentioned above, in this embodiment, at this time, it is determined whether the rotation angle of each three-dimensional vector of the face of the person to be photographed is within the permissible range for facing directly, and if it is outside the permissible range, Binding with related data is performed. This can prevent a large amount of inaccurate data from being accumulated.

That is, as shown in FIG. 4, the types of rotation of the three-dimensional vector of the face image of the person to be photographed are shown. As shown in figure a, for a face image, the roll angle is the rotation on the X-axis before and after the face image facing directly, and the pitch angle is the rotation around the Y-axis on the left and right of the face image facing directly. , the yaw angle is the rotation of the face image facing directly on the upper and lower Z axes.

If the roll angle in figure c is outside the permissible angle range for the face image of the photographed person in figure b, the pitch angle in figure d is set as permissible without binding with the related data. If the yaw angle in figure e is outside the allowed angle range, it will not be bound to the related data.Similarly, if the yaw angle in figure e is outside the allowed angle range, it will not be bound to the related data. good.

That is, by discarding images outside the allowed angle range and not using them in the attribute acquisition means, or discarding the attribute data estimated by the attribute acquisition means for images outside the allowed angle range, Attribute data such as gender and age are estimated from the feature values extracted from the image within the specified angle range, and the subsequent attribute information extraction means extracts attribute information of the photographed person, and the bind data storage means extracts attribute data of the photographed person. The person attribute information and the related data acquired by the related data acquisition means may be bound and stored in the recording means.

Specifically, when a face image of a person to be photographed is recognized, feature points such as the face, eyes, nose, mouth, eyebrows, etc. are detected from the image. At this time, based on the positional relationship between both eyes and nose, the person to be photographed is set to be directly facing the photographed person at 0°, and the roll angle, pitch angle, and yaw angle are calculated with respect to this facing position, and the angles are calculated within the allowable range. Determine whether

More specific images within a pre-allowed angle range include roll angle, pitch angle, and yaw angle, with a face image facing directly at 0°, depending on the resolution of the camera used as a photographing means and the installed lighting, etc. What is necessary is to set the optimum angle for either of them. In this embodiment, each of the roll angle, pitch angle, and yaw angle is operated at ±9°.

11...Camera (photography means),
12...Person identification means (detects the photographed person in the image area and detects the photographed person's face),
13...Attribute acquisition means,
14...Attribute information extraction means,
15... Bind data storage means,
16...recording means,
17...Related data,
18...Date and time data,
20...Person photographed,
21...Register,
22...Panel,
23...Purchase product data,
24...Internet,
25...server,
26...PC (control device),

Claims (4)

a photographing means for photographing within a specific area;
a person identifying means for detecting a photographed person within an image area photographed by the photographing means and detecting the face of the photographed person;
Attribute acquisition means for estimating attribute data such as gender and age from feature quantities extracted from an image including at least a face of the photographed person identified by the person identification means;
Attribute information extraction means for adding date and time data when the photographed person in the specific area was photographed to the attribute data of the photographed person estimated by the attribute acquisition means and extracting the data as photographed person attribute information;
Related data acquisition means for acquiring related data related to the person to be photographed;
comprising a bind data storage means for binding the photographed person attribute information and the related data acquired by the related data acquisition means and storing the bound data in a recording means,
The face image of the person to be photographed identified by the person specifying means is not stored, but the attribute information of the person to be photographed and the related data acquired by the related data acquisition means are stored in the recording means by the bind data storage means. It is
The attribute data estimated by the attribute acquisition means is an image in which each rotation of the three-dimensional vector of the face of the photographed person identified by the person identification means is within a pre-allowed angular range with respect to the front view. A marketing system using a camera characterized by being estimated from feature quantities extracted from a camera . The photographing means photographs an area in front of a cash register counter or an area in front of a vending machine,
The related data is purchase data of the photographed person obtained from a cash register or a vending machine,
The bind data storage means is characterized in that the purchase data and the photographed person attribute information are bound and stored in the recording means at approximately the same time as the photographed date and time data of the photographed person attribute information. A marketing system using the camera according to claim 1. 3. The marketing system using a camera according to claim 1, wherein the photographing means is disguised so as to be difficult to be detected by the photographed person. The person identification means, the attribute acquisition means, the attribute information extraction means, and the bind data storage means are provided in a personal computer connected to the photographing means via a wired or wireless data communication line,
4. The marketing system using a camera according to claim 1 , wherein the personal computer is equipped with an accelerator.

Images