WO2016035632A1

WO2016035632A1 - Data processing device, data processing system, data processing method, and program

Info

Publication number: WO2016035632A1
Application number: PCT/JP2015/073955
Authority: WO
Inventors: 一秀梅田
Original assignee: Ｎｅｃソリューションイノベータ株式会社
Priority date: 2014-09-02
Filing date: 2015-08-26
Publication date: 2016-03-10
Also published as: JP6267350B2; JPWO2016035632A1

Abstract

Provided is a data processing device (10), comprising: an attribute information acquisition unit (11) which acquires user attribute information from each user portable terminal which is located in an area within a prescribed distance from an installed device; a photographic image acquisition unit (12) which acquires image data of a photographic image which is photographed within the area; an image processing unit (13) which, by analyzing the photographic image, recognizes persons appearing in the photographic image, estimates the attribute information of each of the recognized persons, and generates estimated attribute information; and an extraction unit (14) which extracts at least one of the attribute information and the estimated attribute information which represent the same person, the estimated attribute information which represents the person who is not represented with the attribute information, or the attribute information which represents the person who is not represented with the estimated attribute information.

Description

Data processing apparatus, data processing system, data processing method and program

The present invention relates to a data processing device, a data processing system, a data processing method, and a program.

Patent Document 1 discloses an information processing apparatus that performs face recognition on a face image and extracts attribute information representing a user's sex and age group.

JP 2010-282590 A

By using the technique disclosed in Patent Document 1, for example, it is possible to analyze a user who has come to a certain spot (store, facility, etc.). However, since the accuracy of face recognition using a face image is not sufficient, the analysis accuracy becomes insufficient.

This invention makes it a subject to provide the new technique for analyzing the user who came to a certain spot.

According to the present invention,
Attribute information acquisition means for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
Captured image acquisition means for acquiring image data of a captured image captured in the area;
Recognizing a person shown in the photographed image by analyzing the photographed image, image processing means for estimating attribute information for each recognized person and generating estimated attribute information;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. An extracting means for extracting at least one of them,
A data processing apparatus is provided.

Moreover, according to the present invention,
Computer
An attribute information acquisition step for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
A captured image acquisition step of acquiring image data of a captured image captured in the area;
An image processing step of recognizing a person appearing in the captured image by analyzing the captured image and generating estimated attribute information by estimating attribute information for each recognized person;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. An extraction process for extracting at least one of the above,
A data processing method for performing the above is provided.

Moreover, according to the present invention,
Computer
Attribute information acquisition means for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
Captured image acquisition means for acquiring image data of a captured image captured in the area;
An image processing means for recognizing a person shown in the captured image by analyzing the captured image and generating estimated attribute information by estimating attribute information for each recognized person;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. Extraction means for extracting at least one of
A program for functioning as a server is provided.

According to the present invention, a new technique for analyzing a user who has come to a certain spot is realized.

The above-described object and other objects, features, and advantages will be further clarified by a preferred embodiment described below and the following drawings attached thereto.

It is a figure for demonstrating an example of the whole image of the data processing system of this embodiment. It is a figure which shows typically an example of the hardware constitutions of the data processor of this embodiment. It is a figure which shows an example of the functional block diagram of the data processor of this embodiment. It is a figure which shows typically an example of the attribute information which the attribute information acquisition part of this embodiment acquires. It is a figure which shows typically an example of the estimation attribute information which the image process part of this embodiment produces | generates. It is a figure which shows typically an example of the attribute information and the estimated attribute information integrated by the extraction part of this embodiment. It is a figure which shows typically an example of the estimation attribute information which the image process part of this embodiment produces | generates. It is a figure which shows typically an example of the attribute information which the attribute information acquisition part of this embodiment acquires. It is a figure which shows typically an example of the estimation attribute information which the image process part of this embodiment produces | generates. It is a figure which shows typically an example of the attribute information which the attribute information acquisition part of this embodiment acquires. It is a figure which shows typically an example of the estimation attribute information which the image process part of this embodiment produces | generates. It is a figure which shows typically an example of the attribute information which the attribute information acquisition part of this embodiment acquires. It is a flowchart which shows an example of the flow of a process of the data processing method of this embodiment. It is a figure for demonstrating an example of the whole image of the data processing system of this embodiment.

Hereinafter, embodiments of the present invention will be described. Each unit and each unit included in each embodiment of the present embodiment includes a CPU (Central Processing Unit), a memory, and a program loaded in the memory (a program stored in the memory from the stage of shipping the device in advance). In addition, it includes hardware such as CD (Compact Disc) and programs downloaded from servers on the Internet, storage units such as hard disks that store the programs, and network connection interfaces. Realized by any combination of It will be understood by those skilled in the art that there are various modifications to the implementation method and apparatus.

The functional block diagram used in the following description of the embodiment shows a functional unit block, not a hardware unit configuration. In these drawings, each device and each terminal are described as being realized by one device, but the means for realizing the same is not limited thereto. That is, it may be a physically separated configuration or a logically separated configuration. In addition, the same code | symbol is attached | subjected to the same component and description is abbreviate | omitted suitably.

First, the overall image of the data processing system of this embodiment will be described with reference to FIG.

The data processing system of this embodiment includes at least one of the data processing device 10, the installation device 20, the user portable terminal 30, and the photographing device 40.

The installation device 20 is installed at a spot (predetermined place) where user information is collected. Examples of such spots include stores, facilities, buildings, and the like, and more specifically, entrances and exits, arbitrary positions within the spots, and the like. The installation device 20 is configured to be communicable with a predetermined wireless communication standard whose communication distance is within a predetermined distance, and performs wireless communication with the user portable terminal 30 that has entered the communication area 20A. In FIG. 1, one installation device 20 is shown, but a plurality of installation devices 20 may be installed at a certain spot. For example, a plurality of installation devices 20 may be installed at an entrance / exit of a certain spot and an arbitrary position inside the spot.

The user portable terminal 30 is carried by each user. The user portable terminal 30 is configured to be able to perform wireless communication according to the wireless communication standard. The user portable terminal 30 stores user attribute information (name, age, sex, etc.) in advance. Then, when the user portable terminal 30 enters the communication area 20A and performs wireless communication with the installation apparatus 20, the user attribute information is sent via the network 1 such as a wireless LAN (Local Area Network) or the Internet accordingly. To the data processing device 10.

The photographing device 40 photographs at least a part of the communication area 20A. Then, the captured image data is transmitted to the data processing device 10 by wired and / or wireless communication. In FIG. 1, the communication between the data processing device 10 and the photographing device 40 and the communication between the data processing device 10 and the user portable terminal 30 are realized by different networks, but may be realized by the same network. . When a plurality of installation devices 20 are installed, the imaging device 40 is installed corresponding to each of the communication areas 20A of the plurality of installation devices 20, and images the inside of each communication area 20A.

The data processing apparatus 10 accumulates user attribute information collected from the user portable terminal 30. In addition, image analysis is performed on the captured image acquired from the imaging device 40 to recognize a person shown in the image and to estimate the attributes (age, sex, etc.) of the recognized person. Hereinafter, an attribute of a user estimated by image analysis is referred to as “estimated attribute information”.

The data processing apparatus 10 analyzes the user who has come to the spot using the user attribute information (including estimated attribute information) collected by two means in this way. The contents of analysis are various, but the following examples can be considered, for example.

(1) Analyzing attributes of users who have come to the spot For example, by using the attribute information and the estimated attribute information, it is possible to analyze the ratio of users who have come to the spot, the gender ratio, the age group, and the like.

It should be noted that the user attribute information or the estimated attribute is to some extent only by either the means for acquiring the user's estimated attribute information using the face image and the means for acquiring the user's attribute information from the user portable terminal 30. Information can be collected and attributes of users who come to the spot can be analyzed. However, in any case, there is a problem that the accuracy is not sufficient.

In the case of means for acquiring estimated attribute information of a user using a face image, for example, there may be a disadvantage that a plurality of users are photographed in an overlapping state. In this case, a situation may occur in which the user located behind cannot be recognized as a person and the estimated attribute information of the user cannot be collected. Further, in the case of means for acquiring user attribute information from the user portable terminal 30, the user attribute information cannot be acquired when the user does not carry the user portable terminal 30 or the power is turned off. Since the user portable terminal 30 is under the control of the user, it is up to the user whether to carry the user portable terminal 30 or whether to turn the power on or off. For this reason, it is difficult to improve such inconvenience and improve accuracy.

On the other hand, in the present embodiment, user attribute information and estimation collected by both means for acquiring estimated user attribute information using a face image and means for acquiring user attribute information from the user portable terminal 30. Using the attribute information, the attribute of the user who came to the spot is analyzed. In this case, the accuracy can be improved by acquiring the attribute information or the estimated attribute information of a certain user that cannot be acquired by one means by the other means.

In addition, in the case of the said means, the attribute information and estimated attribute information which represent the same user twice may generate | occur | produce. Although details will be described later, in the present embodiment, such attribute information and estimated attribute information can be extracted and integrated. As a result, when analyzing the attributes of the users who have come to the spot, it is possible to eliminate inconvenience such as counting a certain user as two or more users.

(2) Extracting a user who does not carry the user portable terminal 30 Although details will be described later, in the present embodiment, attribute information is not acquired by means for acquiring user attribute information from the user portable terminal 30, It is possible to extract a user whose estimated attribute information has been acquired by means for acquiring estimated attribute information of a user using a face image. It can be estimated that such a user is a user who does not carry the user portable terminal 30 or is turned off.

By this analysis, for example, it is possible to analyze the ratio of users who do not carry the user portable terminal 30 among the users who have come to the spot, the gender ratio, the age group and the like. In addition, when it is ruled that the user portable terminal 30 is carried and come to the spot, or when only a user who has been distributed to the user portable terminal 30 can come to the spot, such a rule is broken to become a spot. The user who is coming can be specified. For example, by associating the face image from which the estimated attribute information was generated with each estimated attribute information, not only the estimated age and estimated gender of the user who broke such a rule but also the face image Can be acquired.

(3) Extracting users who could not obtain estimated attribute information using face images As will be described in detail later, in this embodiment, estimated attribute information may be obtained depending on means for acquiring user estimated attribute information using face images. Although not acquired, a user who has acquired attribute information by means for acquiring user attribute information from the user portable terminal 30 can be extracted. By extracting such users, it is possible to analyze the proportion of users who are difficult to obtain estimated attribute information by analysis using face images, the gender ratio, the age group, and the like. In addition to the extracted attribute information, a case where the collection of estimated attribute information using a face image has failed can be accumulated by associating and accumulating a captured image at the timing when the attribute information is acquired.

Hereinafter, the configuration of each device will be described.

<Installation device 20>
The installation device 20 is configured to be communicable with a predetermined wireless communication standard whose communication distance is within a predetermined distance, and performs wireless communication with the user portable terminal 30 that has entered the communication area 20A.

For example, the detection information transmission unit 21 periodically uses a predetermined wireless communication standard in which a communication distance is within a predetermined distance within a communication area (communication area 20A in FIG. 1) within a predetermined distance from the own device. The detection information is transmitted intermittently or intermittently. The detection information may include identification information of the installation device 20. And the user portable terminal 30 located in the said communication area (communication area 20A of FIG. 1) will receive the said detection information. Examples of the wireless communication standard include, but are not limited to, a Bluetooth standard and a wireless LAN standard. The installation device 20 may be, for example, a so-called beacon terminal or a wireless LAN access point, but is not limited thereto.

<User portable terminal 30>
The user portable terminal 30 stores user attribute information in advance. The attribute information includes various names such as the user's name, nickname, age, gender, occupation, and hobby.

In addition, the user portable terminal 30 is configured to be communicable with the same wireless communication standard as the installation device 20. Then, when the user portable terminal 30 enters the communication area 20A of the installation apparatus 20 and communicates with the installation apparatus 20 according to the wireless communication standard, the user attribute information stored therein is correspondingly stored in the data processing apparatus 10. Send to. The user portable terminal 30 transmits date and time information such as the transmission date and time when the attribute information is transmitted to the data processing device 10 and the date and time when the user portable terminal 30 communicates with the installation device 20 in association with the user attribute information to the data processing device 10. May be.

For example, when the user portable terminal 30 enters the communication area 20 </ b> A of the installation apparatus 20, the user portable terminal 30 receives the detection information transmitted from the installation apparatus 20 regularly or intermittently. Then, the user portable terminal 30 transmits user attribute information to the data processing device 10 in response to reception of the detection information. For example, when receiving the detection information, the user portable terminal 30 may write the user attribute information at a predetermined position in the detection information and transmit it to the data processing apparatus 10.

In addition, while the user portable terminal 30 is in the communication area 20A, the user portable terminal 30 continues to receive detection information transmitted from the installation apparatus 20 periodically or intermittently. Each time the user portable terminal 30 receives the detection information, the user portable terminal 30 may transmit user attribute information to the data processing device 10. Alternatively, the user portable terminal 30 may be configured not to transmit the user's attribute information at the second or subsequent reception when the detection information is continuously received from the installation device 20 at a time interval shorter than a predetermined time. Good.

The user portable terminal 30 may realize the above-described function by installing a predetermined application on an existing portable terminal such as a mobile phone such as a smartphone, a tablet terminal, or a portable game machine. Alternatively, the user portable terminal 30 may be a terminal prepared exclusively for the data processing system of the present embodiment.

<Photographing device 40>
The imaging device 40 is installed corresponding to the installation device 20. Then, the imaging device 40 images the communication area 20 </ b> A of the installation device 20. For example, the imaging device 40 may capture a moving image, or may capture a still image periodically (at a longer time interval than the moving image).

The imaging device 40 transmits the captured image data to the data processing device 10 by wired and / or wireless communication. In addition, the imaging device 40 may transmit information that can identify the device itself to the data processing device 10 so that the transmission source of the transmitted image data can be specified. Further, the imaging device 40 may further transmit to the 10 date / time information specifying the imaging date / time of each image data (each frame data).

It is preferable that the photographing device 40 can photograph a user who has entered the communication area 20A without omission. For this reason, the imaging device 40 may be installed so that the whole communication area 20A can be imaged, for example. Alternatively, when there is a point through which the user who enters the communication area 20A always passes, for example, when the entrance / exit of a spot through which the user always passes is included in the communication area 20A, the imaging device 40 takes an image of the point through which the user always passes. It may be set as follows. In this case, the entire communication area 20A may not be captured.

<Data processing apparatus 10>
FIG. 2 is a diagram conceptually illustrating an example of the hardware configuration of the data processing apparatus 10 of the present embodiment. As shown in the figure, the data processing apparatus 10 of this embodiment includes, for example, a CPU 1A, a RAM (Random Access Memory) 2A, a ROM (Read Only Memory) 3A, a display control unit 4A, and a display 5A that are connected to each other via a bus 10A. , An operation reception unit 6A, an operation unit 7A, a communication unit 8A, an auxiliary storage device 9A, and the like. Although not shown, other elements such as an input / output interface connected to an external device by wire, a microphone, and a speaker may be provided. Further, some of the illustrated elements may not be included.

The CPU 1A controls the entire computer of the data processing apparatus 10 together with each element. The ROM 3A includes an area for storing programs for operating the computer, various application programs, various setting data used when these programs operate. The RAM 2A includes an area for temporarily storing data, such as a work area for operating a program. The auxiliary storage device 9A is, for example, an HDD (Hard Disc Drive), and can store a large amount of data.

The touch panel display 5A includes a display device (LED (Light Emitting Diode) display, liquid crystal display, organic EL (Electro Luminescence) display, etc.) and a touch pad. The display control unit 4A reads data stored in a VRAM (Video RAM), performs predetermined processing on the read data, and then sends the data to the touch panel display 5A to display various screens. The operation reception unit 6A receives various operations via the operation unit 7A. The operation unit 7A includes operation keys, operation buttons, switches, a jog dial, a touch panel display, a keyboard, and the like. The communication unit 8A is wired and / or wirelessly connected to a network such as the Internet or a LAN, and communicates with other electronic devices.

FIG. 3 shows an example of a functional block diagram of the data processing apparatus 10. As illustrated, the data processing apparatus 10 includes an attribute information acquisition unit 11, a captured image acquisition unit 12, an image processing unit 13, and an extraction unit 14.

The attribute information acquisition unit 11 acquires user attribute information from each of the user portable terminals 30 located in an area within a predetermined distance from the installation device 20 (communication area 20A in FIG. 1). That is, the attribute information acquisition unit 11 receives the user attribute information transmitted from the user portable terminal 30 as described above. Note that acquisition date information may be associated with the user attribute information acquired by the attribute information acquisition unit 11. The acquisition date / time information here may be, for example, date / time information associated with the attribute information by the user portable terminal 30, or the date / time when the attribute information acquisition unit 11 acquired the attribute information from the user portable terminal 30. It may be the information shown.

FIG. 4 schematically shows an example of a plurality of pieces of attribute information acquired and accumulated by the attribute information acquisition unit 11. In the illustrated example, the attribute information ID (genus ID), the attribute information, and the acquisition date information are associated with each other. The plurality of attribute information shown in FIG. 4 are arranged in the order of acquisition date information. According to the example, the attribute information of the genus ID “A0001” is “Mr. XX, 42 years old, male”, and the acquisition date information is “July 17, 2014 13:17:15”. I understand.

The captured image acquisition unit 12 acquires from the imaging device 40 image data of a captured image captured in an area within a predetermined distance from the installation device 20 (communication area 20A in FIG. 1). That is, the captured image acquisition unit 12 acquires the image data transmitted from the imaging device 40 as described above.

The image processing unit 13 analyzes the captured image of the image data acquired by the captured image acquisition unit 12, thereby recognizing a person shown in the captured image and estimating attribute information for each recognized person to estimate attributes. Generate information. The estimated attribute information may include, for example, the age and gender of the recognized person. Alternatively, the image processing unit 13 may extract a predetermined feature amount by analyzing a captured image for each recognized person. The feature amount may be a feature amount extracted from each person's face, or may be a feature amount (color, shape, etc.) extracted from each person's clothes or belongings.

The image processing unit 13 analyzes the captured image of a certain frame and recognizes a new predetermined person, then tracks the person at the time of analysis of subsequent frames, and continuously captures the images across a plurality of frames. Human image data may be collected as one image processing unit. Then, image analysis may be performed on the image processing unit to generate one estimated attribute information. Even if the person is the same person, after the frame-in, once the frame is out, and then the frame-in again, the captured image before the frame-out is regarded as one image processing unit, and the captured image after the frame-in is again after the frame-out. Can be another image processing unit.

FIG. 5 schematically shows an example of a plurality of estimated attribute information generated by the image processing unit 13. In the illustrated example, an estimated attribute information ID (inferred ID), estimated attribute information, a feature amount, and acquisition date information are associated with each other. The plurality of attribute information shown in FIG. 5 are arranged in order of acquisition date. The acquisition date / time information indicates a range of shooting date / time of one or more frames included in the image processing unit from which each estimated attribute information is generated. According to the example, the estimated attribute information of the attribute ID “B0001” is “40's, male”, the feature amount extracted from the person is “xxx”, and the acquisition date / time information is “2014”. July 17, 2013 13:17:14 to 2014 July 17, 13:17:17 ".

Note that the estimated attribute information may further be associated with image data from which each estimated attribute information is generated.

The extraction unit 14 includes attribute information and estimated attribute information that represent the same person in duplicate, estimated attribute information that represents a person that is not represented by the attribute information, and attribute information that represents a person that is not represented by the estimated attribute information. Extract at least one of

Furthermore, the extraction unit 14 extracts estimated attribute information that represents the same person in duplicate from a plurality of estimated attribute information generated by analyzing each of a plurality of captured images captured at different timings. Also good. The “plurality of captured images captured at different timings” may be captured images having different image processing units as described above. For example, the extraction unit 14 may extract estimated attribute information that represents the same person in duplicate from the plurality of estimated attribute information based on the feature amount of each user extracted by the image processing unit 13.

Hereinafter, an example of processing performed by the extraction unit 14 will be described.
○ Processing example 1 (extract attribute information and estimated attribute information representing the same person in duplicate)
First, the extraction unit 14 extracts estimated attribute information acquired at the same timing as one piece of attribute information determined as a processing target. For example, the estimated attribute information whose acquisition date / time includes the acquisition date / time of the processing target (attribute information) may be extracted as the estimated attribute information acquired at the same timing. Alternatively, estimated attribute information whose difference between the acquisition date representative time (any date within the estimated attribute information acquisition date) and the acquisition date of the processing target (attribute information) is within a predetermined time is acquired at the same timing. The estimated attribute information may be extracted.

Thereafter, the extraction unit 14 calculates the degree of coincidence between the extracted estimated attribute information and the processing target (attribute information) based on the extracted estimated attribute information and a predetermined algorithm using the value of the processing target (attribute information). . Then, based on the calculated degree of coincidence, it is determined whether or not the attribute information to be processed and the extracted estimated attribute information represent the same person. For example, when the degree of coincidence is a predetermined value or more, it may be determined that the attribute information to be processed and the estimated attribute information represent the same person. If there is a plurality of estimated attribute information whose degree of match with the attribute information to be processed is a predetermined value or more, the extraction unit 14 indicates that the estimated attribute information with the highest degree of match and the attribute information to be processed represent the same person. You may judge.

Here, a specific example will be described. For example, it is assumed that the extraction unit 14 sets the genus ID “A0001” in FIG. 4 as a processing target. In this case, only the attribute information of the attribute IDs “B0001” and “B0002” is extracted from the estimated attribute information shown in FIG. 5 as the estimated attribute information acquired at the same timing.

Thereafter, the extraction unit 14 determines the degree of coincidence between the attribute information of the genus ID “A0001” and the estimated attribute information of the attribute ID “B0001”, and the estimation of the attribute information of the attribute ID “A0001” and the attribute ID “B0002”. The degree of coincidence with the attribute information is calculated. Here, the higher degree of coincidence is calculated for the estimated attribute information of the attribute ID “B0001” having the same gender and age group than the attribute ID “B0002”. In addition, the degree of coincidence of the estimated attribute information of the inferred ID “B0001” having the same gender and age group is equal to or greater than a predetermined value. Therefore, the extraction unit 14 determines that the attribute information of the genus ID “A0001” in FIG. 4 and the attribute information of the genus ID “B0001” in FIG. 5 represent the same value person.

○ Processing example 2 (extract attribute information and estimated attribute information representing the same person in duplicate)
First, the extraction unit 14 extracts attribute information acquired at the same timing as one estimated attribute information determined as a processing target. For example, the attribute information included in the acquisition date / time of the processing target (estimated attribute information) may be extracted as the attribute information acquired at the same timing. Or, attribute information acquired at the same timing as attribute information whose difference between the acquisition date and time of the processing target (estimated attribute information) and the representative date and time (any date and time within the acquisition date and time of the estimated attribute information) is within a predetermined time May be extracted as

Thereafter, the extraction unit 14 calculates the degree of coincidence between the processing target (estimated attribute information) and the extracted attribute information based on a predetermined algorithm using the processing target (estimated attribute information) and the value of the extracted attribute information. . Then, based on the calculated matching degree, it is determined whether or not the estimated attribute information to be processed and the extracted attribute information represent the same person. For example, when the degree of coincidence is a predetermined value or more, it may be determined that the estimated attribute information to be processed and the attribute information represent the same person. If there is a plurality of attribute information whose degree of matching with the estimated attribute information to be processed is a predetermined value or more, the extraction unit 14 indicates that the attribute information having the highest degree of matching and the estimated attribute information to be processed represent the same person. You may judge.

Here, a specific example will be described. For example, it is assumed that the extraction unit 14 sets the inheritance ID “B0001” in FIG. 5 as a processing target. In this case, only the attribute information of the genus IDs “A0001” and “A0002” is extracted from the attribute information shown in FIG. 4 as the attribute information acquired at the same timing.

Thereafter, the extraction unit 14 matches the estimated attribute information of the attribute ID “B0001” with the attribute information of the attribute ID “A0001”, and the estimated attribute information of the attribute ID “B0001” and the attribute ID “A0002”. The degree of coincidence with the attribute information is calculated. Here, a higher degree of coincidence is calculated for the attribute information of the genus ID “A0001” having the same gender and age group than the genus ID “A0002”. Further, the degree of coincidence of the attribute information of the genus ID “A0001” having the same gender and age group is a predetermined value or more. Therefore, the extraction unit 14 determines that the estimated attribute information of the attribute ID “B0001” in FIG. 5 and the attribute information of the attribute ID “A0001” in FIG. 4 represent the same value person.

○ Processing example 3 (Extract estimated attribute information representing a person not represented by attribute information)
First, the extraction unit 14 extracts attribute information acquired at the same timing as one piece of estimated attribute information determined as a processing target, for example, by the same method as in Processing Example 2. Here, when attribute information is not extracted, the extraction unit 14 extracts estimated attribute information to be processed as estimated attribute information representing a person not represented by the attribute information.

On the other hand, when the attribute information is extracted, the extraction unit 14 extracts the processing target (estimated attribute information) and the processing target (estimated attribute information) based on a predetermined algorithm using the value of the extracted attribute information. The degree of coincidence with the attribute information is calculated. Then, based on the calculated degree of coincidence, it is determined whether the processing target (estimated attribute information) and the extracted attribute information represent the same person. Then, when there is no attribute information whose degree of coincidence with the processing target is equal to or greater than a predetermined value, the extraction unit 14 estimates the processing target (estimated attribute information) as estimated attribute information representing a person who is not represented by the attribute information. Extract as

Here, a specific example will be described. For example, it is assumed that the extraction unit 14 sets the inheritance ID “B0003” in FIG. 5 as a processing target. In this case, the attribute information acquired at the same timing is not extracted from the attribute information shown in FIG. Therefore, the extraction unit 14 extracts the estimated attribute information of the attribute ID “B0003” as estimated attribute information representing a person who is not represented by the attribute information.

○ Processing example 4 (extract attribute information representing a person not represented by estimated attribute information)
First, the extraction unit 14 extracts estimated attribute information acquired at the same timing as one piece of attribute information determined as a processing target, for example, using the same method as in Processing Example 1. Here, when the estimated attribute information is not extracted, the extraction unit 14 extracts the attribute information to be processed as attribute information representing a person not represented by the estimated attribute information.

On the other hand, when the estimated attribute information is extracted, the extraction unit 14 extracts the processing object (attribute information) based on a predetermined algorithm using the extracted estimated attribute information and the value of the processing object (attribute information). The degree of coincidence with the estimated attribute information is calculated. Then, based on the calculated degree of coincidence, it is determined whether the processing target (attribute information) and the extracted estimated attribute information represent the same person. Then, when there is no estimated attribute information whose degree of coincidence with the processing target is a predetermined value or more, the extraction unit 14 sets the processing target (attribute information) as attribute information representing a person who is not represented by the estimated attribute information. Extract as

Here, a specific example will be described. For example, it is assumed that the extraction unit 14 sets the genus ID “A0003” in FIG. 4 as a processing target. In this case, the estimated attribute information acquired at the same timing is not extracted from the estimated attribute information shown in FIG. Therefore, the extraction unit 14 extracts the attribute information of the genus ID “A0003” as attribute information representing a person who is not represented by the estimated attribute information.

○ Processing Example 5 (Extracts estimated attribute information that represents the same person from multiple estimated attribute information)
The extraction unit 14 determines one piece of estimated attribute information as a processing target. Thereafter, the extraction unit 14 uses other estimated attribute information as a search target, and based on a predetermined algorithm using the estimated attribute information and the feature value, the degree of coincidence with the estimated attribute information to be processed is equal to or greater than a predetermined value. Extract estimated attribute information. Then, the extraction unit 14 determines that the processing target (estimated attribute information) and the extracted estimated attribute information represent the same person.

Here, a specific example will be described. For example, it is assumed that the extraction unit 14 sets the attribute ID “B0026” in FIG. 5 as a processing target. Then, the extraction unit 14 uses other estimated attribute information as a search target, and the degree of coincidence with the estimated attribute information of the estimated ID “B0026” calculated using the estimated attribute information and the feature value is equal to or greater than a predetermined value. Search for the estimated attribute information.

In the case of the example, the extraction unit 14 uses the estimated attribute information and the estimated attribute information of the attribute ID “B0001” whose feature value values match, and the estimated attribute information whose degree of match with the processing target is equal to or greater than a predetermined value That is, it is extracted as estimated attribute information that represents the same person in duplicate.

○ Processing example 6 (extract attribute information that represents the same person from multiple attribute information)
The extraction unit 14 determines one piece of attribute information as a processing target. Thereafter, the extraction unit 14 uses other attribute information as a search target, and extracts attribute information whose attribute information values match. Then, the extraction unit 14 determines that the processing target (attribute information) and the extracted attribute information represent the same person.

Here, a specific example will be described. For example, it is assumed that the extraction unit 14 sets the genus ID “A0027” in FIG. 4 as a processing target. Then, the extraction unit 14 searches for attribute information that matches the attribute information value of the genus ID “A0026” with other attribute information as a search target.

In the case of this example, the extraction unit 14 extracts the attribute information of the genus ID “A0001” having the same attribute information value as attribute information that represents the same person in duplicate.

○ Processing example 7 (arrangement of estimated attribute information and attribute information extracted in processing examples 1 to 6)
By using a part or all of the processing examples 1 to 6, the attribute information acquired by the attribute information acquisition unit 11 and the estimated attribute information generated by the image processing unit 13 are collected as shown in FIG. 6, for example. be able to. In the example illustrated in FIG. 6, attribute information and estimated attribute information that represents the same person in duplicate, a plurality of estimated attribute information that represents the same person in duplicate, and a plurality of attribute information that represents the same person in duplicate are integrated. One user ID is collected. As a result, inconvenience that two or more user IDs exist corresponding to the same person is reduced.

In the illustrated example, a user ID, an attribute information ID (genus ID), an estimated attribute information ID (inferred ID), attribute information or estimated attribute information, and a feature amount are associated with each other. In addition, the acquisition date and the like of the captured image, the attribute information, and / or the estimated attribute information may be associated.

In the case of the illustrated example, the user ID “00001” is a combination of the attribute information of the genus ID “A0001” and “A0027” and the estimated attribute information of the genus ID “B0001” and “B0026”. I understand. In the case of the illustrated example, attribute information that is considered to be highly accurate is preferentially written in the columns of attribute information and estimated attribute information. That is, when both attribute information and estimated attribute information exist, attribute information is described. And when only attribute information exists, attribute information is described, and when only estimated attribute information exists, estimated attribute information is described. When both attribute information and estimated attribute information exist, estimated attribute information may be described, or both may be described.

○ Processing example 8 (extract attribute information and estimated attribute information representing the same person in duplicate)
There are cases where the estimated attribute information of two persons recognized in a certain photographed image substantially match and two attribute information having the same content as the estimated attribute information is acquired at this timing. For example, as shown in FIG. 7, it is assumed that two pieces of estimated attribute information having the same estimated attribute information and having the same acquisition date and time are acquired. Then, as shown in FIG. 8, it is assumed that two pieces of attribute information having similar contents are acquired at this timing. It is impossible to determine which estimated attribute information and which attribute information represent the same person only by the information.

However, “Mr. XX” identified by the attribute information of the genus ID “B0061” in FIG. 8 is the estimated attribute obtained by acquiring attribute information acquired at other timings at other timings with reference to FIG. It is integrated with the information and is associated with the feature quantity “XX”. Based on this information, it can be seen that the feature amount of “Mr. XX” is “O”. Based on the information, the extraction unit 14 can specify that the attribute ID “B0058” and the attribute ID “B0061” are attribute information and estimated attribute information that represent the same person in duplicate. Then, it can be determined that the remaining attribute ID “B0059” and attribute ID “B0062” are attribute information and estimated attribute information that redundantly represent the same person.

In this way, association (integration) of attribute information acquired at other timing (may be before or after the estimated attribute information and attribute information to be processed) and estimated attribute information. Based on the result, it is possible to extract attribute information and estimated attribute information that represent the same person repeatedly.

○ Processing example 9 (extract attribute information and estimated attribute information representing the same person in duplicate)
In some cases, the estimated attribute information of two persons recognized in a certain photographed image almost match and one attribute information having the same content as the estimated attribute information is acquired at this timing. For example, as shown in FIG. 9, it is assumed that two pieces of estimated attribute information whose estimated attribute information matches and whose acquisition date and time overlap are acquired. Then, as shown in FIG. 10, it is assumed that one piece of attribute information having similar contents is acquired at this timing. It is impossible to determine which estimated attribute information and which attribute information represent the same person only by the information.

However, referring to FIG. 6, there is already a user ID associated with a feature amount corresponding to the feature amount corresponding to the attribute ID “B0072” in FIG. Then, based on the fact that the genus ID column of this user ID is empty, it is understood that the estimated attribute information corresponding to this user ID is estimated attribute information representing a person not represented by the attribute information. Based on this information, the extraction unit 14 can specify that the estimated attribute information of the attribute ID “B0072” is estimated attribute information representing a person not represented by the attribute information. Then, it can be determined that the remaining attribute ID “B0073” and attribute ID “A0076” are attribute information and estimated attribute information that redundantly represent the same person.

As described above, the attribute information acquired at other timing (may be before or after the estimated attribute information and attribute information to be processed) may be associated with the estimated attribute information. Based on this, it is possible to extract attribute information and estimated attribute information that represent the same person repeatedly.

○ Processing example 10 (extract attribute information and estimated attribute information representing the same person in duplicate)
In some cases, three or more persons are recognized in a certain captured image, and three pieces of estimated attribute information are generated. One attribute information may be acquired at this timing. For example, as shown in FIG. 11, it is assumed that three pieces of estimated attribute information having the acquisition dates and times are acquired. Then, as shown in FIG. 12, it is assumed that one piece of attribute information is acquired at this timing.

In this case, the extraction unit 14 can first determine that the attribute ID “B0093” and the attribute ID “A0101” do not represent the same person based on the degree of coincidence between the attribute information and the estimated attribute information.

Next, referring to FIG. 6, there is already a user ID associated with a feature amount corresponding to the feature amount corresponding to the inheritance ID “B0092” in FIG. Then, based on the fact that the genus ID column of this user ID is empty, it is understood that the estimated attribute information corresponding to this user ID is estimated attribute information representing a person not represented by the attribute information. Based on this information, the extraction unit 14 can specify that the estimated attribute information of the attribute ID “B0092” is estimated attribute information representing a person not represented by the attribute information. Then, it can be determined that the remaining attribute ID “B0094” and attribute ID “A0101” are attribute information and estimated attribute information that redundantly represent the same person.

Next, an example of the processing flow of the data processing method of this embodiment will be described using the flowchart of FIG. The process flow is merely an example, and the present invention is not limited to this. Here, the attribute information acquired by the attribute information acquisition unit 11 (see FIG. 4) and the estimated attribute information generated by the image processing unit 13 (see FIG. 5) are integrated to generate integrated data (see FIG. 6). The processing to be performed will be described.

First, the extraction unit 14 acquires one piece of attribute information or estimated attribute information and sets it as a processing target (S10).

For example, the attribute information acquisition unit 11 may sequentially input the attribute information acquired from the user portable terminal 30 to the extraction unit 14 by real-time processing. The captured image acquisition unit 12 may acquire image data from the imaging device 40 by real-time processing. Then, the image processing unit 13 may generate the estimated attribute information by analyzing the image data by real-time processing and input the estimated attribute information to the extraction unit 14. The extraction unit 14 may process the attribute information and the estimated attribute information input from the attribute information acquisition unit 11 and the image processing unit 13 as described above in the order of input and process them in real time.

As another example, the extraction unit 14 may batch-process a plurality of attribute information acquired by the attribute information acquisition unit 11 from the user portable terminal 30 and a plurality of estimated attribute information generated by the image processing unit 13. . In this case, the extraction unit 14 may determine a processing target in order of acquisition date and time from the group of attribute information and estimated attribute information.

Thereafter, the extraction unit 14 performs integration processing based on the processing examples 1 to 10 using the processing target and the integrated data (S11). The processing flow of the integration process is not particularly limited, and various aspects can be adopted. Thereafter, the same processing is repeated (S12).

Here, a modified example of the present embodiment will be described with reference to FIG. In the data processing system of the modification, the user portable terminal 30 transmits detection information including user attribute information in the communication area 30A, and the installation device 20 located in the communication area 30A receives the detection information. And the installation apparatus 20 which received the detection information transmits a user's attribute information to the data processing apparatus 10 according to the said reception. The data processing system of the modification is different from the above example in this respect. Other configurations are the same as in the above example. In the case of this example, the user portable terminal 30 transmits user attribute information to the data processing device 10 via (via) the installation device 20.

The user portable terminal 30 of the modification may be a terminal prepared exclusively for the data processing system of the present embodiment, for example, a beacon terminal. The installation device 20 and the data processing device 10 are communicated by wire and / or wireless.

Next, the function and effect of this embodiment will be described.

According to the present embodiment, the user attribute information (estimated attribute information is obtained by both the means for acquiring the user's estimated attribute information using the face image and the means for acquiring the user's attribute information from the user portable terminal 30. Can be collected). And a predetermined analysis can be performed using the information collected in this way. For example, an attribute of a user who has come to a predetermined spot is analyzed, a user who has come to a predetermined spot, a user who does not carry the user portable terminal 30 is extracted, the attribute is analyzed, and a face image is used Thus, it is possible to extract users who could not obtain estimated attribute information and analyze the attributes or accumulate the data.

As a result, the accuracy of analysis can be improved as compared with the case where user attribute information (including estimated attribute information) is acquired and analyzed by either means. In addition, the range of contents that can be analyzed can be expanded.

Hereinafter, examples of the reference form will be added.
1. Attribute information acquisition means for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
Captured image acquisition means for acquiring image data of a captured image captured in the area;
Recognizing a person shown in the photographed image by analyzing the photographed image, image processing means for estimating attribute information for each recognized person and generating estimated attribute information;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. An extracting means for extracting at least one of them,
A data processing apparatus.
2. In the data processing apparatus according to 1,
The extraction means further extracts the estimated attribute information that represents the same person from a plurality of the estimated attribute information generated by analyzing each of the plurality of captured images captured at different timings. Data processing device.
3. In the data processing apparatus according to 2,
The image processing means extracts a feature amount by analyzing the captured image for each person recognized by the analysis of the captured image,
The data processing device extracts the estimated attribute information that represents the same person in duplicate from the plurality of estimated attribute information based on the feature amount.
4). An installation device installed at a predetermined location and capable of communicating with a predetermined wireless communication standard having a communication distance within a predetermined distance;
A user portable terminal that is carried by the user, can communicate with the wireless communication standard, and enters the area within the predetermined distance from the installation device, and communicates with the installation device with the wireless communication standard;
A photographing device for photographing the inside of the area;
A data processing device according to any one of 1 to 3,
A data processing system.
5. 4, the data processing system according to
When the user portable terminal communicates with the installation apparatus according to the wireless communication standard, the user portable terminal transmits user attribute information stored in advance in the terminal to the data processing apparatus according to the communication.
6). In the data processing system according to 5,
The data processing system in which the user portable terminal transmits the attribute information to the data processing device via the installation device.
7). Computer
An attribute information acquisition step for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
A captured image acquisition step of acquiring image data of a captured image captured in the area;
An image processing step of recognizing a person appearing in the captured image by analyzing the captured image and generating estimated attribute information by estimating attribute information for each recognized person;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. An extraction process for extracting at least one of the above,
Data processing method to execute.
7-2. In the data processing method according to claim 7,
In the extracting step, the estimated attribute information that represents the same person is further extracted from the plurality of estimated attribute information generated by analyzing each of the plurality of captured images captured at different timings. Data processing method.
7-3. In the data processing method described in 7-2,
In the image processing step, for each person recognized by the analysis of the captured image, the feature amount is extracted by analyzing the captured image,
A data processing method for extracting the estimated attribute information representing the same person in duplicate from the plurality of estimated attribute information based on the feature amount in the extracting step.
8). Computer
Attribute information acquisition means for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
Captured image acquisition means for acquiring image data of a captured image captured in the area;
An image processing means for recognizing a person shown in the captured image by analyzing the captured image and generating estimated attribute information by estimating attribute information for each recognized person;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. Extraction means for extracting at least one of
Program to function as.
8-2. In the program described in 8,
The extraction means further extracts the estimated attribute information that represents the same person from among the plurality of estimated attribute information generated by analyzing each of the plurality of captured images captured at different timings. Program to make.
8-3. In the program described in 8-2,
For each person recognized by the analysis of the captured image, the image processing means causes the feature amount to be extracted by analyzing the captured image,
A program that causes the extraction means to extract the estimated attribute information that represents the same person from a plurality of the estimated attribute information based on the feature amount.

This application claims priority based on Japanese Patent Application No. 2014-177858 filed on September 2, 2014, the entire disclosure of which is incorporated herein.

Claims

Attribute information acquisition means for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
Captured image acquisition means for acquiring image data of a captured image captured in the area;
Recognizing a person shown in the photographed image by analyzing the photographed image, image processing means for estimating attribute information for each recognized person and generating estimated attribute information;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. An extracting means for extracting at least one of them,
A data processing apparatus.
The data processing apparatus according to claim 1,
The extraction means further extracts the estimated attribute information that represents the same person from a plurality of the estimated attribute information generated by analyzing each of the plurality of captured images captured at different timings. Data processing device.
The data processing apparatus according to claim 2, wherein
The image processing means extracts a feature amount by analyzing the captured image for each person recognized by the analysis of the captured image,
The data processing device extracts the estimated attribute information that represents the same person in duplicate from the plurality of estimated attribute information based on the feature amount.
An installation device installed at a predetermined location and capable of communicating with a predetermined wireless communication standard having a communication distance within a predetermined distance;
A user portable terminal that is carried by the user, can communicate with the wireless communication standard, and enters the area within the predetermined distance from the installation device, and communicates with the installation device with the wireless communication standard;
A photographing device for photographing the inside of the area;
A data processing device according to any one of claims 1 to 3,
A data processing system.
The data processing system according to claim 4, wherein
When the user portable terminal communicates with the installation apparatus according to the wireless communication standard, the user portable terminal transmits user attribute information stored in advance in the terminal to the data processing apparatus according to the communication.
The data processing system according to claim 5, wherein
The data processing system in which the user portable terminal transmits the attribute information to the data processing device via the installation device.
Computer
An attribute information acquisition step for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
A captured image acquisition step of acquiring image data of a captured image captured in the area;
An image processing step of recognizing a person appearing in the captured image by analyzing the captured image and generating estimated attribute information by estimating attribute information for each recognized person;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. An extraction process for extracting at least one of the above,
Data processing method to execute.
Computer
Attribute information acquisition means for acquiring user attribute information from each of the user portable terminals located in an area within a predetermined distance from the installation device;
Captured image acquisition means for acquiring image data of a captured image captured in the area;
An image processing means for recognizing a person shown in the captured image by analyzing the captured image and generating estimated attribute information by estimating attribute information for each recognized person;
The attribute information and the estimated attribute information representing the same person in duplicate, the estimated attribute information representing a person not represented by the attribute information, and the attribute information representing a person not represented by the estimated attribute information. Extraction means for extracting at least one of
Program to function as.