JP6757009B1 - Computer program, object identification method, object identification device and object identification system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a computer program, an object identification method, an object identification device and an object identification system capable of specifying a correspondence relationship between an image data of an object captured by a camera and a sensor of the object. An object identification system according to the present embodiment acquires sensor data from a camera, a sensor provided on the object, an image data acquisition unit that acquires image data captured by the camera, and the sensor. Based on the sensor data acquisition unit, the image data acquired by the image data acquisition unit, and the sensor data acquired by the sensor data acquisition unit, the correspondence between the image data obtained by photographing the object and the sensor is specified. It is provided with an object identification device having a specific unit. [Selection diagram] Fig. 1

Description

The present invention relates to a computer program for specifying image data obtained by photographing an object with a camera, an object identification method, an object identification device, and an object identification system.

Technology that authenticates a person based on image data taken by a camera is beginning to be widely used. For example, in a face recognition system, a person is authenticated by comparing a face image or a feature amount thereof stored in advance with a face of a person taken by a camera or a feature amount thereof.

Patent Document 1 describes an authentication system in which a person's face is photographed and the vein of the person is photographed, and authentication is performed by using face authentication and vein authentication in combination. This authentication system reduces the processing time of vein authentication by performing vein authentication using the vein authentication registration data corresponding to the person who succeeded in the authentication by face authentication.

JP-A-2019-128880

However, in the conventional face recognition technique, this person cannot be authenticated unless the face image of the person to be authenticated or the feature amount thereof is registered in a database or the like in advance. This also applies to the authentication system described in Patent Document 1.

The present invention has been made in view of such circumstances, and an object of the present invention is a computer program for specifying a correspondence relationship between image data obtained by photographing an object with a camera and a sensor possessed by the object. The purpose of the present invention is to provide an object identification method, an object identification device, and an object identification system.

The computer program according to the embodiment acquires image data taken by a camera on a computer, acquires sensor data from a sensor possessed by the object , estimates the mounting position of the sensor with respect to the target, and estimates the mounting. Based on the position and the acquired image data and the sensor data , a process of specifying the correspondence between the image data obtained by photographing the object and the sensor is executed.

In the case of one embodiment, it is possible to specify the correspondence between the image data obtained by capturing the object with the camera and the sensor of the object.

It is a schematic diagram for demonstrating the outline of the object identification system which concerns on this Embodiment. It is a schematic diagram for demonstrating the outline of data transmission | transmission of the object identification system which concerns on this Embodiment. It is a block diagram which shows the structure of the server apparatus which concerns on this embodiment. It is a schematic diagram which shows one configuration example of a specific information DB. It is a block diagram which shows the structure of the user terminal apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating the calculation process of the feature amount from the image data performed by a server apparatus. It is a schematic diagram which shows an example of the sensor data acquired by a server device. It is a schematic diagram for demonstrating the identification method by a server apparatus. It is a flowchart which shows an example of the procedure of the specific processing performed by the server apparatus which concerns on this Embodiment. It is a flowchart which shows an example of the procedure of the specific processing performed by the server apparatus which concerns on this Embodiment. It is a flowchart which shows an example of the procedure of the authentication processing performed by the server apparatus which concerns on this embodiment. It is a flowchart which shows an example of the procedure of the specific processing which combined with the face authentication performed by the server apparatus which concerns on this embodiment. It is a block diagram which shows the structure of the server apparatus which concerns on Embodiment 2. It is a schematic diagram which shows one structural example of the learning model provided in the server apparatus which concerns on Embodiment 2. FIG. It is a flowchart which shows an example of the procedure of the specific processing performed by the server apparatus which concerns on Embodiment 2. It is a flowchart which shows an example of the procedure of the specific processing performed by the server apparatus which concerns on Embodiment 2. It is a schematic diagram for demonstrating the outline of the identification process performed by the object identification system which concerns on the modification of Embodiment 2. It is a flowchart which shows an example of the procedure of the specific processing performed by the server apparatus which concerns on the modification of Embodiment 2. It is a schematic diagram for demonstrating the outline of the identification process performed by the object identification system which concerns on Embodiment 3. FIG. It is a schematic diagram for demonstrating the outline of the identification process performed by the object identification system which concerns on Embodiment 3. FIG. It is a schematic diagram for demonstrating the outline of data transmission | transmission of the object identification system which concerns on modification 1 of Embodiment 3.

A specific example of the object identification system according to the embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

<System overview>
FIG. 1 is a schematic diagram for explaining the outline of the object identification system according to the present embodiment. The object identification system according to the present embodiment is a process for identifying the correspondence between the image data obtained by photographing a person (user) from the image data obtained from the camera 2 and the user terminal device 1 possessed by each person. It is a system that performs. The camera 2 may be installed, for example, in a room or corridor inside the building, or may be installed, for example, on a road or garden outside the building. The object identification system may include a plurality of cameras 2. In the present embodiment, when it is necessary to distinguish between a plurality of cameras 2, they will be described with different reference numerals such as cameras 2A and 2B.

Further, in the present embodiment, each user possesses at least one user terminal device 1. Although the user terminal device 1 is shown as a smartphone in FIG. 1, the user terminal device 1 is not limited to a smartphone. The user terminal device 1 may be, for example, a wearable device, or may be, for example, a set of a wearable device and a smartphone. Further, the user terminal device 1 may be a device such as a key holder type or a card type device, or may be a device such as an embedded tag or the like, or may be a device other than these. In the present embodiment, the user terminal device 1 includes at least a sensor that detects the movement or activity of the user. As the sensor included in the user terminal device 1, for example, a sensor that detects acceleration, angular velocity, or geomagnetism can be adopted. Further, in the present embodiment, the sensor may include a functional unit that receives a GPS (Global Positioning System) signal and identifies a position.

Further, it is preferable that the user terminal device 1 has a wireless communication function of transmitting sensor data, which is a detection result of the sensor, to a remote server device or the like that performs specific processing. However, when it is not necessary to identify the correspondence between the image data of the person captured by the camera 2 and the user terminal device 1 in real time, the user terminal device 1 has a function of storing the sensor data and is connected by wire after the fact. Alternatively, the sensor data may be exchanged via a recording medium or the like.

In the present embodiment, when it is necessary to distinguish the user terminal device 1, for example, the user terminal device 1A of the user A, the user terminal device 1B of the user B, the user terminal device 1C of the user C, and the user D. It will be described with different reference numerals as in the user terminal device 1D. In the example shown in FIG. 1, it is specified that the person imaged by the camera 2A is the user A who possesses the user terminal device 1 device A. Further, it is specified that the three persons photographed by the camera 2B are the user B who possesses the user terminal device 1B, the user C who possesses the user terminal device 1C, and the user D who possesses the user terminal device 1D. There is.

FIG. 2 is a schematic diagram for explaining an outline of data transmission / reception of the object identification system according to the present embodiment. In the object identification system according to the present embodiment, the server device 3 performs the identification process. The image data taken by each camera 2 and the sensor data detected by the sensor of the user terminal device 1 owned by the user are transmitted to the server device 3 via a wired or wireless network. The camera 2 takes pictures at a cycle of several to several tens of times per second, and transmits the taken image data to the server device 3. A camera ID that identifies the camera 2 is attached to the image data transmitted from the camera 2. Similarly, the user terminal device 1 detects the movement or activity of the user by the sensor at a cycle of several to several tens of times per second, and transmits the sensor data as the detection result to the server device 3. The sensor data transmitted from the user terminal device 1 is given a user ID that identifies the user, a terminal ID that identifies the user terminal device 1, and the like.

The server device 3 according to the present embodiment calculates a time-series displacement due to the movement or activity of the user based on a plurality of time-series image data transmitted by the camera 2. Further, the server device 3 receives a plurality of time-series sensor data transmitted by the user terminal device 1 owned by the user. There is a relationship between the displacement caused by the movement or activity of the user and the change in the sensor data of the user terminal device 1 possessed by the user. Therefore, the server device 3 according to the present embodiment corresponds between the user and the user terminal device 1 based on the relationship between the displacement of the user based on the image data of the camera 2 and the change in the sensor data of the user terminal device 1. Identify.

As a result, for example, the server device 3 determines whether or not the person who is going to enter the building possesses the predetermined user terminal device 1 based on the image data taken by the camera 2 installed at the entrance of the building or the like. be able to. In the predetermined user terminal device 1, for example, a user terminal device 1 in which a user ID or a terminal ID or the like is registered in advance in the server device 3, an application program for acquiring sensor data and transmitting the sensor data to the server device 3 is installed. User terminal device 1. When the person photographed by the camera 2 is a person who does not possess the user terminal device 1, the server device 3 can determine that this person may be an unauthorized intruder. In the object identification system according to the present embodiment, it is possible to determine whether or not the person photographed by the camera 2 possesses the predetermined user terminal device 1, and this is used for person authentication or the like. Can be done. The server device 3 can authenticate a person who possesses a predetermined user terminal device 1 as a legitimate user, and can determine a person who does not possess the user terminal device 1 as a suspicious person.

In the present embodiment, a specific object is a human being (user, person), but the object is not limited to this. The object may be an animal such as a dog or a cat. In this case, for example, a key holder type terminal device having a sensor mounted on a collar or the like is attached to the animal, and the server device 3 specifies the correspondence between the animal and the terminal device photographed by the camera 2. When the object is a human, the human may be riding a vehicle such as a bicycle, a wheelchair, or a motorcycle. The object is not limited to animals (living organisms) such as humans and dogs, but may be moving objects such as automobiles or motorcycles. In this case, a device such as a navigation device mounted on the moving body can play the role of the user terminal device 1.

<Device configuration>
FIG. 3 is a block diagram showing a configuration of the server device 3 according to the present embodiment. The server device 3 according to the present embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, and the like. In the present embodiment, it is assumed that the processing is performed by one server device 3, but a plurality of server devices 3 may be distributed to perform the processing.

The processing unit 31 uses an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. It is composed of. The processing unit 31 reads and executes the server program 32a stored in the storage unit 32 to perform various processes related to specifying the correspondence between the user image data captured by the camera 2 and the user terminal device 1.

The storage unit 32 is configured by using a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31. In the present embodiment, the storage unit 32 stores the server program 32a executed by the processing unit 31. Further, the storage unit 32 has a specific information DB (database) 32b that stores specific information that is a specific result of the correspondence between the user and the user terminal device 1 captured by the camera 2, and an image captured by the camera 2. An image storage unit 32c for storing the above is provided.

In the present embodiment, the server program 32a is provided in a mode of being recorded on a recording medium 99 such as a memory card or an optical disk, and the server device 3 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. However, the server program 32a may be written to the storage unit 32, for example, at the manufacturing stage of the server device 3. Further, for example, in the server program 32a, the server device 3 may acquire what is distributed by another remote server device or the like by communication. For example, in the server program 32a, the writing device may read what was recorded on the recording medium 99 and write it in the storage unit 32 of the server device 3. The server program 32a may be provided in the form of distribution via the network, or may be provided in the form recorded on the recording medium 99.

The specific information DB 32b is information on a specific result of the correspondence between the user's image data captured by the camera 2 and the user terminal device 1. In the specific information DB 32b, image data captured by the camera 2, identification information such as the user ID or terminal ID of the specified user, and information such as the position of the user in the image data are stored in association with each other. .. The configuration of the specific information DB 32b will be described later.

The image storage unit 32c stores image data taken by the camera 2. As described above, the image data of the user and the information regarding the specific result of the correspondence of the user terminal are stored in the specific information DB 32b. However, the actual image data taken by the camera 2 is stored in the image storage unit 32c, and information such as the file name of the image data or the address of the storage location is stored in the specific information DB 32b. In the present embodiment, the server device 3 is provided with the image storage unit 32c, but the present invention is not limited to this, and a storage device different from the server device 3 may be configured to store images.

The communication unit 33 communicates with various devices via a network N including the Internet, a wireless LAN (Local Area Network), a mobile phone communication network, and the like. In the present embodiment, the communication unit 33 communicates with the camera 2 and the user terminal device 1 via the network N. The communication unit 33 transmits the data given by the processing unit 31 to another device, and gives the data received from the other device to the processing unit 31.

The storage unit 32 may be an external storage device connected to the server device 3. Further, the server device 3 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the server device 3 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that accepts operation input, a display unit that displays an image, and the like. ..

Further, in the processing unit 31 of the server device 3 according to the present embodiment, the processing unit 31 reads out and executes the server program 32a stored in the storage unit 32, so that the image data acquisition unit 31a and the sensor data acquisition unit 31b And the specific unit 31c and the like are realized as software-like functional units. It should be noted that these functional units are functional units related to processing for specifying the correspondence between the user and the user terminal device 1 photographed by the camera 2, and illustration and description of other functional units will be omitted.

The image data acquisition unit 31a performs a process of acquiring image data captured by the camera 2 by communicating with the camera 2 in the communication unit 33. The image data acquisition unit 31a acquires the image data transmitted by the camera 2 by receiving it in the communication unit 33, and stores and stores the acquired image data in the image storage unit 32c of the storage unit 32.

The sensor data acquisition unit 31b performs a process of acquiring sensor data, which is a detection result of the sensor of the user terminal device 1, by communicating with the user terminal device 1 in the communication unit 33. The sensor data acquisition unit 31b acquires the sensor data transmitted by the user terminal device 1 by receiving it in the communication unit 33, and stores the acquired sensor data in the storage unit 32. The sensor data stored in the storage unit 32 may be deleted from the storage unit 32 after being used for the corresponding identification process of the user and the user terminal device 1.

The specific unit 31c is based on the image data of the camera 2 acquired by the image data acquisition unit 31a and the sensor data of the user terminal device 1 acquired by the sensor data acquisition unit 31b, and the image data captured by the user and the user terminal device. Perform the process of specifying the correspondence with 1. Although the details will be described later, the specific unit 31c is based on the relationship between the displacement caused by the movement or activity of the user captured by the camera 2 and the change in the sensor data of the user terminal device 1 possessed by the user. The correspondence between the photographed image data and the user terminal device 1 is specified. The specific unit 31c stores and stores the information regarding the specified correspondence in the specific information DB 32b of the storage unit 32.

FIG. 4 is a schematic diagram showing a configuration example of the specific information DB 32b. The specific information DB 32b is, for example, a database in which "image data" and one or more "specific information" are associated with each other. The "specific information" includes information such as "terminal ID" and "coordinate information". In the example shown in FIG. 4, two "specific information" are associated with the "image data", but the "specific information" may exist as many as the number of users specified from the "image data".

The "image data" is digital data of an image taken by the camera 2. In the present embodiment, the substance of the image data is stored in the image storage unit 32c of the storage unit 32, and the "image data" of the specific information DB 32b includes the file name or the storage location (address) of the image data which is the substance. Information is stored. However, the substance of the image data may be stored in the specific information DB 32b. As the image data, for example, when the data acquired from the camera 2 is saved as a moving image, a file format such as MPEG (Moving Picture Experts Group), AVI (Audio Video Interleave) or FLV (Flash Video) can be adopted. When the image data is saved as a still image, a file format such as PNG (Portable Network Graphics), GIF (Graphics Interchange Format) or JPEG (Joint Photographic Experts Group) can be adopted. However, these file formats are examples, and the image data may be saved in any format.

The "specific information" is information about the user terminal device 1 specified with respect to the image data of the camera 2. In the specific information DB 32b, one "specific information" is stored for one specified user terminal device 1.

The "terminal ID" included in the "specific information" is an ID assigned to the specified user terminal device 1. In the present embodiment, a terminal ID is attached to the sensor data transmitted by the user terminal device 1, and the terminal ID attached to the sensor data used for identification is stored in the specific information DB 32b. When the terminal ID of the user terminal device 1 and the user ID of the user who owns the terminal ID are linked in advance, the server device 3 can specify the user ID based on the terminal ID, and the user terminal device 1 can specify the user ID. Can be identified as the user who owns the. In the "specific information" of the specific information DB 32b, a "user ID" assigned to the user may be stored in place of the "terminal ID" or together with the "terminal ID".

The "coordinate information" included in the "specific information" is coordinates indicating a range taken by the user in the image data. In this example, it is assumed that the shooting range of the user in the image data is rectangular, and the coordinates of the two vertices existing at the physique position of this rectangle are, for example, (coordinates X1, coordinates Y1)-(coordinates X2, coordinates Y2). It is shown as.

FIG. 5 is a block diagram showing the configuration of the user terminal device 1 according to the present embodiment. The user terminal device 1 according to the present embodiment includes a processing unit 11, a storage unit (storage) 12, a sensor 13, a communication unit (transceiver) 14, a display unit (display) 15, an operation unit 16, a GPS receiving unit 17, and the like. It is configured to prepare.

The processing unit 11 is configured by using an arithmetic processing unit such as a CPU or MPU, a ROM, a RAM, and the like. The processing unit 11 reads and executes the program 12a stored in the storage unit 12 to perform various processes related to acquisition and transmission of the detection result of the sensor 13.

The storage unit 12 is configured by using a non-volatile memory element such as a flash memory. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for processing by the processing unit 11. In the present embodiment, the storage unit 12 stores the program 12a executed by the processing unit 11 and the terminal ID 12b attached to the user terminal device 1.

In the present embodiment, the program 12a is distributed by a remote server device or the like, which is acquired by the user terminal device 1 by communication and stored in the storage unit 12. However, the program 12a may be written in the storage unit 12 at the manufacturing stage of the user terminal device 1, for example. For example, in the program 12a, the user terminal device 1 may read the program 12a recorded on a recording medium such as a memory card or an optical disk and store it in the storage unit 12. For example, in the program 12a, the writing device may read what has been recorded on the recording medium and write it in the storage unit 12 of the user terminal device 1. The program 12a may be provided in a mode of distribution via a network, or may be provided in a mode recorded on a recording medium.

The terminal ID 12b is an identifier that uniquely identifies the user terminal device 1, is determined, for example, at the manufacturing stage of the user terminal device 1, and is stored in the storage unit 12. The terminal ID 12b may be, for example, information such as the serial number of the user terminal device 1, or may be, for example, a dedicated ID for the object identification system according to the present embodiment.

The sensor 13 is, for example, a sensor such as an acceleration sensor, an angular velocity sensor (gyro sensor), or a geomagnetic sensor. In the present embodiment, one of these sensors is used for the specific process, but the user terminal device 1 may be equipped with a plurality of sensors, and a plurality of types of sensors are used for the specific process. May be done. The sensor 13 outputs an electric signal according to a physical quantity such as the detected acceleration or angular velocity, or a digital data of the electric signal to the processing unit 11. In the present embodiment, the electric signal or digital data output by the sensor 13 is referred to as sensor data.

The communication unit 14 communicates with various devices via a public mobile phone communication network and a network N including the Internet and the like. In the present embodiment, the communication unit 14 communicates with the server device 3 via the network N. The communication unit 14 transmits the data given by the processing unit 11 to the server device 3, and also gives the data received from the server device 3 to the processing unit 11.

The display unit 15 is configured by using a liquid crystal display or the like, and displays various images, characters, and the like based on the processing of the processing unit 11.

The operation unit 16 accepts a user's operation by an input device such as a mechanical button or a touch panel provided on the surface of the display unit 15, and notifies the processing unit 11 of the accepted operation.

The GPS receiving unit 17 receives radio waves transmitted from a GPS artificial satellite and gives information included in the received radio waves to the processing unit 11. The processing unit 11 can perform processing for specifying the position (for example, latitude and longitude) of the user terminal device 1 based on the information given from the GPS receiving unit 17.

Further, in the user terminal device 1 according to the present embodiment, the processing unit 11 reads and executes the program 12a stored in the storage unit 12, so that the sensor data transmitting unit 11a and the like act as software-like functional units in the processing unit 11. Will be realized. The sensor data transmission unit 11a acquires the sensor data output by the sensor 13 and transmits the acquired sensor data to the server device 3. At this time, the sensor data transmission unit 11a transmits the sensor data to the server device 3 with the terminal ID 12b stored in the storage unit 12 and information such as the date and time when the sensor data was acquired. The sensor 13 outputs sensor data at all times or repeatedly at a predetermined cycle. For example, the sensor data transmission unit 11a may transmit the acquired sensor data to the server device 3 each time the sensor data is acquired from the sensor 13. Further, for example, the sensor data transmission unit 11a may store the sensor data acquired from the sensor 13 in the storage unit 12 and transmit the stored one or more sensor data to the server device 3 at a predetermined cycle.

The block diagram showing the configuration of the camera 2 included in the object identification system according to the present embodiment is omitted. The camera 2 is an image processing IC (Integrated) that generates image data by performing predetermined image processing on an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) and an output signal of the image sensor. Circuit) and a communication unit that performs wired or wireless communication are provided. The camera 2 shoots at a cycle of several to several tens of times per second, and transmits the image data obtained by the shooting to the server device 3. At this time, the camera 2 attaches information such as the camera ID and the shooting date and time attached to itself to the image data and transmits the information to the server device 3.

<Specific processing of correspondence between users and user terminal devices>
The server device 3 according to the present embodiment performs a process of specifying the correspondence between the user captured in the image data acquired from the camera 2 and the user terminal device 1 possessed by the user. In this specific process, the server device 3 receives the image data from the camera 2, extracts the area captured by the user from the received image data, and calculates the feature amount of the user from this image area. The server device 3 sequentially receives the image data transmitted from the camera 2 and sequentially calculates the feature amount from the image data to calculate the time-series change in the feature amount of the user in the image data. As a result, the server device 3 can calculate the displacement due to the movement or activity of the user in the image data. The feature amount calculated by the server device 3 may include, for example, the angle of the user's skeleton, the height (length) from the top of the head to the tip of the foot, the ground contact timing of the foot, and the like, but is not limited thereto.

FIG. 6 is a schematic diagram for explaining the process of calculating the feature amount from the image data performed by the server device 3. In the example shown on the left side of FIG. 6, three people are photographed in the image data 101 acquired by the server device 3 from the camera 2. The server device 3 performs a process of extracting an area in which a person is photographed from the acquired image data. Since the process of extracting the shooting area of a person from the image data is an existing technique, detailed description thereof will be omitted. In FIG. 6, three areas 102 to 104 extracted by the server device 3 are shown as rectangular frames.

The server device 3 calculates a feature amount indicating the characteristics of a person photographed in this area for each extracted area. Here, for example, the server device 3 calculates the height from the top of the photographed person to the toes as a feature amount. The server device 3 repeatedly performs the same area extraction and feature amount calculation for a plurality of image data photographed by the camera 2 in time series, and acquires a change in the feature amount of the person in the image data. On the right side of FIG. 6, the changes in the feature amounts calculated for each of the three persons are shown as a graph.

Further, the server device 3 receives the sensor data transmitted from the user terminal device 1. FIG. 7 is a schematic diagram showing an example of sensor data acquired by the server device 3. Note that in FIG. 7, three sensor data acquired by the server device 3 from the user terminal device 1 possessed by the three persons captured in the image data shown in FIG. 6 are arranged vertically as three graphs. It is shown. In each graph, the horizontal axis is the time t and the vertical axis is the output value of the sensor. The three sensor data shown in FIG. 7 are, in order from the top, sensor data acquired from the user terminal device 1a, sensor data acquired from the user terminal device 1b, and sensor data acquired from the user terminal device 1c.

FIG. 8 is a schematic diagram for explaining a specific method by the server device 3. FIG. 8 shows a graph of the sensor data acquired from the user terminal device 1a shown in FIG. 7 and a graph of the feature amount extracted from the area 102 shown in FIG. 6 side by side. The server device 3 has a person in the image data and the user terminal device 1 based on the correlation between the change in the sensor data acquired from the user terminal device 1a and the change in the feature amount extracted from the image data of the camera 2. Identify the response. The server device 3 compares, for example, the timing at which a peak occurs in a change in sensor data and a feature amount. In the example shown in FIG. 8, both timings coincide with each other at the timings shown by the broken lines. In the server device 3, when both timings match or are close to each other, a person photographed in the area 102 from which the feature amount is calculated is a user who possesses the user terminal device 1a that is the source of the sensor data. Can be identified as being.

In this example, the correspondence of the user terminal device 1 is specified by comparing the timing at which the peak occurs with respect to the change in the sensor data and the change in the feature amount, but the specific method determines the peak occurrence timing. It is not limited to the method of comparison. For example, the rate of change of the sensor data and the feature amount, the period of change, the frequency, and the like may be compared, and the characteristics of changes other than these may be compared.

Further, in this example, the processing for specifying the user terminal device 1 is performed based on the correlation between the change in the sensor data and the change in the feature amount, but the specific method is not limited to the method based on the correlation. Depending on the type of sensor mounted on the user terminal device 1, there may be no correlation between the change in the feature amount extracted from the image data and the change in the sensor data. In such a case, a specific method suitable for the characteristics of the sensor or the characteristics of the sensor data output by the sensor can be adopted.

9 and 10 are flowcharts showing an example of the procedure of the specific processing performed by the server device 3 according to the present embodiment. The image data acquisition unit 31a of the processing unit 31 of the server device 3 according to the present embodiment acquires the image data by receiving the image data transmitted from the camera 2 by the communication unit 33 (step S1). .. The image data acquisition unit 31a stores the image data acquired from the camera 2 in the image storage unit 32c of the storage unit 32 (step S2).

Next, the specific unit 31c of the processing unit 31 performs a process of extracting an area in which a person is photographed from the image data acquired by the image data acquisition unit 31a (step S3). The specific unit 31c calculates a feature amount indicating the characteristics of a person photographed in this region from the extracted image region (step S4). The specific unit 31c stores the calculated feature amount in the storage unit 32, RAM, or the like (step S5).

Further, the sensor data acquisition unit 31b of the processing unit 31 acquires the sensor data by receiving the sensor data transmitted from the user terminal device 1 by the communication unit 33 (step S6). The sensor data acquisition unit 31b stores the sensor data acquired from the user terminal device 1 in the storage unit 32, RAM, or the like (step S7).

The processing unit 31 determines whether or not a sufficient amount of information has been accumulated to identify the correspondence between the user and the user terminal device 1 by the processing of steps S1 to S7 (step S8). When sufficient information is not accumulated (S8: NO), the processing unit 31 returns the processing to step S1 and repeatedly acquires the image data and the sensor data.

When sufficient information is accumulated (S8: YES), the specific unit 31c of the processing unit 31 targets one person captured in the image data of the camera 2, and the time series accumulated for this target person. A plurality of characteristic quantities are acquired from the storage unit 32 or the RAM (step S9). The specific unit 31c acquires the timing at which the peak of the change in the feature amount over time occurs based on the acquired feature amount (step S10).

Similarly, the specific unit 31c uses one user terminal device 1 as a target terminal, and acquires a plurality of time-series sensor data accumulated for the target terminal from the storage unit 32 or the RAM (step S11). The specific unit 31c acquires the timing at which the peak of the sensor data in time series occurs based on the acquired sensor data (step S12).

The specific unit 31c compares the timing of the peak of the feature amount acquired in step S10 with the timing of the peak of the sensor data acquired in step S12, and determines whether or not both timings match (step S13). ). It should be noted that a predetermined error or the like can be tolerated in matching the timings determined in step S13. When both timings match (S13: YES), the specific unit 31c identifies that the target person and the target user terminal device 1 correspond to each other (step S14), and stores the specific result. It is stored in the specific information DB 32b of the unit 32 (step S15), and the process proceeds to step S18.

When both timings do not match (S13: NO), the specific unit 31c determines whether or not the comparison of peak timings has been completed for all the user terminal devices 1 in which the sensor data has been accumulated (step S16). When the comparison has not been completed for all the user terminal devices 1 (S16: NO), the specific unit 31c returns the process to step S11 and repeatedly compares the peak timings with another user terminal device 1 as the target terminal. ..

When the comparison is completed for all the user terminal devices 1 (S16: YES), the specific unit 31c determines that the corresponding user terminal device 1 does not exist for the target person (step S17).

The identification unit 31c determines whether or not the identification process corresponding to the user terminal device 1 has been completed for all the persons photographed by the image data of the camera 2 (step S18). When the processing has not been completed for all the persons (S18: NO), the specific unit 31c returns the processing to step S9 and performs the same processing for another person photographed in the image data. When the processing is completed for all the persons (S18: YES), the specifying unit 31c ends the processing for specifying the correspondence between the user and the user terminal device 1.

<Other processing>
(1) Combined use of face recognition processing 1
The identification process performed by the object identification system according to the present embodiment may be used in combination with the existing face recognition process. For example, the server device 3 may further perform face recognition processing on the user who has specified the correspondence between the image data of the camera 2 and the user terminal device 1. The server device 3 stores in a database information for face authentication that collects a face image of a user to be authenticated or a feature amount thereof and the like. The server device 3 acquires information for face authentication corresponding to the user who owns the user terminal device 1 specified by the above-mentioned specific process from the database, and extracts from the user's face image taken by the camera 2. Make a comparison with the quantity. As a result, the server device 3 can determine whether or not the user photographed by the camera 2 is a user whose face authentication information is registered in advance. Since the face recognition process is an existing technique, detailed description thereof will be omitted. In addition, the object identification system may be used in combination with biometric authentication other than face authentication.

By specifying in advance the correspondence between the image data and the user terminal device 1 based on the image data of the camera 2 and the sensor data of the user terminal device 1, the server device 3 can be used for a large number of face recognitions registered in advance in the database. It is sufficient to acquire only the information about the user who owns the specified user terminal device 1 from the information in the above and perform face recognition. Therefore, the processing load and processing time of the face authentication processing by the server device 3 can be reduced. In addition, spoofing using another person's user terminal device 1 can be detected.

FIG. 11 is a flowchart showing an example of the procedure of the authentication process performed by the server device 3 according to the present embodiment. The processing unit 31 of the server device 3 according to the present embodiment is based on the image data acquired from the camera 2 and the sensor data acquired from the user terminal device 1, the person photographed in the image data and the user terminal device 1. A specific process for specifying the correspondence with is performed (step S31). Since the details of the process performed in step S31 are shown in the flowcharts of FIGS. 9 and 10, the details of the process are not shown in this figure.

Based on the result of the identification process in step S31, the processing unit 31 determines whether or not the correspondence between the person captured in the image data and the user terminal device 1 can be specified (step S32). If the correspondence cannot be specified (S32: NO), the processing unit 31 determines that the person photographed in the image data is a suspicious person (step S38), and ends the process.

When the correspondence can be specified (S32: YES), the processing unit 31 acquires the face authentication information registered for the specified user from the database (step S33). The server device 3 stores identification information such as a user ID or terminal ID and face recognition information such as a face image or a facial feature amount in a database for a user to be face-authenticated. .. The processing unit 31 extracts an area corresponding to the user's face as a face image from the image data captured by the camera 2 (step S34).

The processing unit 31 performs face authentication processing using the face authentication information acquired in step S33 and the user's face image extracted in step S34 (step S35). The processing unit 31 determines whether or not the result of the face recognition processing is the authentication success (step S36). The authentication success is when the user's face extracted from the image data matches the user's face indicated by the face authentication information registered in advance.

If the result of the authentication process is successful authentication (S36: YES), the processing unit 31 determines that the specified person is a legitimate user (step S37), and ends the process. If the authentication fails (S36: NO), it is determined that the person photographed in the image data is a suspicious person (step S38), and the process ends.

(2) Combined use of face recognition processing 2
It is difficult to specify the correspondence between the image data of the camera 2 and the image data of the user based on the sensor data of the user terminal device 1 and the user terminal device 1, for example, when the user is in a stationary state. Therefore, when the server device 3 determines that the person included in the image data of the camera 2 is in a stationary state or has a small displacement and it is difficult to perform specific processing using the sensor data, the server device 3 performs face recognition processing on this person. You may identify the person photographed by the camera 2 by performing.

FIG. 12 is a flowchart showing an example of a procedure of specific processing using face recognition performed by the server device 3 according to the present embodiment. The processing unit 31 of the server device 3 according to the present embodiment combines the person photographed in the image data and the user terminal device 1 based on the image data acquired from the camera 2 and the sensor data acquired from the user terminal device 1. It is determined whether or not it is difficult to specify the correspondence (step S51). The processing unit 31 can determine that it is difficult to specify, for example, when the change in the feature amount of the acquired image data or the change in the sensor data is small.

When the identification is not difficult (S51: NO), the processing unit 31 performs the normal identification processing shown in FIGS. 9 and 10 (step S52), and ends the processing.

When it is difficult to identify (S51: YES), the processing unit 31 extracts a person's face image from the image data acquired from the camera 2 (step S53). Further, the processing unit 31 acquires one face authentication information registered in advance in the database (step S54). Using the acquired face recognition information, the processing unit 31 performs face recognition processing on the person photographed by the image data acquired from the camera 2 (step S55).

The processing unit 31 determines whether or not the result of the face recognition processing is the authentication success (step S56). If the authentication is successful (S56: YES), the processing unit 31 identifies the person photographed in the image data as the user associated with the face recognition information for which the authentication was successful (step S57), and ends the process. ..

If the authentication fails (S56: NO), the processing unit 31 determines whether or not the face recognition processing has been completed for all the face recognition information registered in advance in the database (step S58). When the face recognition processing has not been completed for all the face recognition information (S58: NO), the processing unit 31 returns the processing to step S54 and performs the face recognition processing using another face recognition information.

When the face recognition process is completed for all the face recognition information (S58: YES), the processing unit 31 determines that the person photographed in the image data is a suspicious person (step S59), and ends the process.

(3) Tracking a person by a plurality of cameras For example, a plurality of cameras 2 may be installed at predetermined intervals along an outdoor road or along a corridor of a building. The server device 3 individually performs the above-mentioned specific processing on the image data obtained from the plurality of cameras 2 to identify the user photographed by each camera. Based on these individual specific results, the server device 3 can track, for example, the movement of a specific user from the shooting range of the first camera 2a to the shooting range of the second camera 2b. For example, the server device 3 stores and stores information such as the camera ID and the specific time of the specified camera 2 in a database for a specific user.

Further, the server device 3 tracks the movement of the user from the image data of the plurality of cameras 2, so that even if the moving user becomes stationary and it becomes difficult to identify the user based on the sensor data. It is possible to estimate the user photographed by the camera 2 based on the movement history of the user up to.

<Summary>
The server device 3 according to the present embodiment acquires image data of a user who is a specific object taken by the camera 2, and also acquires sensor data which is a detection result of a sensor of the user terminal device 1 possessed by the user. To do. Based on the plurality of image data acquired in time series, the server device 3 acquires a change in the feature amount indicating displacement due to the movement or activity of the user captured in the image data. The server device 3 identifies the correspondence between the user photographed by the camera 2 and the user terminal device 1 based on the time-series change of the feature amount and the time-series change of the sensor data.

As a result, in the object identification system according to the present embodiment, the user photographed by the camera 2 and the user, without the server device 3 storing the face image or the feature amount of the person to be specified in advance, and the user. It is possible to specify the correspondence with the user terminal device 1 possessed by the user.

<Embodiment 2>
The object identification system according to the second embodiment performs a process of specifying the correspondence between the user and the user terminal device 1 by using artificial intelligence. FIG. 13 is a block diagram showing a configuration of the server device 3 according to the second embodiment. The server device 3 according to the second embodiment is learned in advance to specify the correspondence between the user captured in the image data and the user terminal device 1 based on the image data of the camera 2 and the sensor data of the user terminal device 1. The learning model 200 is stored in the storage unit 32.

FIG. 14 is a schematic diagram showing a configuration example of a learning model included in the server device 3 according to the second embodiment. The learning model 200 included in the server device 3 is for realizing the function of the artificial intelligence of the server device 3, and is a learning model in which learning processing such as machine learning, deep learning, or reinforcement learning is performed in advance. The learning model performs a predetermined operation on the input value and outputs the operation result, and the storage unit 32 stores data such as the coefficient and the threshold of the function that defines this operation as the learning model 200. ..

The learning model 200 according to the second embodiment has the structure of a neural network in which a plurality of neurons are interconnected. Since it is an existing technique, detailed description thereof will be omitted, but a neuron is an element that performs an operation on a plurality of inputs and outputs one value as an operation result. Neurons have information such as weighting coefficients and thresholds used in calculations. The neural network learning model 200 aggregates the input layer that accepts the input of one or more data, the intermediate layer that performs arithmetic processing on the data received by the input layer, and the arithmetic results of the intermediate layer. Alternatively, it is provided with an output layer that outputs a plurality of values. Learning processes such as deep learning and reinforcement learning are processes in which appropriate values are set for the coefficients and thresholds of each neuron constituting the neural network by using a large number of learning data given in advance. The learning model 200 according to the second embodiment is a trained model in which deep learning using teacher data is performed on the training model of the neural network, and is, for example, a gradient descent method, a stochastic gradient descent method, or an error backpropagation. Learning is done by methods such as law.

In the learning model 200 according to the second embodiment, time-series information of the image data captured by the camera 2 and the sensor data which is the detection result of the sensor of the user terminal device 1 is input to the input layer. However, the image data is obtained by extracting the area in which the person was photographed, and is the data of the image area for one person. Further, the image data may be a plurality of still images arranged in chronological order, or may be moving image data. The sensor data is sensor data for one user terminal device 1. As the image data and the sensor data, for example, those for a time of several seconds to several tens of seconds are input in chronological order. The learning model 200 outputs probabilistic information indicating that the user terminal device 1 possessed by the person captured in the input image data may be the user terminal device 1 that has detected the input sensor data as a specific result. .. For the learning model 200 in which time-series information is input, for example, an RNN (Recurrent Neural Network) model can be adopted. Further, as the learning model 200, a CNN (Convolution Neural Network) model including a convolutional layer, a pooling layer, and the like for extracting features from the input image data can be adopted.

However, the input information and the output information of the learning model 200 are not limited to the above. For example, the learning model 200 may accept the image data taken by the camera 2 as input information instead of the image data extracted from the area where the person is photographed. In this case, the learning model 200 may accept the person from the input image data. Performs the process of extracting the area of. Further, instead of inputting the image data into the learning model 200, the feature amount calculated from the image data may be input. Further, the learning model 200 may accept sensor data of a plurality of user terminal devices 1 as input information. In this case, the learning model 200 may output probability information for each user terminal device 1, or may output which user terminal device 1 has the highest probability as a specific result.

The server device 3 according to the second embodiment is a learning model in which the image data obtained by extracting the shooting area of a person from the image data acquired from the camera 2 and the sensor data acquired from the user terminal device 1 are stored in the storage unit 32. Enter in 200. The learning model 200 has a probability that a person photographed by the input image data possesses the user terminal device 1 related to the input sensor data in response to the input of the time-series image data and the sensor data. Output the information as a specific result. The server device 3 acquires a specific result output by the learning model 200.

When the server device 3 acquires sensor data from a plurality of user terminal devices 1, the server device 3 performs specific processing using the learning model 200 for each user terminal device 1, and the user terminal having the highest probability of being output by the learning model 200. The device 1 is specified as a user terminal device 1 possessed by a person with image data. Further, when a plurality of persons are photographed in the image data acquired from the camera 2, the server device 3 performs the same processing for each person and identifies the correspondence between each person and the user terminal device 1. The server device 3 stores and stores the specific result in the specific information DB 32b of the storage unit 32.

Further, the learning model 200 according to the second embodiment is a trained model in which deep learning has been performed in advance by the server device 3 or another device. For example, the server device 3 is a teacher in which image data taken by a person, sensor data output by a sensor of the user terminal device 1, and a specific result indicating whether or not both data are in a corresponding relationship are associated with each other. Deep learning is performed using the data. The teacher data used in the learning process may be performed by the designer of this system or the like, or may be performed by a device such as the server device 3. Further, the server device 3 may use the teacher data created in advance, or may use the teacher data created by itself based on the information accumulated by the server device 3. At least in the first learning process, the teacher data created in advance is used. For example, teacher data can be created based on information collected by a security system using a conventional surveillance camera or the like, or information obtained by a demonstration experiment or the like conducted in this system or a similar system.

In the second and subsequent learning processes (re-learning processes), teacher data may be generated based on the information collected and accumulated by the server device 3. The server device 3 according to the second embodiment stores and stores information related to the specific result in the specific information DB 32b of the storage unit 32. The server device 3 may generate teacher data for learning based on the information stored in the specific information DB 32b. However, in this case, it is preferable that the specific information DB 32b stores the sensor data used for the specific data in addition to the information shown in FIG. The server device 3 can relearn the learning model 200 using the teacher data generated based on the accumulated information every time a predetermined period (1 day, 1 week, 1 month, etc.) elapses.

15 and 16 are flowcharts showing an example of the procedure of the specific processing performed by the server device 3 according to the second embodiment. The image data acquisition unit 31a of the processing unit 31 of the server device 3 according to the second embodiment acquires the image data by receiving the image data transmitted from the camera 2 by the communication unit 33 (step S71). .. The image data acquisition unit 31a stores the image data acquired from the camera 2 in the image storage unit 32c of the storage unit 32 (step S72).

Next, the specific unit 31c of the processing unit 31 performs a process of extracting an area in which a person is photographed from the image data acquired by the image data acquisition unit 31a (step S73). The specific unit 31c stores the image data of the extracted area in the storage unit 32, RAM, or the like (step S74).

Further, the sensor data acquisition unit 31b of the processing unit 31 acquires the sensor data by receiving the sensor data transmitted from the user terminal device 1 by the communication unit 33 (step S75). The sensor data acquisition unit 31b stores the sensor data acquired from the user terminal device 1 in the storage unit 32, RAM, or the like (step S76).

The processing unit 31 determines whether or not a sufficient amount of information has been accumulated to identify the correspondence between the user and the user terminal device 1 by the processing of steps S71 to S76 (step S77). When sufficient information is not accumulated (S77: NO), the processing unit 31 returns the processing to step S71 and repeatedly acquires the image data and the sensor data.

When sufficient information is accumulated (S77: YES), the specific unit 31c of the processing unit 31 targets one person photographed in the image data of the camera 2, and the time series accumulated for this target person. Image data of a specific area is acquired from the storage unit 32 or RAM (step S78). Similarly, the specific unit 31c uses one user terminal device 1 as a target terminal, and acquires a plurality of time-series sensor data accumulated for the target terminal from the storage unit 32 or the RAM (step S79). Information such as the acquired date and time is attached to the image data and the sensor data, and the specific unit 31c acquires a set of the image data and the sensor data having the same date and time in steps S78 and S79.

The specific unit 31c inputs the input information including the image data of the area of the target person acquired in step S78 and the sensor data acquired in step S79 into the learning model 200 stored in the storage unit 32 (step). S80). The specific unit 31c acquires the output information of the learning model 200 for this input (step S81).

The identification unit 31c determines whether or not the learning model 200 has specified all the user terminal devices 1 in which the sensor data has been accumulated (step S82). When the identification of all the user terminal devices 1 by the learning model 200 has not been completed (S82: NO), the identification unit 31c returns the process to step S79 and performs the same process with another user terminal device 1 as the target terminal. Repeat.

When the identification using the learning model 200 is completed for all the user terminal devices 1 (S82: YES), the identification unit 31c compares the specific results for each user terminal device 1, and the probability output by the learning model 200 is high. The highest user terminal device 1 is identified as the user terminal device 1 possessed by the target person (step S83). The specific unit 31c stores the specific result in the specific information DB 32b of the storage unit 32 (step S84).

The identification unit 31c determines whether or not the identification process corresponding to the user terminal device 1 has been completed for all the persons photographed by the image data of the camera 2 (step S85). When the processing has not been completed for all the persons (S85: NO), the specific unit 31c returns the processing to step S78 and performs the same processing for another person photographed in the image data. When the processing is completed for all the persons (S85: YES), the specifying unit 31c ends the processing for specifying the correspondence between the user and the user terminal device 1.

In the object identification system according to the second embodiment of the above configuration, the server device 3 includes a learning model 200 learned in advance. The server device 3 inputs the input information based on the image data acquired from the camera 2 and the sensor data acquired from the user terminal device 1 into the learning model 200, and acquires the output information of the learning model 200 in the camera 2. It is possible to specify the correspondence between the photographed user and the user terminal device 1 possessed by this user.

(Modification example)
FIG. 17 is a schematic diagram for explaining an outline of the identification process performed by the object identification system according to the modified example of the second embodiment. Based on the sensor data acquired from the user terminal device 1, the server device 3 of the object identification system according to the modified example is possessed, gripped, attached, or attached to any place on the user's body. It is provided with a position estimation unit 201 for estimating whether or not the system is used, and a plurality of learning models 200a, 200b, 200c ... For identifying the correspondence between the user and the user terminal device 1 from image data and sensor data.

The position estimation unit 201 performs a process of estimating at which position the user terminal device 1 equipped with the sensor is attached, for example, the arm, chest, or leg of the human body. The position estimation unit 201 takes time-series sensor data as input and outputs information regarding the estimated mounting position. The mounting position may be estimated by the position estimation unit 201 by any method.

For example, the position estimation unit 201 can be realized by using a learning model in which deep learning or the like has been performed in advance. The learning model for position estimation is learned in advance using teacher data using sensor data as input information and mounting position as output information. The server device 3 stores the learning model for position estimation in the storage unit 32, inputs the sensor data acquired from the user terminal device 1 to the learning model for position estimation, and acquires the output of the learning model. The mounting position of the terminal device 1 can be estimated.

Further, for example, the position estimation unit 201 may estimate the mounting position by comparing the sensor data acquired from the user terminal device 1 with a predetermined determination standard. The position estimation unit 201 can perform position estimation by comparing the maximum value, the minimum value, the cycle of change, etc. of the sensor data with the threshold value set as the determination standard. Further, the position estimation unit 201 may convert the time-series sensor data into data in the frequency domain by Fourier transform or the like, and estimate the position based on the frequency characteristics obtained by the conversion.

The server device 3 includes a plurality of learning models 200a, 200b, 200c ... According to the mounting position of the user terminal device 1. In this example, the server device 3 includes a learning model 200a for the arm, a learning model 200b for the chest, a learning model 200c for the legs, and so on, depending on the mounting position. The server device 3 selects any one of the plurality of learning models 200a, 200b, 200c ... Based on the estimation result of the position estimation unit 201, and image data is applied to the selected one learning model 200a, 200b, 200c ... And the sensor data is input to identify the correspondence between the user and the user terminal device 1.

The learning models 200a, 200b, 200c ... Are subjected to deep learning or the like in advance using the sensor data detected by the sensors of the user terminal device 1 mounted at each mounting position. As a result, each of the learning models 200a, 200b, 200c ... can be specified with higher accuracy by specializing in the mounting position of the user terminal device 1.

FIG. 18 is a flowchart showing an example of a procedure of specific processing performed by the server device 3 according to the modified example of the second embodiment. The position estimation unit 201 of the server device 3 according to the modified example of the second embodiment acquires the sensor data used for the specific process (step S91). The position estimation unit 201 inputs the acquired sensor data to, for example, a learning model for position estimation and acquires the output thereof to perform a process of estimating the mounting position of the user terminal device 1 that is the source of the sensor data ( Step S92).

The specific unit 31c of the processing unit 31 of the server device 3 is one learning model 200 corresponding to the estimated mounting position of the position estimation unit 201 from among the plurality of learning models 200a, 200b, 200c ... Stored in the storage unit 32. Is read (step S93). The identification unit 31c performs the identification process shown in FIGS. 15 and 16 using one learning model 200 read out in step S93 (step S94), and ends the process.

Since the other configurations of the object identification system according to the second embodiment are the same as those of the object identification system according to the first embodiment, the same reference numerals are given to the same parts, and detailed description thereof will be omitted.

<Embodiment 3>
19 and 20 are schematic views for explaining the outline of the specific processing performed by the object identification system according to the third embodiment. In the object identification system according to the third embodiment, the beacon device 5 is installed in the vicinity of the camera 2. The object identification system according to the third embodiment estimates the user terminal device 1 existing within the shooting range of the camera 2 based on the information transmitted by the beacon device 5. In the present embodiment, when it is necessary to distinguish between the plurality of beacon devices 5, they will be described with different reference numerals as in the beacon devices 5A and 5B.

The beacon device 5 transmits a wireless signal (beacon signal) at regular intervals, for example, based on a Bluetooth (registered trademark) communication standard. However, the beacon device 5 may transmit a wireless signal based on a communication standard other than Bluetooth. It should be noted that the camera 2 may have a function of transmitting a beacon signal instead of providing the beacon device 5 separately.

The reachable range of the beacon signal transmitted by the beacon device 5 is, for example, about several meters to several tens of meters. In the third embodiment, it is preferable that the reach range of the beacon signal transmitted by the beacon device 5 is as wide as the shooting range of the camera 2 or wider than the shooting range of the camera 2. The beacon signal transmitted by the beacon device 5 is received by the user terminal device 1 existing within the reach of the beacon signal.

In the example shown in FIG. 19, the user terminal devices 1B, 1C, and 1D receive the beacon signal transmitted by the beacon device 5, but the user terminal device 1A does not receive the beacon signal. Further, in the example shown in FIG. 20, the user terminal device 1A receives the beacon signal transmitted by the beacon device 5A, and the user terminal devices 1B and 1C receive the beacon signal transmitted by the beacon device 5B.

Further, the beacon signal transmitted by the beacon device 5 includes a beacon ID uniquely assigned to the beacon device 5. The user terminal device 1 that has received the beacon signal acquires the beacon ID included in the beacon signal. When the user terminal device 1 transmits the sensor data to the server device 3, the user terminal device 1 attaches a beacon signal to the sensor data and transmits the signal data.

The server device 3 according to the third embodiment receives the sensor data transmitted from the user terminal device 1 and acquires the beacon ID attached to the received sensor data. The server device 3 stores, for example, a table in which a camera ID uniquely assigned to the camera 2 and a beacon ID of the beacon device 5 installed in the vicinity of the camera 2 are associated with each other in the storage unit 32. ing. The server device 3 is associated with a table when performing a process of identifying the correspondence between the user and the user terminal device 1 based on the image data acquired from the camera 2 and the sensor data acquired from the user terminal device 1. Specific processing is performed on the image data and sensor data related to the camera ID and the beacon ID as the data to be processed. As a result, the server device 3 can estimate which user terminal device 1 exists in the shooting range of the camera 2 based on the beacon ID, and limits the user terminal device 1 to be the target of the specific processing. be able to.

As a result, in the object identification system according to the third embodiment, the server device 3 performs identification processing based on the plurality of image data acquired from the plurality of cameras 2 and the plurality of sensor data acquired from the plurality of user terminal devices 1. Since the data to be handled can be limited even in the case of performing the above, the processing load of the server device 3 can be reduced.

In the object identification system according to the third embodiment, the user terminal device 1 receives the beacon signal transmitted by the beacon device 5. However, the object identification system may be configured such that the beacon device 5 receives the beacon signal transmitted by the user terminal device 1 as shown in the following modification.

(Modification example 1)
FIG. 21 is a schematic diagram for explaining an outline of data transmission / reception of the object identification system according to the first modification of the third embodiment. In the object identification system according to the first modification of the third embodiment, the image data captured by the camera 2 is transmitted to the server device 3 via a wired or wireless network.

The user terminal device 1 according to the first modification of the third embodiment does not transmit the sensor data to the server device 3. The user terminal device 1 according to the first modification transmits a wireless signal (beacon signal) at a fixed cycle, for example, based on a Bluetooth communication standard. The beacon signal transmitted by the user terminal device 1 includes identification information such as a user ID or a terminal ID, and sensor data which is a detection result of the sensor 13. The reach of the beacon signal transmitted by the user terminal device 1 is, for example, about several meters to several tens of meters. This beacon signal is received by the beacon device 5 existing within the reachable range.

The beacon device 5 according to the first modification of the third embodiment is a device that receives a beacon signal instead of transmitting the beacon signal. The beacon device 5 according to the first modification can communicate with the server device 3 by wired or wireless communication. The beacon device 5 is installed in association with a predetermined camera 2. It is preferable that the range in which the beacon device 5 receives the beacon signal is as wide as the shooting range of the corresponding camera 2 or wider than the shooting range of the camera 2.

When the beacon device 5 according to the first modification receives the beacon signal transmitted by the user terminal device 1, it acquires identification information such as a user ID or terminal ID included in the beacon signal and sensor data. The beacon device 5 transmits the acquired identification information, sensor data, and the like with its own beacon ID to the server device 3.

The server device 3 according to the first modification of the third embodiment is captured by the image data of the camera 2 by using the image data acquired from the camera 2 and the beacon ID and the sensor data acquired from the beacon device 5. A process for identifying the correspondence between the person and the user terminal device 1 is performed.

(Modification 2)
The method of estimating the user terminal device 1 existing in the shooting range of the camera 2 is not limited to the method of using the beacon signal. In the object identification system according to the second modification of the third embodiment, the user terminal device 1 transmits its own position information based on the GPS signal received by the GPS receiving unit 17 to the server device 3 together with the sensor data.

For example, the server device 3 includes a database that stores location information of a place where the camera 2 is installed. The server device 3 can estimate the user terminal device 1 existing within the shooting range of the camera 2 by comparing the position information of the installation location of the camera 2 with the position information acquired from the user terminal device 1. ..

(Modification 3)
In the object identification system according to the second modification of the third embodiment, for example, the object identification system is installed at the venue or the building in cooperation with the entrance / exit management system that manages the entrance / exit to the venue or the specific building. The user terminal device 1 existing in the shooting range of one or more cameras 2 is estimated.

The entrance / exit management system acquires the terminal ID of the user terminal device 1 of the user who enters / exits the venue or building at the entrance / exit of the venue or building, and stores the entry / exit history. For example, the wireless communication device of the entrance / exit management system installed at the entrance / exit performs wireless communication with the user terminal device 1 and acquires the terminal ID of the user terminal device 1 to enter / exit. However, the entrance / exit management system may acquire the user ID of the user entering / exiting, and may acquire the terminal ID associated with the user ID by referring to a database or the like.

In the object identification system according to the second modification of the third embodiment, the server device 3 acquires the terminal ID of the user terminal device 1 existing inside the venue, the building, or the like from the entrance / exit management system. The server device 3 is presumed that the user terminal device 1 existing within the shooting range of the camera 2 installed in the venue or building is the user terminal device 1 to which the terminal ID acquired from the entrance / exit management system is attached. can do.

In the object identification system according to the third embodiment and the modified examples 1 to 3 thereof, it is not necessary to estimate so that the camera 2 and the user terminal device 1 have a one-to-one relationship. For example, the server device 3 may estimate the user terminal device 1 existing in any of the imaging ranges of several cameras 2.

Since the other configurations of the object identification system according to the third embodiment are the same as those of the object identification system according to the first embodiment, the same reference numerals are given to the same parts, and detailed description thereof will be omitted.

The embodiments disclosed this time should be considered as exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1, 1A to 1D User terminal device 2, 2A, 2B Camera 3 Server device 5, 5A, 5B Beacon device 11 Processing unit 11a Sensor data transmission unit 12 Storage unit 12a Program 12b Terminal ID
13 Sensor 14 Communication unit 15 Display unit 16 Operation unit 17 GPS receiver 31 Processing unit 31a Image data acquisition unit 31b Sensor data acquisition unit 31c Specific unit 32 Storage unit 32a Server program 32b Specific information DB
33 Communication unit 99 Recording medium 200 Learning model 201 Position estimation unit

Claims (14)

On the computer
Acquire the image data taken by the camera and
Obtain sensor data from the sensor of the object and
Estimate the mounting position of the sensor with respect to the object,
A computer program that executes a process of identifying the correspondence between the image data obtained by photographing the object and the sensor based on the estimated mounting position and the acquired image data and the sensor data. Acquire image data of multiple objects taken with a camera and
Acquire multiple sensor data from multiple sensors of multiple objects,
Based on the acquired image data and the plurality of sensor data, the correspondence between the image data obtained by photographing the plurality of objects and the plurality of sensors is specified.
The computer program according to claim 1. The image data taken by the camera in time series is acquired, and the image data is acquired.
The sensor data in time series is acquired from the sensor, and the sensor data is acquired.
Based on the acquired time-series image data and the sensor data, the correspondence between the image data obtained by photographing the object and the sensor is specified.
The computer program according to claim 1 or 2. The feature amount of the object included in the image data is calculated.
Obtain the timing at which the peak of the change in the feature amount occurs, and
Acquire the timing at which the peak of change in the sensor data occurs, and
Based on the timing related to the feature amount and the timing related to the sensor data, the correspondence between the image data obtained by photographing the object and the sensor is specified.
The computer program according to claim 3. Using a trained model that has been trained to accept the image data and the sensor data as inputs and output the image data obtained by photographing the object and the specific result of the correspondence of the sensor.
The acquired image data and the sensor data are input to the trained model.
Acquire the specific result output by the trained model,
The computer program according to any one of claims 1 to 3. The acquired image data and the correspondence between the object included in the image data and the sensor are stored in a database in association with each other.
The computer program according to any one of claims 1 to 5 . The sensor existing within the shooting range of the camera is estimated, and the sensor is estimated.
Identify the correspondence between the estimated sensor and the image data obtained by capturing the object within the imaging range of the camera.
The computer program according to any one of claims 1 to 6 . The image data is acquired from a plurality of cameras having different shooting ranges, and the image data is acquired.
Based on the acquired plurality of image data, the movement of the object captured by the image data that specifies the correspondence with the sensor is determined.
The computer program according to any one of claims 1 to 7 . When the correspondence between the image data obtained by photographing the object and the sensor cannot be specified based on the sensor data, the object was photographed based on the image data and the information registered in advance for the object. Identify the correspondence between the image data and the sensor,
The computer program according to any one of claims 1 to 8 . Acquire the image data taken by the camera and
Obtain sensor data from the sensor of the object and
Estimate the mounting position of the sensor with respect to the object,
An object identification method for specifying the correspondence between the image data obtained by photographing the object and the sensor based on the estimated mounting position and the acquired image data and the sensor data. It is determined whether or not the object has a predetermined sensor according to the image data obtained by photographing the object and the specific result of the correspondence of the sensor.
Authentication of the object is performed based on the determination result.
The object identification method according to claim 10 . The object is a living thing
Biometric authentication of the object is performed based on the image data obtained by photographing the object and the specific result of the correspondence of the sensor.
The object identification method according to claim 10 or 11 . An image data acquisition unit that acquires image data taken with a camera,
A sensor data acquisition unit that acquires sensor data from the sensor of the object,
An estimation unit that estimates the mounting position of the sensor with respect to the object,
Based on the mounting position estimated by the estimation unit, the image data acquired by the image data acquisition unit, and the sensor data acquired by the sensor data acquisition unit, the correspondence between the image data obtained by photographing the object and the sensor is determined. An object identification device including a specific part to be specified. With the camera
Sensors provided on the object and
Image data acquisition unit, sensor data acquisition unit that acquires sensor data from the sensor, estimator for estimating the mounting position of the sensor relative to the object, and the estimation unit estimates that acquires image data captured by the camera Based on the mounting position, the image data acquired by the image data acquisition unit, and the sensor data acquired by the sensor data acquisition unit, the image data obtained by photographing the object and the specific unit that specifies the correspondence of the sensor are determined. An object identification system including an object identification device to have.

Images