JP2020025258A - Communication terminal, communication system, communication method, and program - Google Patents

Abstract

To solve the problem in which: in a conventional communication terminal, the communication terminal transmits photographic image data and a recognition request to a server (another device) every time a change occurs in an object, and when the same object is photographed by the communication terminal, unnecessary transmission of the photographic data and recognition request is therefore performed, and in some cases, unnecessary communication fee is generated.SOLUTION: A communication terminal (2) transmits image data of a predetermined portion of an object to cause another device (7) to perform collation of an image, and comprises transmission means that transmits, to the another device, image data of a predetermined portion of a first object at a first time. When the distance between a first position indicating the position of the first object at the first time and a second position indicating the position of a second object at a second time after a predetermined time from the first time is equal to or less than a predetermined distance or less than the predetermined distance, the transmission means does not transmit image data of a predetermined portion of the second object (S218).SELECTED DRAWING: Figure 28

Description

  The present disclosure relates to a communication terminal, a communication system, a communication method, and a program.

  2. Description of the Related Art In recent years, as one of profit models of Web API (Application Programming Interface), there is a “billing model” for collecting a fee when using a Web API. Although it is necessary to set a low unit price, if the number of uses increases, a certain profit can be expected, and thus it is already known as one of the mainstream models.

  In addition, conventionally, using a Web API, a communication terminal equipped with a camera shoots an object such as a person, and when there is a change due to movement of the object or the like, transmits the shot image data to a server and performs object authentication. There is disclosed a technology in which a communication terminal makes a request and a communication terminal receives a recognition result from a server (see Patent Document 1).

  However, in the related art, the communication terminal transmits the captured image data and the recognition request to the server every time a change occurs in the imaging target. Therefore, when the same object is captured by the communication terminal, the captured image data and There has been a problem that the transmission of the recognition request is wasted, and in some cases, an unnecessary communication fee is generated.

  The invention according to claim 1 is a communication terminal that transmits image data of a predetermined portion of an object to cause another device to perform image collation, wherein the communication terminal transmits the image data of a predetermined portion of the first object at a first time point. Transmitting means for transmitting image data to the other device, a first position indicating a position of the first object at the first time, and a second time after a predetermined time from the first time If the distance between the second object and the second position indicating the position of the second object is less than or equal to a predetermined value or less than a predetermined value, the transmission unit does not transmit the image data of the predetermined portion of the second object. A communication terminal characterized in that:

  As described above, according to the present invention, by performing a recognition request or the like in accordance with the situation of an imaging target, there is an effect that time and money involved in processing can be reduced.

It is a schematic diagram of a communication system concerning this embodiment. It is a hardware block diagram of a real-time data processing terminal. It is a hardware block diagram of an imaging unit. 4A is a hemispherical image (front side) photographed by the imaging unit 40b, FIG. 4B is a hemispherical image (rear side) photographed by the imaging unit 40b, and FIG. 4C is a equirectangular projection. FIG. 4 is a diagram showing a displayed image. It is a hardware block diagram of a near terminal data processing apparatus or a distributed data processing terminal. It is a hardware block diagram of a centralized data processing server. FIG. 3 is a software configuration diagram of a real-time data processing terminal and a near terminal data processing device in the image acquisition terminal. It is a functional block diagram of a communication system. It is a functional block diagram of a communication system. (A) is a conceptual diagram of an image sensor information management table, and (b) is a conceptual diagram of a cycle value management table. (A) is a conceptual diagram of an image acquisition program management table, (b) is a conceptual diagram of a synthesis processing program management table, (c) is a conceptual diagram of a distortion correction program management table, and (d) is a conceptual diagram of a service program management table. It is. It is a conceptual diagram of a collation data management table. It is a conceptual diagram of a session management table. It is a conceptual diagram of an object information management table. It is a conceptual diagram of an object displacement management table. It is a conceptual diagram of a terminal ID. It is a conceptual diagram of an authentication server management table. It is a conceptual diagram of an authentication management table. FIG. 7 is a sequence diagram illustrating an authentication process. FIG. 7 is a sequence diagram illustrating an authentication process. It is a figure showing the example of a screen of a distributed data processing terminal. It is a figure showing the example of a screen of a distributed data processing terminal. FIG. 9 is a sequence diagram illustrating a process of a request for starting image recognition. FIG. 9 is a sequence diagram illustrating a preparation process of a real-time process of the real-time data processing terminal. FIG. 9 is a sequence diagram illustrating a program acquisition process. FIG. 4 is a sequence diagram illustrating a process of image recognition. It is the flowchart which showed the object detection processing of the real time processing. It is the flowchart which showed the process of event generation in real time processing. It is the flowchart which showed the process of event generation in real time processing. FIG. 7 is a sequence diagram illustrating a process of performing collation. 9 is a flowchart illustrating a matching process. FIG. 5 is a diagram illustrating a display example of a captured image in a distributed data processing terminal. FIG. 29 is a flowchart illustrating a modification of the process illustrated in FIG. 28 according to the second embodiment. FIG. 9 is a sequence diagram illustrating a process of a display update request. It is a modification of the display example shown in FIG. 32 according to the second embodiment. According to the third embodiment, (a) is a hardware configuration diagram of an imaging unit equipped with a LiDAR mechanism in FIG. 3 (a), and FIG. 36 (b) is equipped with a LiDAR mechanism in FIG. 3 (b). It is a hardware block diagram of an imaging unit. According to the third embodiment, it is a modification of the real-time data processing terminal in the functional block diagram of the communication system. FIG. 29 is a flowchart illustrating a modification of the processing illustrated in FIG. 28 according to the third embodiment. It is a modification of the display example shown in FIG. 32 according to the third embodiment. This is a modification of the display example shown in FIG. 32 according to the third embodiment, and is a display example in a case where the LiDAR function is not used together. This is a modification of the display example shown in FIG. 32 according to the third embodiment, and is a display example in the case where the function of LiDAR is used together.

First Embodiment First, a first embodiment of the present invention will be described.

[Overall overview]
Hereinafter, an overall outline of an embodiment of the present invention will be described with reference to the drawings. This embodiment discloses a communication system that realizes edge computing. That is, the present embodiment discloses an invention that realizes a service provided from the service providing server 8 in the image acquisition terminal 2 by cooperation of the processing in the service providing server 8 and the processing in the image acquisition terminal 2.

<<< Overall Configuration of Embodiment >>>
FIG. 1 is a schematic diagram of a communication system according to the present embodiment. As shown in FIG. 1, the communication system of the present embodiment includes a real-time data processing terminal 3a, a near terminal data processing device 5, a distributed data processing terminal 6, a centralized data processing server 7, a service providing server 8, It is constructed by authentication servers 9a, 9b, 9c. The real-time data processing terminal 3a, the near-terminal data processing device 5, and the distributed data processing terminal 6 constitute a distributed processing system 100.

  The near terminal data processing device 5 is communicably connected to the distributed data processing terminal 6 via the intranet 200. The distributed data processing terminal 6 is communicably connected to the centralized data processing server 7, the service providing server 8, and the authentication servers 9a, 9b, 9c via the Internet 600. The authentication server 9 is a general term for the authentication servers 9a, 9b, 9c.

  Among these, the real-time data processing terminal 3a is a terminal that performs real-time processing (real-time processing) for obtaining data of a captured image. The real-time data processing terminal 3a is detachably connected to an imaging unit 40 having an image sensor for imaging a subject such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. Accordingly, the real-time data processing terminal 3a digitizes the data of the captured image (hereinafter, referred to as “captured image data”) input from the image capturing unit 40, and converts the object (here, every 1/60 second) into an object (here, 1/60 second). Then, the image of the face) is detected. The real-time data processing terminal 3 a transmits data of a partial image (hereinafter, referred to as “partial image data”), which is a region of an object portion in a captured image, to the near-terminal data processing device 5.

  The near-terminal data processing device 5 is arranged closest to the real-time data processing terminal 3a, and is closely connected, for example, in terms of hardware in a one-to-one manner by a data bus or a USB (Universal Serial Bus). The near-terminal data processing apparatus 5 encodes the partial image data received from the real-time data processing terminal 3a into a general-purpose format such as JPEG (Joint Photographic Experts Group), and then performs processing on the face image when collation is performed. The data is transmitted to the distributed data processing terminal 6 via the intranet 200 as collation data. Normally, the real-time data processing terminal 3a and the near-terminal data processing device 5 are connected and integrated. Here, the real-time data processing terminal 3a and the near-terminal data processing device 5 connect the image acquisition terminal 2 to each other. Make up.

  The distributed data processing terminal 6 is a computer used by a user at a position relatively close to the near terminal data processing device 5 and capable of receiving various operations by the user. The distributed data processing terminal 6 registers and stores in advance collation data when face images are collated. The distributed data processing terminal 6 can request the centralized data processing server 7 to collate the collation data with the collated data via the Internet 600. In this case, the distributed data processing terminal 6 also transmits the collated data received from the near terminal data processing device 5 and the collation data registered in advance. Further, the distributed data processing terminal 6 can receive, from the centralized data processing server 7, collation result information indicating a collation result as a response. Further, the distributed data processing terminal 6 can display the comparison result via the graphic interface.

  The centralized data processing server 7 is installed at a position relatively far from the near terminal data processing device 5, and can communicate with the distributed data processing terminal 6 via a communication network such as the Internet 600. When receiving the collation request information indicating the collation request, the collation data, and the data to be collated, the centralized data processing server 7 collates the collation data and the data to be collated, and determines the similarity. Then, the centralized data processing server 7 transmits the collation result information indicating the collation result including the similarity to the distributed data processing terminal 6.

  The service providing server 8 is a server that provides various services to the image acquisition terminal 2.

  The authentication server 9a performs ID authentication in order to determine whether the image acquisition terminal 2 has a legitimate right to receive a service from the service providing server 8. The same applies to the authentication servers 9b and 9c.

<< hardware configuration >>
Next, each hardware configuration of the communication system according to the present embodiment will be described with reference to FIGS.

<Hardware configuration of real-time data processing terminal>
FIG. 2 is a hardware configuration diagram of the real-time data processing terminal. The real-time data processing terminal 3a includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, an acceleration / direction sensor 306, a media I / F 308, and a GPS receiving unit 309.

  Among them, the CPU 301 controls the operation of the whole real-time data processing terminal 3a. The ROM 302 stores a program used for driving the CPU 301. The RAM 303 is used as a work area of the CPU 301. The EEPROM 304 reads or writes various data such as a program for a real-time data processing terminal under the control of the CPU 301. The CMOS sensor 305 captures an image of a subject (mainly a blind spot of the image capturing unit 40) under the control of the CPU 301 to obtain captured image data. The acceleration / azimuth sensor 306 is various sensors such as an electronic magnetic compass for detecting geomagnetism, a gyro compass, and an acceleration sensor. The media I / F 308 controls reading or writing (storage) of data from or to a recording medium 307 such as a flash memory. The GPS receiver 309 receives a GPS signal from a GPS satellite.

  Further, the real-time data processing terminal 3a includes an imaging unit I / F 313, a microphone 314, a speaker 315, a sound input / output I / F 316, a display 317, an external device connection I / F 318, and a touch panel 321.

  Among these, the imaging unit I / F 313 is a circuit that controls the driving of the imaging unit 40 when the external imaging unit 40 is connected. The microphone 314 is a kind of a built-in sound collecting means for inputting sound. The sound input / output I / F 316 is a circuit that processes input / output of a sound signal between the microphone 314 and the speaker 315 under the control of the CPU 301. The display 317 is a type of display means such as a liquid crystal display or an organic EL display that displays an image of a subject and various icons. The external device connection I / F 318 is an interface for connecting various external devices. The touch panel 321 is a type of input means for operating the real-time data processing terminal 3a when the user presses the display 317.

  The real-time data processing terminal 3a includes a bus line 310. The bus line 310 is an address bus, a data bus, and the like for electrically connecting each component such as the CPU 301 shown in FIG.

<Hardware configuration of imaging unit>
FIG. 3 is a hardware configuration diagram of the imaging unit. In particular, FIG. 3A is a hardware configuration diagram of a monocular imaging unit 40 a among the imaging units 40. FIG. 3B is a hardware configuration diagram of a compound-eye imaging unit 40 b among the imaging units 40. Note that the image pickup unit 40 is a general term for a plurality of types of image pickup units (image pickup units 40a, 40b, etc.) having different numbers of image pickup devices.

  As shown in FIG. 3A, the imaging unit 40a is electrically connected to an imaging element 401a such as a CMOS or a CCD, a lens 402a, and an imaging unit I / F 313 of the real-time data processing terminal 3a. Connection I / F 408a. When the imaging unit 40a is connected to the imaging unit I / F 313 of the real-time data processing terminal 3a, the imaging element 401a sends an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408a. , And transmits the captured image data to the imaging unit I / F 313 via the connection I / F 408a.

  Further, as shown in FIG. 3B, the imaging unit 40b includes imaging elements 401b1 and 401b2 such as CMOS and CCD, lenses 402b1 and 402b2, and an imaging unit I / F 313 of the real-time data processing terminal 3a. It has a connection I / F 408b for electrical connection. The lenses 402b1 and 402b2 are, for example, fisheye lenses. When the imaging unit 40a is connected to the imaging unit I / F 313 of the real-time data processing terminal 3a, the imaging device 401a sends an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408a. And transmits a plurality of captured image data to the imaging unit I / F 313 via the connection I / F 408b.

  While the imaging unit 40a shown in FIG. 3A obtains a general planar image, the imaging unit 40b shown in FIG. 3B can obtain a spherical image. Here, an outline of a process until an equirectangular projected image EC is created from an image captured by the imaging unit 40b will be described with reference to FIG. 4A is a hemispherical image (front side) captured by the imaging unit 40b, FIG. 4B is a hemispherical image (rear side) captured by the imaging unit 40b, and FIG. FIG. 2 is a diagram illustrating an image represented by a projection method (hereinafter, referred to as an “equidistant cylinder projected image”).

  As shown in FIG. 4A, the image obtained by the image sensor 401b1 is a hemispherical image (front side) curved by the lens 402b1. Also, as shown in FIG. 4B, the image obtained by the image sensor 403b is a hemispherical image (rear side) curved by the lens 402b2. Then, the hemispherical image (front side) and the 180 degree-reversed hemispherical image (rear side) are combined to form an equirectangular projected image EC as shown in FIG. 4C.

<Hardware configuration of near terminal data processing device and distributed data processing terminal>
FIG. 5 is a hardware configuration diagram of the near terminal data processing device or the distributed data processing terminal. Here, since the near terminal data processing device 5 and the distributed data processing terminal 6 have the same hardware configuration, the near terminal data processing device 5 will be described, and the distributed data processing terminal 6 will be described. Is omitted.

  As shown in FIG. 5, the near terminal data processing device 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an acceleration / direction sensor 506, a media I / F 508, and a GPS receiving unit 509.

  Among these, the CPU 501 controls the operation of the entire near terminal data processing device 5. The ROM 502 stores a program used for driving the CPU 501. The RAM 503 is used as a work area of the CPU 501. The EEPROM 504 reads or writes various data such as a near-terminal data processing device program under the control of the CPU 501. The CMOS sensor 505 captures an image of a subject (mainly, a self-portrait of the user who operates the near-terminal data processing device 5) under the control of the CPU 501, and obtains captured image data. The acceleration / azimuth sensor 506 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. The media I / F 508 controls reading or writing (storage) of data from or to a recording medium 507 such as a flash memory. The GPS receiving unit 509 receives a GPS signal from a GPS satellite.

  The near-terminal data processing device 5 includes a long-distance communication circuit 511, an antenna 511a of the long-distance communication circuit 511, a camera 512, an image sensor I / F 513, a microphone 514, a speaker 515, a sound input / output I / F 516, a display 517, An external device connection I / F 518, a short-range communication circuit 519, an antenna 519a of the short-range communication circuit 519, and a touch panel 521 are provided.

  Among them, the long-distance communication circuit 511 is a circuit that communicates with other devices via the intranet 200. The camera 512 is a type of a built-in imaging unit that captures an image of a subject under the control of the CPU 501 and obtains captured image data. The image sensor I / F 513 is a circuit that controls driving of the camera 512. The microphone 514 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 516 is a circuit that processes input / output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a type of display means such as a liquid crystal display or an organic EL display that displays an image of a subject and various icons. The external device connection I / F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 521 is a type of input unit that operates the near terminal data processing device 5 when the user presses the display 517.

  Further, the near terminal data processing device 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

<Hardware configuration of centralized data processing server, service providing server, and authentication server>
FIG. 6 is a hardware configuration diagram of the centralized data processing server, the service providing server, or the authentication server. Here, since the centralized data processing server 7, the service providing server 8, and the authentication server 9 have the same hardware configuration, the centralized data processing server 7 will be described. The description of the server 9 is omitted.

  FIG. 6 is a hardware configuration diagram of the centralized data processing server. The centralized data processing server 7 is constructed by a computer, and as shown in FIG. 6, a CPU 701, a ROM 702, a RAM 703, an HD 704, an HDD (Hard Disk Drive) 705, a recording medium 706, a media I / F 707, and a display. 708, a network I / F 709, a keyboard 711, a mouse 712, a CD-RW drive 714, and a bus line 710. Since the centralized data processing server 7 functions as a server, an input device such as the keyboard 711 and the mouse 712 and an output device such as the display 708 may not be provided.

  Among these, the CPU 701 controls the operation of the centralized data processing server 7 as a whole. The ROM 702 stores a program used for driving the CPU 701. The RAM 703 is used as a work area of the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls reading or writing of various data from or to the HD 704 under the control of the CPU 701. A media I / F 707 controls reading or writing (storage) of data from or to a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, a menu, a window, characters, or an image. The network I / F 709 is an interface for performing data communication using the Internet 600. The keyboard 711 is a type of input unit having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 714 controls reading of various data from a CD-RW (Compact Disc-ReWritable) 713 as an example of a removable recording medium.

  Further, the centralized data processing server 7 includes a bus line 710. The bus line 710 is an address bus, a data bus, and the like for electrically connecting each component such as the CPU 701 shown in FIG.

<< Software Configuration >>
FIG. 7 is a software configuration diagram of the real-time data processing terminal and the near-terminal data processing device in the image acquisition terminal.

  As shown in FIG. 7, the real-time data processing terminal 3a has an OS 300 and an image recognition application (hereinafter, “application” is referred to as “app”) AP1. The image recognition application AP1 operates on the work of the RAM 303 of the real-time data processing terminal 3a. The OS 300 is basic software that provides basic functions and manages the entire real-time data processing terminal 3a. The image recognition application AP1 is an application for recognizing a face of a person, an animal, or the like from a captured image.

  The near terminal data processing device 5 has an OS 500 and a communication application AP2. The communication application AP2 operates on the work of the RAM 503 of the near terminal data processing device 5. The OS 500 is basic software that provides basic functions and manages the entire near terminal data processing device 5. The communication application AP2 is an application for communicating with another terminal (apparatus) such as the distributed data processing terminal 6.

  As described above, in the image acquisition terminal 2, while the real-time data processing terminal 3a performs image recognition, the near-terminal data processing device 5 communicates with the distributed data processing terminal 6 via the intranet 200, Distributed processing can be performed.

  The real-time data processing terminal 3a and the near-terminal data processing device 5 are equipped with not only an OS but also a driver, a software development kit (SDK), or an application programming interface (API). Good.

  Next, a functional configuration and processing according to an embodiment of the present invention will be described with reference to the drawings.

<< Functional Configuration of Embodiment >>
First, the functional configurations of terminals, devices, and servers that configure the communication system according to the present embodiment will be described with reference to FIGS. FIG. 8 is a functional block diagram of the communication system according to the embodiment. FIG. 8 particularly shows functional blocks of the image acquisition terminal 2.

<Functional configuration of real-time data processing terminal>
As shown in FIG. 8, the real-time data processing terminal 3a includes a calculation unit 31, a determination unit 33, an image processing unit 34, an object detection unit 35, an event generation unit 36, a display control unit 37, a connection unit 38, It has a storage / readout unit 39 and a communication unit 48. These units are functions or means realized by operating any of the components shown in FIG. 2 according to a command from the CPU 301 in accordance with a program developed on the RAM 303 from the EEPROM 304.

  The real-time data processing terminal 3a has a storage unit 3000 constructed by the ROM 302, the RAM 303, and the EEPROM 304 shown in FIG. The storage unit 3000 stores shape model data described later. Further, in the storage unit 3000, an imaging device information management DB 3001, a cycle value management DB 3002, an image acquisition program management DB 3003, a synthesis processing program management DB 3004, a distortion correction program management DB 3005, and a service program management DB 3006 are constructed.

  The image sensor information management DB 3001 includes an image sensor information management table described later. The cycle value management DB 3002 is configured by a cycle value management table described later. The image acquisition program management DB 3003 is configured by an image acquisition program management table described later. The composition processing program management DB 3004 is composed of a composition processing program management table described later. The distortion correction program management DB 3005 includes a distortion correction program management table described later. The service program management DB 3006 is configured by a service program management table described later.

(Imaging element information management table)
FIG. 10A is a conceptual diagram illustrating an image sensor information management table. In the imaging device information management table, the model number of the imaging unit 40, the number of imaging devices included in the imaging unit 40, and the type of lens used in the imaging device are managed in association with each other. The model number is an example of type information indicating the type of image pickup unit according to the number of image pickup devices and the type of lens used in the image pickup device.

(Cycle value management table)
FIG. 10B is a conceptual diagram showing a cycle value management table. In the cycle value management table, the number of imaging elements included in the imaging unit 40 and a cycle value (Frames Per Second) of an object recognition process described below are managed in association with each other.

  Each table shown in FIG. 11 is a table for managing whether each program is installed in the real-time data processing terminal 3a.

(Image acquisition program management table)
FIG. 11A is a conceptual diagram illustrating an image acquisition program management table. In the image acquisition program management table, the number of image sensors included in the imaging unit 40 and the image acquisition program (or name) are managed in association with each other. For example, if the image acquisition program for the case where the number of imaging elements is “1” is installed in the real-time data processing terminal 3a, the program “ProgC01 (one system of image acquisition)” is managed in association with the program. When the image acquisition program for the case where the number of imaging elements is "2" is installed in the real-time data processing terminal 3a, the program "ProgC02 (two systems of image acquisition)" is managed in association.

(Synthesis processing program management table)
FIG. 11B is a conceptual diagram showing a synthesis processing program management table. In the synthesis processing program management table, the number of image sensors included in the imaging unit 40 and the name of the synthesis processing program are managed in association with each other.

(Distortion correction program management table)
FIG. 11C is a conceptual diagram showing a distortion correction program management table. In the distortion correction program management table, the types of lenses included in the imaging unit 40 and distortion correction programs (or names thereof) are managed in association with each other. For example, when the distortion correction program for the lens type “wide angle” is installed in the real-time data processing terminal 3a, the program “ProgW01 (wide angle distortion correction)” is managed in association with the program. When the distortion correction program for the lens type “fisheye” is installed in the real-time data processing terminal 3a, the program “ProgW02 (fisheye distortion correction)” is managed in association with the program.

(Service program management table)
FIG. 11D is a conceptual diagram showing a service program management table. In the service program management table, an authentication server ID for identifying an authentication server and a name of a service program for executing a service provided by the image acquisition terminal 2 are managed in association with each other. For example, when the service program for the authentication server ID “a01” is installed in the real-time data processing terminal 3a, the program is managed in association with the program “ProgD01 (object detection)”. When the service program for the authentication server ID “a02” is installed in the real-time data processing terminal 3a, the program “ProgD02 (object counting)” is managed in association with the service program.

(Object information management table)
FIG. 14 is a conceptual diagram showing the object information management table. In this object information management table, the (organization) number, the position of the rectangular partial image in the detectable range P described later, the width of the partial image, the height of the partial image, and the end flag (here, “*”) are associated with each other. Managed. The position of the partial image indicates the position of the coordinates of the upper left corner of the rectangular image.

(Object displacement management table)
FIG. 15 is a conceptual diagram illustrating an object displacement management table. In this object displacement management table, the partial image ID for identifying the partial image, the position of the partial image in the detectable range P, the width of the partial image, the height of the partial image, and the partial image are stored in the centralized data processing server 7. The collation progress status indicating the collation progress status, the name of the person indicated by the partial image, and the end flag (here, “*”) are managed in association with each other. Among these, the position of the partial image in the detectable range P, the width of the partial image, and the height of the partial image are the object identification information transferred from the object information management table (see FIG. 14). That is, the information (position, width, height) in the object information management table is partial image specifying information for specifying the current partial image, and the information (position, width, height) in the object displacement management table is past information. This is partial image specifying information for specifying a partial image. The past partial image is a partial image of the object at the first time point. The current partial image is a partial image of the object at a second time point after a predetermined time from the first time point.

  The “end flag” in the object information management table (see FIG. 14) indicates that the current position of the object in an arbitrary record in the object information management table indicates that the current position of the object in each record in the object displacement management table (see FIG. 15). Is added to indicate that the determination has been completed when it is determined whether the position is within a predetermined distance (threshold) from the past position. On the other hand, the “end flag” in the object displacement management table is added to a record including the partial image specifying information when there is partial image specifying information transferred from the object information management table.

(Functional configuration of real-time data processing terminal)
Next, each functional configuration of the real-time data processing terminal 3a will be described in more detail with reference to FIG.

  The calculation unit 31 of the real-time data processing terminal 3a is realized by the processing of the CPU 301, and calculates the distance between the latest position of the object and the past position of the object.

  The determination unit 33 is realized by the processing of the CPU 301 and performs various determinations. For example, the determining unit 33 determines the number of image sensors based on the model number sent from the image capturing unit 40 and referring to the image sensor information management DB 3001.

  The image processing unit 34 is realized by the processing of the CPU 301, and performs various types of image processing by executing the programs (image acquisition program, image composition program, distortion correction program, and service program) shown in FIG. Specifically, the image processing unit 34 executes a first program (for example, an image acquisition program, an image synthesis program, and a distortion correction program) that does not require authentication in order to acquire image data. First image processing (for example, image acquisition, image synthesis, distortion correction) is performed. In addition, the image processing unit 34 executes a second program (for example, a service program) that requires authentication in order to perform second image processing (for example, object detection processing, object Counting process).

  The object detection unit 35 is realized by the processing of the CPU 301, detects a feature point that is a candidate for a predetermined specific object such as a face in the data of the captured image acquired by the image processing unit 34, and The position of the object in the captured image is detected with reference to shape model data indicating a predetermined shape model of the specific object.

  The event generation unit 36 is realized by the processing of the CPU 301, and generates detection information (event information) indicating that the position of the object has been detected by the object detection unit 35.

  The display control unit 37 is realized by the processing of the CPU 301, and causes the display 317, which is an example of a display unit, to display various screens.

  The connection unit 38 is realized by the processing of the imaging unit I / F 313 and the CPU 301, and is an interface for mechanically and electrically connecting the imaging unit 40 to the real-time data processing terminal 3a.

  The storage / read unit 39 is realized by the processing of the CPU 301, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.

  The communication unit 48 is realized by the processing of the external device connection I / F 318 and the CPU 301, and transmits and receives various data (or information) to and from a communication unit 58 of the near terminal data processing device 5 via one-to-one communication. Do. This communication may be wireless as well as wired.

<Functional configuration of near terminal data processing device>
As shown in FIG. 8, the near terminal data processing device 5 includes a transmission / reception unit 51, a data detection unit 56, a display control unit 57, a communication unit 58, and a storage / readout unit 59. These units are realized by operating any of the components shown in FIG. 5 in the near-terminal data processing device 5 according to an instruction from the CPU 501 in accordance with a program developed from the EEPROM 504 to the RAM 503. Function or means to be performed.

  In addition, the near terminal data processing device 5 includes a storage unit 5000 configured by the ROM 502, the RAM 503, and the EEPROM 504 illustrated in FIG.

(Each functional configuration of the near terminal data processing device)
Next, each functional configuration of the near terminal data processing device 5 will be described in more detail with reference to FIG.

  The transmission / reception unit 51 of the near-terminal data processing device 5 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501, and communicates with the distributed data processing terminal 6 via a communication network (here, the intranet 200). To send and receive various data (or information).

  The data detection unit 56 is realized by the processing of the CPU 501, and detects whether an event for receiving data from the real-time data processing terminal 3a has occurred and whether data reception has been completed.

  The display control unit 57 is realized by the processing of the CPU 501, and causes the display 517, which is an example of a display unit, to display various screens.

  The communication unit 58 is realized by processing of the external device connection I / F 518 and the CPU 501, and transmits and receives various data (or information) to and from the communication unit 48 of the real-time data processing terminal 3a via one-to-one communication. Note that this communication may be not only wired but also wireless.

  The storage / readout unit 59 is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000.

<Functional configuration of distributed data processing terminal>
As illustrated in FIG. 9, the distributed data processing terminal 6 includes a transmitting / receiving unit 61, a receiving unit 62, a determining unit 63, a display control unit 67, and a storage / readout unit 69. These units are realized by the operation of any one of the components shown in FIG. 5 in the distributed data processing terminal 6 by an instruction from the CPU 501 in accordance with a program developed from the EEPROM 504 to the RAM 503. Function or means.

  Further, the distributed data processing terminal 6 includes a storage unit 6000 configured by the ROM 502, the RAM 503, and the EEPROM 504 illustrated in FIG.

(Each functional configuration of distributed data processing terminal)
The transmission / reception unit 61 of the distributed data processing terminal 6 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501 in the distributed data processing terminal 6, and transmits the centralized data via the communication network (here, the Internet 600). It sends and receives various data (or information) to and from the processing server 7. For example, the transmission / reception unit 61 transmits a request for collation between collation data and data to be collated to the centralized data processing server 7, and performs processing on the collation result sent from the centralized data processing server 7.

  The receiving unit 62 is realized by the processing of the touch panel 521 and the CPU 501 in the distributed data processing terminal 6, and receives various operations of the user.

  The determination unit 63 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and makes various determinations.

  The display control unit 67 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and causes the display 517, which is an example of a display unit, to display various screens.

  The storage / readout unit 69 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000. For example, the storage / readout unit 69 stores the collation data (here, face image data) in the storage unit 6000 and registers it in response to the registration request received by the reception unit 62.

<Functional configuration of centralized data processing server>
As illustrated in FIG. 9, the centralized data processing server 7 includes a transmitting / receiving unit 71, a receiving unit 72, a determining unit 73, a feature amount generating unit 74, a collating unit 75, and a storage / readout unit 79. . These units are realized by the operation of one of the components shown in FIG. 6 in the distributed data processing terminal 6 by the instruction from the CPU 701 according to the program expanded from the HD 704 on the RAM 703. Function or means.

  Further, the centralized data processing server 7 has a storage unit 7000 constructed by the ROM 702, the RAM 703, and the HD 704 shown in FIG. The storage unit 7000 stores feature amount data on the collation side described later. This storage unit 7000 stores collation data. Further, a collation data management DB 7001 is constructed in the storage unit 7000. The collation data management DB 7001 includes a collation data management table described later. The collation data may be stored in a data management server other than the centralized data processing server 7 or the like.

(Collation data management table)
FIG. 12 is a conceptual diagram showing a collation data management table. In the collation data management table, an image file indicating the file name of the collation data and the name of the person indicated by the collation image related to the image file are managed in association with each other. Note that the name of a person is an example of person identification information. The person identification information may be indicated by not only kanji but also alphanumeric characters and symbols. For example, the person identification information includes an employee number, a student ID number, a driver's license number, a My Number used in the Japanese system, and the like.

(Each functional configuration of the centralized data processing server)
The transmission / reception unit 71 of the centralized data processing server 7 is realized by the processing of the network I / F 709 and the CPU 701, and communicates with the distributed data processing terminal 6 and various data (or information) via a communication network (here, the Internet 600). Send and receive. The transmission / reception unit 71 receives a collation request between collation data and data to be collated from the distributed data processing terminal 6, and transmits collation result information indicating a collation result to the distributed data processing terminal 6.

  The receiving unit 72 is realized by the processing of the keyboard 711, the mouse 712, and the CPU 701, and receives various operations of the user.

  The determination unit 73 is realized by the processing of the CPU 501 in the centralized data processing server 7, and makes various determinations.

  The feature amount generation unit 74 is realized by the processing of the CPU 701, and generates a parameter of the feature amount from the data to be verified (partial image data) and the verification data received by the transmission / reception unit 71.

  The matching unit 75 is implemented by the processing of the CPU 701, and compares the feature amount on the matching data side with the feature amount on the data to be matched using the feature amount generated by the feature amount generating unit 74, and obtains a score indicating the similarity. (Point) is calculated.

  The storage / readout unit 79 is realized by the processing of the CPU 701, and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000.

<Functional configuration of service providing server>
Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIGS. 9 and 13 to 17. As shown in FIG. 9, the service providing server 8 includes a transmission / reception unit 81, a determination unit 82, an extraction unit 87, and a storage / readout unit 89. Each of these units has a function or a function realized by operating any one of the components shown in FIG. 6 according to an instruction from the CPU 701 according to the program for the service providing server 8 expanded on the RAM 703 from the HD 704 or Means.

  In addition, the service providing server 8 includes a storage unit 8000 configured by the RAM 703 and the HD 704 illustrated in FIG. The storage unit 8000 stores various data sent from the distributed data processing terminal 6 or the authentication server 9. The storage unit 8000 stores all the programs shown in FIG. 11, and the service providing server 8 transmits the requested program in response to a request from the real-time data processing terminal 3a. Can be.

  Further, a session management DB 8001 and an authentication server management DB 8002 are constructed in the storage unit 8000. Among them, the session management DB 8001 is configured by a session management table described later. The authentication server management DB 8002 is configured by an authentication server management table described later. Hereinafter, each table will be described in detail.

(Session management table)
FIG. 13 is a conceptual diagram showing a session management table. The session management table includes a session ID for identifying a communication session established with the distributed data processing terminal 6 and a user for identifying a user of the distributed data processing terminal 6 in order to provide a service to the distributed data processing terminal 6. The terminal ID and the IP address of the distributed data processing terminal 6 of the user indicated by the terminal ID are stored and managed in association with each other.

  FIGS. 16A, 16B, and 16C show an e-mail address as an example of the terminal ID (identification), and are each composed of an authentication target portion and a non-authentication target portion. The authentication target portion is a user ID used when authentication is performed by the authentication server 9. The non-authentication target portion is a portion that is not used when authentication is performed by the authentication server 9.

  Of these, in the first pattern shown in FIG. 16A, the part to be authenticated is composed of the account name “asai”, the host name “myhost”, and the former part “ricoo.com” of the domain name. ing. On the other hand, the part not to be authenticated is constituted by the latter part “theta1” of the domain name. In this case, the extraction unit 87 distinguishes the part to be authenticated from the part not to be authenticated by “/”.

  FIG. 16B also shows the first pattern, but the part not to be authenticated is different from FIG. 16A. That is, the authentication server 9 authenticates the terminal ID shown in FIG. 16A and the terminal ID shown in FIG. 16B as the same ID because the authentication target portion is the same.

  Note that the terminal ID may be the second pattern shown in FIG. In the second pattern, the authentication target portion is constituted by the first part “asai” of the account name. On the other hand, the non-authentication target part is composed of the latter part of the account name “theta2”, the host name “myhost”, and the domain name “ricoo.com”. In this case, the extraction unit 87 distinguishes the part to be authenticated from the part not to be authenticated by “+”.

(Authentication server management table)
FIG. 17 is a conceptual diagram showing the authentication server management table. In the authentication server management table, a URL (Uniform Resource Locator) for accessing each authentication server 9 is stored and managed in association with each authentication server ID for identifying each authentication server 9.

(Each functional configuration of the service providing server)
Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIG.

  The transmission / reception unit 81 of the service providing server 8 is mainly realized by a command from the CPU 701 shown in FIG. 6 and the network I / F 709, and via the Internet 600, the distributed data processing terminal 6 or the authentication server 9. And various data (or information) are transmitted and received.

  The determination unit 82 is mainly realized by an instruction from the CPU 701 shown in FIG. 6, and determines, for example, whether a communication session for providing a service to the distributed data processing terminal 6 has already been established. .

  The extracting unit 87 is mainly realized by an instruction from the CPU 701 illustrated in FIG. 6, and performs a process of extracting an rID (authentication target portion) as illustrated in FIG. 16 from the terminal ID.

  The storage / readout unit 89 is mainly realized by an instruction from the CPU 701 shown in FIG. 6 and the HDD 705, and stores various data in the storage unit 8000 and reads various data from the storage unit 8000.

<Functional configuration of authentication server>
Next, the functional configuration of the authentication server 9 will be described in detail with reference to FIGS. The authentication server 9 includes a transmission / reception unit 91, an authentication unit 92, and a storage / readout unit 99. These units are functions or means realized by operating any one of the constituent elements shown in FIG. 6 by a command from the CPU 701 according to the program for the authentication server 9 expanded on the RAM 703 from the HD 704. It is.

  Further, the authentication server 9 has a storage unit 9000 configured by the RAM 703 and the HD 704 illustrated in FIG. Various data sent from the distributed data processing terminal 6 or the service providing server 8 is stored in the storage unit 9000.

  Further, an authentication management DB 9001 is constructed in the storage unit 9000. The authentication management DB 9001 is configured by an authentication management table described later. Hereinafter, this table will be described in detail.

(Authentication management table)
18A is a conceptual diagram illustrating an authentication management table of the authentication server 9a, FIG. 18B is a conceptual diagram illustrating an authentication management table of the authentication server 9b, and FIG. 18C is an authentication of the authentication server 9c. It is a conceptual diagram showing a management table.

  In each authentication management table, a user ID (authentication target portion) of the terminal ID and a password are stored in association with each other and managed.

(Each functional configuration of the authentication server)
Next, each functional configuration of the authentication server 9 will be described in detail with reference to FIG.

  The transmission / reception unit 91 of the authentication server 9 is mainly realized by instructions from the CPU 701 shown in FIG. 6 and the network I / F 709, and communicates with the distributed data processing terminal 6 and the service providing server 8 via the Internet 600. Send and receive data (or information).

  The authentication unit 92 is implemented mainly by a command from the CPU 701 shown in FIG. 6, and determines whether or not the image acquisition terminal 2 that has transmitted the authentication request has valid authority to receive the service. Authenticates the ID.

  The storage / readout unit 99 is mainly realized by an instruction from the CPU 701 shown in FIG. 6 and the HDD 705, and stores various data (or information) in the storage unit 9000, and stores various data (or information) from the storage unit 9000. Or information).

<< processing or operation of the present embodiment >>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIGS.

<Authentication process>
First, the authentication process will be described with reference to FIGS. FIG. 19 and FIG. 20 are sequence diagrams showing the authentication processing. FIG. 21 and FIG. 22 are diagrams illustrating examples of screens of the distributed data processing terminal.

  As shown in FIG. 19, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits a request for an authentication destination selection screen to the service providing server 8 (step S21). This request includes the terminal ID of the distributed data processing terminal 6. At this time, the transmitting / receiving unit 61 transmits the IP address of the terminal itself. Thereby, the transmission / reception unit 81 of the service providing server 8 receives the request for the authentication destination selection screen and the IP address of the distributed data processing terminal 6.

  Next, the determination unit 82 of the service providing server 8 determines whether the terminal ID received in step S21 is managed in the session management table (see FIG. 13) in association with the predetermined session ID (step S21). S22). Hereinafter, a case where the terminal ID is not managed will be described.

  The transmission / reception unit 81 of the service providing server 8 transmits the data of the authentication destination selection screen to the distributed data processing terminal 6 (Step S23). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the data of the authentication destination selection screen.

  Next, the display control unit 67 of the distributed data processing terminal 6 causes the display 517 to display an authentication destination selection screen s1 as shown in FIG. 21 (step S24). FIG. 21 shows a screen example as the distributed data processing terminal 6. The authentication destination selection screen s1 displays a terminal ID input field b1, a password input field b2, and a login button b3 for requesting a login (authentication request). Further, on the authentication destination selection screen s1, authentication server selection buttons a1, a2, and a3 for selecting the authentication servers 9a, 9b, and 9c, respectively, are displayed. For example, the authentication server selection button a1 is a button used when the user receives the service of the object detection processing. The authentication server selection button a2 is a button used when the user receives the service of the object counting process.

  Here, the user inputs his / her terminal ID in the input field b1 and his / her password in the input field b2, presses a desired button among the authentication server selection buttons a1, a2, a3, and presses the login button b3. Then, the receiving unit 62 receives each input and selection (Step S25). Here, a case will be described in which the object detection processing is executed by the object detection processing program ProgD01 in the image recognition processing when the authentication server selection button a1 is selected.

  The transmission / reception unit 61 transmits an authentication request for an ID (here, a terminal ID or a user ID) to the service providing server 8 (step S26). This authentication request includes the terminal ID and password received in step S25, the selection result of the authentication server 9, and the URL of the distributed data processing terminal 6. This authentication result indicates an authentication server ID for identifying the authentication server 9. Thereby, the transmitting / receiving unit 81 of the service providing server 8 receives the ID authentication request.

  Next, the storage / read unit 89 of the service providing server 8 responds by searching the authentication server management table (see FIG. 17) using the authentication server ID as the selection result received in step S26 as a search key. The URL of the authentication server is read (step S27).

  Next, the extraction unit 87 extracts only the user ID (authentication target portion) from the terminal IDs received in step S26 (step S28). Then, the transmission / reception unit 81 transmits an ID authentication request to the authentication server 9 indicated by the URL read in step S27 (step S29). The ID authentication request includes the user ID (the part to be authenticated) extracted in step S28, the password received in step S26, and the URL of the distributed data processing terminal 6 received in step S26. . Thereby, the transmission / reception unit 71 of the authentication server 9 receives the user authentication request.

  Next, the storage / readout unit 99 of the authentication server 9 uses the set of the user ID (authentication target portion) and the password received in step S29 as a search key, and sets the same set of authentication in the authentication management table (see FIG. 18). The authentication unit 92 performs authentication using the search result of the target part and the password (step S30). If the same set is managed, the authentication unit 92 determines that the distributed data processing terminal 6 is a legitimate terminal for receiving a service from the service providing server 8, and if the same set is not managed, Determines that the distributed data processing terminal 6 is not a legitimate terminal for receiving a service from the service providing server 8.

  In step S28, the extracting unit 87 extracts the part to be authenticated from the terminal ID, but the present invention is not limited to this. For example, the service providing server 8 does not have the extracting unit 87, and in step S29, the transmitting / receiving unit 81 transmits only the user ID (authentication target portion) of the terminal ID in addition to the password and the URL. It may be.

  Subsequently, as shown in FIG. 20, the authentication unit 92 of the authentication server 9 encrypts the token (transmission right) (step S41). Then, based on the URL of the distributed data processing terminal 6 received in step S29, the transmission / reception unit 91 transmits an authentication result to the distributed data processing terminal 6 (step S42). The authentication result indicates whether the distributed data processing terminal 6 is valid or not, and includes the token encrypted in step S41. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the authentication result of the user. Hereinafter, a case in which the user has valid authority will be described.

  The transmission / reception unit 61 of the distributed data processing terminal 6 transmits a session establishment request to the service providing server 8 (Step S43). This session establishment request includes the terminal ID and the encrypted token received in step S42. Thereby, the transmission / reception unit 81 of the service providing server 8 receives the session establishment request.

  Next, the service providing server 8 transmits the token establishment request to the authentication server 9 in order to confirm that the distributed data processing terminal 6 that has transmitted the session establishment request is determined to be valid in step S30. An authentication request is transmitted (step S44). This token authentication request includes the encrypted token received in step S43. Thereby, the transmission / reception unit 91 of the authentication server 9 receives the token authentication request.

  Next, the authentication unit 92 decrypts the encrypted token received in step S44 (step S45). Then, the authentication unit 92 authenticates the token by comparing the token before encryption in step S41 with the token after decryption in step S45 (step S46). Then, the transmission / reception unit 91 of the authentication server 9 transmits the authentication result of Step S46 to the service providing server 8 (Step S47). Thereby, the transmission / reception unit 81 of the service providing server 8 receives the authentication result. Hereinafter, a case where the token is determined to be valid in step S46 will be described.

  Next, the storage / readout unit 89 of the service providing server 8 assigns a new session ID in the session management table (see FIG. 13), and assigns the terminal ID and the IP received in step S26 to this session ID. The addresses are managed in association with each other (step S48). Then, the transmitting and receiving unit 81 transmits the data of the service providing screen to the distributed data processing terminal 6 (Step S49). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the data of the service providing screen.

  Next, the display control unit 67 of the distributed data processing terminal 6 displays a service providing screen s2 as shown in FIG. 22 on the display 517 (step S50). FIG. 22 shows a screen example of the distributed data processing terminal 6. Here, as a service providing example, a remote operation service for realizing remote operation from the distributed data processing terminal 6 to the image acquisition terminal 2 will be described. On the service providing screen s2 shown in FIG. 22, an input field c1 of an IP address for specifying a remote operation target and a "start remote operation" button c2 are displayed.

<Preparation processing for image recognition>
Subsequently, a preparation process for image recognition will be described with reference to FIGS. FIG. 23 is a sequence diagram showing a process of an image recognition start request.

  As shown in FIG. 23, in the distributed data processing terminal 6, the receiving unit 62 receives a request for starting image recognition from the user (step S61). In this case, a GUI (Graphical User Interface) of the distributed data processing terminal 6 is used. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits the image recognition start request information indicating the image recognition start request to the near terminal data processing device 5 (step S62). The start request information includes an authentication server ID for identifying the authentication server 9 that has performed authentication in the above-described authentication processing (see FIGS. 19 and 20). Thereby, the transmission / reception unit 51 of the near terminal data processing device 5 receives the image recognition start request information. Then, the communication unit 58 transmits the image recognition start request information to the real-time data processing terminal 3a (step S63). Thereby, the communication unit 48 of the real-time data processing terminal 3a receives the image recognition start request information. In this way, by separating the user interface from the real-time data processing terminal 3a, remote operation from the distributed data processing terminal 6 becomes possible.

FIG. 24 is a sequence diagram showing a preparation process for real-time processing of the real-time data processing terminal.
As shown in FIG. 24, the connection unit 38 of the real-time data processing terminal 3a acquires the model number of the imaging unit 40 from the imaging unit 40 (Step S71). In this case, the connection unit 38 requests a model number from the imaging unit 40, and the imaging unit 40 transmits the model number of the own unit in response to the request.

  Next, the storage / readout unit 39 searches the image sensor information management DB 3001 (see FIG. 10A) using the model number obtained in step S71 as a search key, thereby obtaining the number of corresponding image sensors and the lens. Is read out (step S72). Further, the storage / readout unit 39 reads out the corresponding cycle value by searching the cycle value management DB 3002 (see FIG. 10B) using the number of image sensors read in step S72 as a search key (see FIG. 10B). Step S73).

  Next, the storage / readout unit 39 searches for an image acquisition program managed in the image acquisition program management DB 3003 (see FIG. 11A) using the number of image sensors read in step S72 as a search key. (Step S74). Next, the storage / readout unit 39 searches for a combination processing program managed in the combination processing program management DB 3004 (see FIG. 11B), using the number of imaging elements read in step S72 as a search key. (Step S75). Next, the storage / readout unit 39 searches for a distortion correction program managed in the distortion correction program management DB 3005 (see FIG. 11C) using the lens type read in step S72 as a search key (see FIG. 11C). Step S76). Next, the storage / readout unit 39 searches for a service program managed in the service program management DB 3006 (see FIG. 11D) using the lens type read out in step S72 as a search key (step S77). ).

  Next, the determining unit 33 determines whether or not all the programs to be executed have been installed based on whether or not the programs to be executed are managed based on the search results of the above steps S74 to S77 (step S78). For example, in step S74, as a result of the search for the image processing program by the storage / readout unit 39, if the image processing program is managed, the determination unit 33 determines that it is installed, and if it is not managed, The determination unit 33 determines that the installation has not been completed.

  Then, when the determining unit 33 determines that all four programs are managed (step S78; YES), the processing illustrated in FIG. 24 ends. On the other hand, when the determining unit 33 determines that at least one of the four programs is not installed (step S78; NO), the process proceeds to step S91 described below.

  FIG. 25 is a sequence diagram illustrating a program acquisition process. Here, a process is shown in which the real-time data processing terminal 3a acquires, from the service providing server 8, a program determined to have not been installed by the process shown in FIG.

  First, as shown in FIG. 25, the communication unit 48 of the real-time data processing terminal 3a sends request information indicating a request for a program not installed to the communication unit 58 of the near-terminal data processing device 5. It transmits (step S91). This request information includes the name of the requested program.

  Next, the transmission / reception unit 51 of the near terminal data processing device 5 transmits the request information received by the communication unit 58 to the transmission / reception unit 61 of the distributed data processing terminal 6 (Step S92). Then, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits the request information to the transmission / reception unit 81 of the service providing server 8 (Step S93).

  Next, in the service providing server 8, the storage / readout unit 89 reads out the program indicated by the program name included in the request information (Step S94). Then, the transmission / reception unit 81 transmits the read program to the transmission / reception unit 61 of the distributed data processing terminal 6 (Step S95). At this time, the program name is also transmitted.

  Next, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits the program including the program name to the transmission / reception unit 51 of the near terminal data processing device 5 (Step S96). Then, the communication unit 58 of the near terminal data processing device 5 transmits the program including the program name to the communication unit 48 of the real-time data processing terminal 3a (step S97).

  Next, the storage / readout unit 39 of the real-time data processing terminal 3a installs the program acquired by the communication unit 48 and manages the name of the program to be installed among the tables shown in FIG. 11 ( Step S98).

  Next, the storage / readout unit 39 activates all programs necessary for the image recognition processing (step S99). Thereby, the real-time data processing terminal 3a starts the real-time processing described below by executing the activated program.

<Image recognition processing>
Here, a case will be described in which the “login to object detection processing service” button b1 shown in FIG. 21 is pressed and the service providing server 8 executes the object detection processing service among the image recognition processing services. . FIG. 26 is a sequence diagram showing the image recognition processing. First, the real-time data processing terminal 3a performs real-time processing (step S111).

(Object detection processing)
Here, the processing of object detection in the real-time processing will be described with reference to FIG. FIG. 27 is a flowchart showing the object detection processing in the real-time processing.

  First, the determining unit 33 determines whether or not the number of the imaging elements in the connected imaging unit 40 is one (Step S201). In this case, the determination unit 33 makes the determination based on the number of the imaging elements read by the processing in step S72 described above. If the number of image sensors is one (step S201; YES), the image processing unit 34 sets a cycle value for repeating the process to 1/60 seconds (step S202). In this case, the image processing unit 34 sets the cycle value read by the processing in step S73 described above.

  Next, the connection unit 38 acquires one system of captured image data from the imaging unit 40 (step S203). The captured image data is digital image data, for example, data of a 4K image (3840 images wide × 2160 pixels high). In this case, the connection unit 38 executes processing by the image acquisition program (ProgC01 (one system)) shown in FIG.

  Next, the object detection unit 35 detects an object by detecting a feature point that is a candidate for the object in the captured image data (step S204). In this case, the image processing unit 34 executes the processing by the service program (ProgD01 (object detection processing)) shown in FIG. The image processing unit 34 searches for a rectangle from the end of the captured image while comparing it with the shape model data of the object stored in the storage unit 3000 in advance, and selects a position of a plausible feature point of the object. The processing in step S204 is described in, for example, “Face Recognition Technology and Its Application: Special Issue on Elemental Technologies and Solutions Supporting Public Safety; Biometrics Authentication, NEC Technical Report, Vol. 63, no. 3, pp. 26-30 , 2010-09 "can be used.

Next, the image processing unit 34 corrects the distortion of the image of the detected object (Step S205).
In this case, the image processing unit 34 executes a process using an image processing program (ProgW01 (wide-angle distortion correction)) shown in FIG.

  On the other hand, if the number of imaging elements is not one in step S201 (step S201; NO), the image processing unit 34 sets the cycle value for repeating the real-time processing to 1/30 seconds (step S206). ). In this case, the image processing unit 34 sets the cycle value read by the processing in step S73 described above. By setting the cycle value to 1/30 second, it is possible to prevent a later-described image synthesizing process from being overtaken by setting the cycle value larger than one input.

  Next, the connection unit 38 acquires two pieces of captured image data of two systems from the imaging unit 40 (Step S207). The two captured image data are hemispherical image data as shown in FIGS. 4A and 4B, respectively. In this case, the connection unit 38 executes processing by the image acquisition program (ProgC02 (two systems)) shown in FIG.

  Then, the image processing unit 34 combines the two captured image data to create an equirectangular projected image EC as shown in FIG. 4C (step S208). In this case, the image processing unit 34 executes the processing by the synthesis processing program (ProgS02 (synthesis processing)) shown in FIG. Then, the process proceeds to step S204, in which the object detection unit 35 detects an object by detecting a feature point that is a candidate for the object in the data of the equirectangular projection image EC.

  Next, in step S205, the image processing unit 34 corrects the distortion of the image of the detected object. In this case, the image processing unit 34 executes the processing by the image processing program (ProgW02 (fisheye distortion correction)) shown in FIG.

  As described above, as shown in FIG. 26, the communication unit 48 of the real-time data processing terminal 3a transmits the captured image data to the communication unit 58 of the near terminal data processing device 5 (step S112). Then, the transmission / reception unit 51 of the near terminal data processing device 5 transmits the captured image data received in step S112 to the transmission / reception unit 61 of the distributed data processing terminal 6 (step S113). Accordingly, the display control unit 67 of the distributed data processing terminal 6 causes the display 517 to display a captured image as shown in FIG. 32 in real time (step S114). FIG. 32 is a diagram illustrating a display example of a captured image in the distributed data processing terminal. In this captured image, a frame indicating a rectangle in which an object has been detected (here, a face has been detected) is displayed. Steps S112 to S114 are streaming processing.

(Event generation processing)
Next, the event generation processing in the real-time processing will be described with reference to FIGS. FIGS. 28 and 29 are flowcharts showing the event generation processing in the real-time processing.

  If the object detection unit 35 does not detect an object (here, a face) by the processing in step S204 (S211; NO), after waiting for a predetermined time (set cycle value) (S212), The process returns to step S211. For example, detection is performed 30 times per second.

  On the other hand, if the object detection unit 35 detects an object (here, a face) by the processing in step S204 described above (S211: YES), the process proceeds to step S213. If the position of the detected object is not in the collation range Q (outside the collation range Q) (S211; NO), the process proceeds to step S212.

  In step S213, the storage / readout unit 39 stores the object information regarding the object detected in step S204 as a new record in the object information management table (see FIG. 14). In this case, the storage / readout unit 39 deletes the end flag for the record already stored in the object information management table for resetting.

  Next, in order to initialize the end flag stored in the object displacement management table, the storage / readout unit 39 deletes the end flag for all records in the object displacement management table (S214). The end flag is a flag added in step S216 described later.

  Next, when records to be read remain in the object displacement management table (S215; YES), the storage / readout unit 39 reads data of one record from the object displacement management table (S216). On the other hand, if all records have been read (S215; NO), the process proceeds to step S220 described later.

  Next, the calculation unit 31 calculates the latest (current) object detection position stored in step S213 and the past object detection position (here, coordinates (x, y)) stored in step S216. Is calculated (S217). Then, the determining unit 33 determines whether or not the distance calculated in step S217 is shorter (less than) a predetermined threshold (S218). In the present embodiment, it is calculated using the square Euclidean distance shown in (Equation 1). Note that the determination unit 33 may determine whether the value is equal to or less than a predetermined threshold.

  Since the square Euclidean distance has a larger weight as the distance from the target increases, there is an advantage that the position of a past object having a shorter distance can be easily found. It should be noted that other well-known distance calculation methods may be used, for example, a process of counting the number of pixels shifted only in an urban area distance or a specific direction (x direction or y direction) may be used. it can.

  Next, if the distance calculated in step S217 is shorter than the predetermined threshold (S218; YES), the distance between the current object position and the past position of the remaining records managed in the object displacement management table is determined. The process proceeds to step S219 without calculating the distance between them. If the distance calculated in step S217 is not shorter (same or longer) than the predetermined threshold (S218; NO), the process returns to step S215.

  Next, the storage / readout unit 39 stores, in the object displacement management table, the position information indicating the position of the past object whose distance is determined to be shorter than the threshold, and the position information of the latest object whose distance is determined to be shorter than the threshold. Is updated by updating the object displacement management table (S219). In this case, the storage / readout unit 39 adds “end flag” to the end flag column of the record including the position information used for the replacement in the object information management table and the object displacement management table.

  Next, in the object displacement management table, when a record to which the end flag is not added remains (S220; YES), the storage / readout unit 39 deletes the record to which the end flag is not added. (S221), the process proceeds to step S231 shown in FIG. In this processing, if the end flag is not added in the object displacement management table and the object remains, it can be estimated that the object (here, a person) has already come out of the collation range Q described later. This is a process for deleting a record from the table and excluding it from future comparison targets.

  On the other hand, if there is no record to which the end flag is not added in the object displacement management table (S220; NO), the process of step S221 is omitted, and the process proceeds to step S231.

  Next, when there is a record to which the end flag is not added in the object information management table (S231; YES), the storage / readout unit 39 stores the position information of the remaining record in the object displacement management table. It is added as a new record (S232). In this case, the storage / readout unit 39 manages the collation progress status as “0 (before recognition)” in the added record. This process is a process for managing subsequent displacement since it can be estimated that a new object (here, a person) has entered the detectable range P described later.

  In this case, the storage / read unit 39 stores “+1” in the ID column of the added record with the value of the ID added last time. Note that the storage / readout unit 39 does not renumber IDs when deleting a record in step S221. By doing so, a correct result can be stored for a record whose collation progress status is “1 (collating)”.

  On the other hand, when there is no record in the object information management table to which the end flag is not added (S231; NO), the process proceeds to step S236 described below.

  Next, after the process of step S232, the determination unit 33 determines whether or not the object with the collation progress status “0 (before collation)” is within the collatable range Q in the object displacement management table (S233). . This determination is made based on the position information of the object. If it is determined in step S233 that the object is within the collation range Q (S233; YES), the image processing unit 34 determines the position and width of the object within the collation range Q in the object displacement management table. , And the height, the partial image data determined by these values is cut out, and encoded in a general-purpose format such as JPEG (S234). In this case, the storage / readout unit 39 changes the collation progress status of the corresponding record in the object displacement management table to “1 (collating)”.

  On the other hand, if it is determined in step S233 that it is not within the collationable range Q (S233; NO), the process proceeds to step S236.

  Next, the event generation unit 36 generates an event message for notifying the near-terminal data processing device 5 of transmitting the partial image data and the partial image ID associated with the partial image data (S235). Specifically, the event generation unit 36 generates an event message such as “Send”.

  Subsequently, after clearing the displayed character image, the image processing unit 34 associates the collation progress status of the object displacement management table with “1 (collating)” or “2 (collation completed)”. With reference to the position, a character image is synthesized at a position corresponding to the captured image (S236).

Here, FIG. 32 shows an example in which a character combines an image. FIG. 32 is a diagram illustrating a shooting range of the imaging unit when an image acquisition terminal is installed in a certain room. The image processing unit 34 synthesizes a standby state mark m1 represented by “!” To indicate that the partial image at the position (x ₂ , y ₂ ) is being compared. In addition, the image processing unit 34 refers to the name to indicate the collation result with respect to the partial image at the position (x ₃ , y ₃ ), and refers to the name and sets the name information m2 represented by “Kato” at the corresponding position of the partial image. Are synthesized. This name information m2 is information read from the object displacement management DB 3008. The combined image data is transmitted to the distributed data processing terminal 6 in steps S112 and S113 shown in FIG. 26, and is displayed in step S114.

  Next, when the communication unit 48 has not received the collation result in step S318 described below (S237; NO), the process proceeds to step S212.

  On the other hand, when the communication unit 48 receives the collation result (the name and the partial image ID) (S237; YES), the storage / readout unit 39 sets the name associated with the received partial image ID in the object displacement management table. The received name is stored in the column (S238). In this case, the storage / readout unit 39 changes the collation progress status from “1 (collating)” to “2 (collation completed)”. Then, after clearing the displayed character image, the image processing unit 34 refers to the position of the object displacement management table associated with the received partial image ID, and receives the received name at the corresponding position in the captured image. To compose a character image (S239). Then, the process returns to step S212. In step S239, the display control unit 37 updates the display of "!" Being collated to the display of the name such as "Kato". Thereby, the user of the distributed data processing terminal 6 can quickly know the progress of the collation.

  Thus, the event generation processing of the real-time processing ends.

  Subsequently, returning to FIG. 26, the communication unit 48 of the real-time data processing terminal 3a transmits the partial image data and the partial image ID of the object (here, the face) to the communication unit 58 of the near-terminal data processing device 5. It is transmitted (S115).

  Next, in the near-terminal data processing device 5, the data detection unit 56 detects whether or not the “Send” event message has been received by the communication unit 58 (S116). When the reception of the event message is detected (S116; YES), the communication unit 58 receives the partial image data and the partial image ID transmitted together with the event message (S117). Then, the storage / readout unit 59 temporarily stores the partial image data in the storage unit 5000 (S118).

  Next, the data detection unit 56 monitors whether the reception of the partial image data has been completed (S119). The process of step S119 is repeated until all partial image data and partial image IDs are received for one event message (S119; NO). Then, when the reception of the partial image data is completed (S119; YES), the storage / readout unit 59 sends all the data temporarily transmitted from the storage unit 5000 together with one event message and temporarily stored in the storage unit 5000. The partial image data and partial image ID are read out (step S120). Thereafter, the transmission / reception unit 51 transmits all the partial image data and the partial image ID read in step S120 to the transmission / reception unit 61 of the distributed data processing terminal 6 via the intranet 200 (S121). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives all the partial image data and the partial image ID. The partial image data is used later as data to be compared.

(Collation processing)
Subsequently, the collation processing will be described with reference to FIGS. 5 is a flowchart illustrating a generation process. FIG. 30 is a sequence diagram illustrating a process of matching. FIG. 31 is a flowchart showing the collation processing. FIG. 32 is a diagram illustrating a shooting range of the imaging unit when an image acquisition terminal is installed in a certain room.

  First, in order to register collation data, the user accesses the centralized data processing server 7 from the distributed data processing terminal 6 to register the collation data in the centralized data processing server 7 (S311). Specifically, the storage / readout unit 79 stores the storage unit 7000 collation data, and stores the collation data management DB 7001 in association with the file name and the name.

  Next, in the centralized data processing server 7, the feature amount generation unit 74 performs a conversion for decoding the collation data registered in step S311 into bitmap data, and performs processing such as unevenness and inclination of the eyes and nose of the face image related to the collation data. A parameter of a feature amount for identifying the individual is generated (S312).

  Next, when the transmission / reception unit 61 of the distributed data processing terminal 6 receives the partial image data and the partial image ID as the data to be collated by the processing in step S121 described above, the transmission / reception unit 61 sends the centralized data processing server 7 Then, collation request information indicating the collation request is transmitted (S313). The collation request information includes the collated data and the partial image ID. Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives the collation request information.

  Next, the centralized data processing server 7 performs a matching process (S314). Here, the collation processing will be described with reference to FIG. FIG. 31 is a sequence diagram illustrating the collation processing.

  As shown in FIG. 31, the feature amount generation unit 74 of the centralized data processing server 7 performs conversion for decoding the data to be verified received in step S313 into bitmap data, A parameter of a feature amount for identifying an individual such as the unevenness and inclination of the eyes and nose of the image is generated (S401).

  Next, the storage / readout unit 79 determines whether registered collation data remains by searching the collation data management DB 7001 (S402). If it is determined that the data remains, the matching unit 75 compares the parameters of the feature amounts of both data (the matching data and the data to be matched) to calculate the similarity (S403). Then, the storage / readout unit 79 temporarily stores the “name” attached to the collation data registered in step S312 and the “similarity” calculated in step S403 in the storage unit 7000 as a set. (S404). Thereafter, the collation data is changed to the next record in the collation data management table shown in FIG. 12, and the process of step S402 is performed again.

  On the other hand, if it is determined in step S402 that no matching data remains (including the case where there is no matching data), the process proceeds to step S405. Then, the determination unit 73 determines whether the maximum similarity among the similarities temporarily stored in the storage unit 7000 is larger than a threshold (S405).

  Then, in step S405, when the determining unit 73 determines that the maximum similarity is greater than the threshold (YES), the storage / read unit 79 converts the collation data management DB 7001 into the collation data having the maximum similarity. The attached "name" is read out (S409). The threshold is, for example, “80%”. As a result, when the similarity is low, the “name” is not read.

On the other hand, if the determination unit 73 determines in step S405 that the maximum similarity among the similarities temporarily stored in the storage unit 7000 is equal to or smaller than the threshold (NO), the storage / readout unit 79 reads information indicating “Unknown” from the storage unit 7000 (S407). Thus, the collation processing in step S314 is completed. Subsequently, returning to FIG. 30, the transmission / reception unit 71 of the centralized data processing server 7 transmits collation result information indicating the collation result to the distributed data processing terminal 6 ( S315). The collation result information includes the “name” read in step S406 or the information read in step S407, and the partial image ID received in step S313. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the collation result information.

  Next, in the distributed data processing terminal 6, the display control unit 67 causes the collation result as shown in FIG. 32 to be displayed (S316). Here, the display of the matching result will be specifically described with reference to FIG.

  The detectable range P is a range in which the imaging range of the imaging unit 40 can detect an object (here, a face). The detectable range P includes a collatable range Q in which an image of an object (here, a face) can be collated, and a non-collatable range R that is outside the collatable range Q and cannot be collated. In FIG. 32, a margin a and a margin b are predetermined values provided at the left and right ends in the x direction and the upper and lower ends in the y direction in the detectable range P, respectively. These values are stored in the service program acquired by the real-time data processing terminal 3a in step S97 shown in FIG.

  Thus, the reason that the collation range Q is provided separately from the detection range P is that the detection range P is fixed as an imaging range of the imaging element 401a or the like, so that it matches the size of an arbitrary room or area. This is because the object can be detected in the range in which the object is detected. In addition, when the object enters the detectable range P and then goes out of the detectable range P before entering the collation range Q, the real-time data processing terminal 3 a Since there is no need to transmit the data to be collated to the centralized data processing server 7 via the processing device 5 and the distributed data processing terminal 6, unnecessary communication charges can be reduced.

  The face detection by the processing in step S204 described above is performed over the entire detectable range P that is the photographing range. If the object is outside the rectangular range determined by the margins a and b, the object is not within the collatable range Q. In this case, the object is detected within the detectable range P, but not detected within the collatable range Q. As a result, the object is not collated in a range other than the collatable range Q within the detectable range P. This means that it exists in the possible range R.

  Here, (x1, y1) indicates the coordinates of the upper left corner of the partial image data when an object existing in the non-collating range R is detected.

  (X2, y2) indicates the coordinates of the upper left corner of the partial image data when an object existing in the collationable range Q is detected. Here, a standby state mark m1 represented by “!” Is displayed. This standby state mark m1 is displayed after the distributed data processing terminal 6 transmits the collation request information (S313) and receives the collation result information from the centralized data processing server 7 (S315).

  (X3, y3) indicates the coordinates of the upper left corner of the partial image data when an object existing in the collationable range Q is detected. Here, the name information m2 represented by “Kato” is displayed. This name information is the “name” received from the centralized data processing server 7 in step S315. As described above, the object in the collation range Q is displayed with the standby state mark m1 at first, but after 1-2 seconds, the name information m2 is displayed. This allows the viewer to recognize (identify) who the displayed object (here, the face) is. If the name is unknown, "Unknown" is displayed. This display is performed when the process of step 407 is executed and the “name” received in step S315 is “Unknown”.

<< Main effects of the embodiment >>
As described above, according to the present embodiment, when the same object is photographed by the real-time data processing terminal 3a, the real-time data processing terminal 3a transmits the same object through the near terminal data processing device 5 and the distributed data processing terminal 6. Thus, the transmission of the captured image data and the collation request to the centralized data processing server 7 is not performed. As a result, it is possible to prevent the user of the real-time data processing terminal 3a from being charged more than necessary for the communication fee.

Second Embodiment Next, a second embodiment of the present invention will be described with reference to FIGS. Note that the present embodiment is the same as the first embodiment except that the function of the calculation unit 31 is increased, and thus the description of the hardware configuration and the functional configuration other than the calculation unit 31 is omitted.

<< Functional Configuration of Embodiment >>
The calculating unit 31 further increases the count in order to count the number of persons who have entered the collatable range Q when the object once enters the collatable range Q and then exits from the collatable range Q. .

<< processing or operation of the present embodiment >>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIGS. The processing or operation of this embodiment will be described as a modification of the event generation processing of the first embodiment shown in FIG. FIG. 33 is a flowchart showing a modification of the process shown in FIG. FIG. 34 is a sequence diagram showing the processing of the display update request. FIG. 35 is a modification of the display example shown in FIG.

  In step S220 in FIG. 28, if there is a record in the object displacement management table to which the end flag is not added (S220; YES), the calculation unit 31 calculates the number of objects in the collation range Q. In order to count, the count is increased (S2201a). Then, the communication unit 48 of the real-time data processing terminal 3a transmits display update request information to the near terminal data processing terminal 5 (S2202a). The display update request information indicates a request for updating the image displayed on the display 517 of the distributed data processing terminal 6, and includes the count number incremented by the calculation unit 31. Thereby, the communication unit 58 of the near terminal data processing terminal 5 receives the display update request information.

  Next, the transmission / reception unit 51 of the near terminal data processing terminal 5 transmits the display update request information (including the count number) received by the communication unit 58 to the distributed data processing terminal 6 (S2203a). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the display update request information. Then, when the collation result as shown in FIG. 32 is displayed, the display control unit 67 of the distributed data processing terminal 6 further displays the count number m3 as shown in FIG. The display is changed so as to cause the display (S2204a). FIG. 35 shows that three persons have entered the collatable range Q from the count to the present.

<< Main effects of the embodiment >>
As described above, according to the present embodiment, as shown in FIG. 35, by displaying not only the collation result but also the total number of persons who have entered the collatable range Q, the effect of the first embodiment can be obtained. In addition to this, there is an effect that the usefulness of the display content can be improved.

Third Embodiment Next, a third embodiment of the present invention will be described with reference to FIGS.

  In the third embodiment, instead of the imaging units 40a and 40b and the real-time data processing terminal 3a shown in the first embodiment, in the present embodiment, the imaging units 40c and 40d and the real-time data processing terminal 3b is used. Therefore, the hardware configuration of the imaging units 40c and 40d and the functional configuration of the real-time data processing terminal 3b will be described below. Note that the hardware configuration of the real-time data processing terminal 3a is the same as that of the real-time data processing terminal 3b, and a description thereof will be omitted.

  In the present embodiment, a LiDAR (Light Detection and Ranging) technique is added to the imaging units 40a and 40b of the first embodiment shown in FIG. , The accuracy of object detection is increased. LiDAR is one of the remote sensing technologies using light, and is a technology that measures scattered light in response to laser irradiation that emits a pulsed light, and analyzes the distance to a distant object and the properties of the object.

<< hardware configuration >>
Next, the hardware configuration of the imaging unit of the present embodiment will be described with reference to FIG. FIG. 36A is a hardware configuration diagram of an imaging unit equipped with the LiDAR mechanism shown in FIG. 3A. FIG. 36B is a hardware configuration diagram of an imaging unit equipped with the LiDAR mechanism in FIG. 3B. Note that the imaging unit 40 is a general term for a plurality of types of imaging units (imaging units 40c and 40d) having different numbers of imaging elements.

  First, the imaging unit 40c shown in FIG. 36A will be described. As shown in FIG. 36A, the imaging unit 40c is to be electrically connected to an imaging element 401c such as a CMOS or a CCD, a lens 402c, and an imaging unit I / F 313 of the real-time data processing terminal 3b. Has a connection I / F 408c. When the imaging unit 40c is connected to the imaging unit I / F 313 of the real-time data processing terminal 3b, the imaging device 401c transmits an imaging control signal transmitted from the imaging unit I / F 313 via the connection I / F 408c. , And transmits the captured image data to the imaging unit I / F 313 via the connection I / F 408c.

  Further, the imaging unit 40c has a LiDAR mechanism 403c. The LiDAR mechanism 403c includes a light source 404c, a lens 405c, a light receiving element 406c, and a lens 407c. The light source 404c is configured by, for example, a laser diode. The LiDAR mechanism 403c can irradiate light at least within the detectable range P of the image sensor 401c.

  The light source 404c emits light toward a measurement target that is an object. The emitted light is near-infrared light having a wavelength range of 800 nm to 950 nm, which is an electromagnetic wave. Further, the light source 404c is provided with a drive circuit that boosts the voltage supplied from the real-time data processing terminal 3b through the connection I / F 408c to a specified voltage. This drive circuit generates an oscillation signal for emitting light from the light source 404c. The light source 404c periodically emits short pulse light having a pulse width of about several nanoseconds to several hundred nanoseconds as modulated light by an oscillation signal.

  The lens 405c allows the light emitted from the light source 404c to pass therethrough and controls the state such as the emission direction and the emission angle of the emitted light. The lens 405c collimates the light emitted from the light source 404c into parallel light (including substantially parallel light). For this reason, the LiDAR mechanism 403c can perform distance measurement even on a minute region to be detected.

  The light receiving element 406c is configured using various photodiodes such as a silicon PIN (P-Intrinsic-N) photodiode and an avalanche photodiode (APD). Further, the light receiving element 406c is provided with a light receiving signal amplifying circuit for amplifying a received signal. The light receiving signal amplifying circuit amplifies the electric signal output from the light receiving element 406c and sends it out as a reflected signal to the real-time data processing terminal 3b via the connection I / F 408c.

  Of the light emitted from the light source 404c, the light that passes through the lens 407c and is reflected by the measurement object (hereinafter, referred to as “reflected light”) is received and converted into an electrical signal, and is converted via the connection I / F 408c. An electric signal is transmitted to the real-time data processing terminal 3b. Here, the reflected light is one in which near-infrared light, which is an electromagnetic wave emitted from the light source 404c, is reflected by a measurement target (reflected wave).

  The lens 407c allows the reflected light to pass through and controls the state such as the incident direction and the incident angle of the reflected light.

  Next, the imaging unit 40d shown in FIG. 36B will be described. As shown in FIG. 36B, the imaging unit 40d is electrically connected to imaging elements 401d1 and 401d2 such as CMOS and CCD, lenses 402d1 and 402d2, and an imaging unit I / F 313 of the real-time data processing terminal 3b. Has a connection I / F 408b for connecting to the.

  Further, the imaging unit 40d has LiDAR mechanisms 403d1 and 403d2. The LiDAR mechanism 403d1 includes a light source 404d1, a lens 405d1, a light receiving element 406d1, and a lens 407d1. The LiDAR mechanism 403d1 is provided on the image sensor 401d1 side, and can irradiate light at least in the detectable range P of the image sensor 401d1 within the detectable range of the image sensor 401d1.

  Note that the light source 404d1, the lens 405d1, the light receiving element 406d1, and the lens 407d1 have the same configuration as the light source 404c, the lens 405c, the light receiving element 406c, and the lens 407c, respectively, and thus the description is omitted.

  The LiDAR mechanism 403d2 includes a light source 404d2, a lens 405d2, a light receiving element 406d2, and a lens 407d2. The LiDAR mechanism 403d2 is provided on the image sensor 401d2 side, and can irradiate light at least in the detectable range P of the image sensor 401d2 within the detectable range of the image sensor 401d2.

  Note that the light source 404d2, the lens 405d2, the light receiving element 406d2, and the lens 407d2 have the same configuration as the light source 404c, the lens 405c, the light receiving element 406c, and the lens 407c, respectively, and thus description thereof is omitted.

  Next, the functional configuration and processing of the present embodiment will be described with reference to the drawings.

<< Functional Configuration of Embodiment >>
Subsequently, a functional configuration of the real-time data processing terminal 3b of the present embodiment will be described with reference to FIG. FIG. 37 is a modification of the real-time data processing terminal in the functional block diagram of the communication system.

  The real-time data processing terminal 3b of the present embodiment further includes an issuance control unit 41 and a distance measurement unit 42 with respect to the first real-time data processing terminal 3a. Only the functions of the unit 42 will be described, and descriptions of the other functional configurations will be omitted. Note that the issuance control unit 41 and the distance measurement unit 42 operate according to an instruction from the CPU 301 in accordance with a program expanded from the EEPROM 304 onto the RAM 303 by using any one of the components shown in FIG. Is a function or means realized by.

  The light emission control unit 41 controls light emission of the light source 404c (or the light sources 404d1 and 404d2).

  The distance measurement unit 42 stops the time measurement process started from the time when the signal was generated by the above-described drive circuit when the signal converted from the reflected light is generated, until the emitted signal is received. By measuring the time, the distance from the imaging unit 40c to the object to be measured is measured. That is, the distance from the light emission from the light source 404c to the reception of the reflected light of the light is used to measure the distance from the image acquisition terminal 2 including the imaging unit 40c to the object to be measured.

  With the above configuration, when the imaging unit 40c is connected to the real-time data processing terminal 3b, the light modulated by the light emission control unit 41 and emitted from the light source 404c is a light having a small divergence angle via the lens 405c. It becomes a beam. The light beam emitted from the light source 404c is applied to an object (for example, a face) as a measurement target. The light beam applied to the measurement target becomes reflected light that is scattered and reflected in a uniform direction at the reflection point of the measurement target. Of the reflected light, only the light component reflected through the same optical path as the light beam irradiated on the measurement target enters the light receiving element 406c via the lens 407c arranged substantially coaxially with the light source 404c. The reflected light that has entered the light receiving element 406c is detected as a reflected signal by the light receiving element 406c. Thus, the distance measurement unit 42 can calculate the distance to the measurement target by measuring the time difference between the emission time of the light source 404c and the reception time of the reflected light.

  When the imaging unit 40d is connected to the real-time data processing terminal 3b, the light modulated by the light emission control unit 41 and emitted from the light sources 404d1 and 404d2 has a small spread angle through the lenses 405d1 and 405d2, respectively. Light beam. The light beam emitted from the light source 404c is applied to an object (for example, a face) as a measurement target. The light beam applied to the measurement target becomes reflected light that is scattered and reflected in a uniform direction at the reflection point of the measurement target. Of the reflected light, only light components reflected through the same optical path as the light beam irradiated on the measurement target are respectively received by the light receiving element 406d1 via the lenses 407d1 and 407d2 arranged substantially coaxially with the light sources 404d1 and 404d2. , 406d2. The reflected lights entering the light receiving elements 406d1 and 406d2 are detected as reflected signals by the light receiving elements 406d1 and 406d2, respectively. Accordingly, the distance measurement unit 42 can calculate the distance to the measurement target by measuring the time difference between the emission time of the light sources 404d1 and 404d2 and the reception time of the reflected light.

<< processing or operation of the present embodiment >>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIGS. Note that the processing or operation of this embodiment will be described as a modification of the event generation processing of the first embodiment shown in FIG. FIG. 38 is a flowchart showing a modified example of the process shown in FIG. FIG. 39 is a modification of the display example shown in FIG. FIG. 40 is a modification of the display example shown in FIG. 32, and is a display example in which the LiDAR function is not used together. FIG. 41 is a modification of the display example shown in FIG. 32, and is a display example in the case of using the LiDAR function together. Here, the case where the imaging unit 40c is connected to the real-time data processing terminal 3b will be described. However, the same processing is performed when the imaging unit 40d is connected to the real-time data processing terminal 3b. Omitted.

  In step S220 in FIG. 28, if there is a record in the object displacement management table to which the end flag is not added (S220; YES), the light emission control unit 41 emits light to the light source 404c of the imaging unit 40. The control is performed (S2201b). Then, the distance measurement unit 42 measures the time until the emitted signal is received, thereby measuring the distance from the imaging unit 40c to the object to be measured (S2202b).

  Next, based on the measurement result by the distance measurement unit 42, the determination unit 33 determines whether one object is recognized by the object detection unit 35 and determines whether there is a predetermined distance difference. Is determined (S2203b). The predetermined distance difference is, for example, 20 cm. Since the distance between the tip of the nose and the cheek is several centimeters, if the difference is about this degree, the determination unit 33 determines that there is one object (here, face) (S2203b; YES). . For example, as shown in FIG. 39, each object is recognized. Here, a face of a man and a woman represented by partial image data in which the coordinates of the upper left corner are (x2, y2) and (x4, y4) are shown in the collationable range Q. Here, the men and women are present at separate positions within the detectable range P. In this case, the process proceeds to step S221.

  However, as shown in FIG. 40, when two men and women overlap (for example, a woman is positioned behind a man), and the object detection unit 35 detects the man and woman as one object, Can not count the number of people. Here, a face in which men and women represented by partial image data whose coordinates of the upper left corner are (x5, y5) overlap and are integrated within the collationable range Q is shown.

  Therefore, in the present embodiment, even if the object detection unit 35 detects the object as one object, if there is a predetermined distance difference between the objects detected as one, the objects are regarded as overlapping, and It is determined that (here, the face) is not one (a plurality of faces) (S2203b; NO). Here, the faces of men and women represented by the partial image data in which the coordinates of the upper left corner are (x2, y2) and (x6, y6) are shown in the collationable range Q, respectively. Here, men and women are present at overlapping positions within the detectable range P. In this case, the process proceeds to step S231 shown in FIG. 29 without executing the process of step S221.

  In this way, at the timing at which the determination unit 33 determines that the object has fallen out of the collation range Q, the processing of step S221 is not immediately performed, and the LiDAR technology is used, so that the object is really out of the collation range Q. It is determined whether or not it has been removed.

  With this, when a difference of a predetermined distance or more occurs between the overlapped male and female (second object) at the present time (second time point), it is determined that the second object is a plurality of objects, and the ) Is at least one of a predetermined distance or less and a distance between the current position of the plurality of objects and the position of the object in the past (first time point) of each of the plurality of objects, The image data of the predetermined portion relating to at least one of the male and female is not transmitted.

  In addition, to save energy, after step S220, the real-time data processing terminal 2b performs light emission control (see S2201b) and distance calculation (see S2202b). However, LiDAR technology may be used before the processing in step S220.

  Although the count number m3 is displayed in FIGS. 39 to 41 as in the second embodiment, the count number m3 may not be displayed as in the first embodiment.

<< Main effects of the embodiment >>
As described above, according to the present embodiment, even if the determination unit 33 once determines that the object has fallen out of the collatable range Q by the image processing, it is possible to determine the overlap of the objects using the LiDAR technology. Thus, an effect is obtained that the object authentication can be performed more accurately than in the first embodiment.

[Supplement]
In the above embodiment, in steps S217 and S218, only the distance between the detection position of the latest object and the detection position of the past object is used as a criterion, but the present invention is not limited to this. For example, the area difference between the partial image of the latest object and the partial image of the past object may be used as a judgment material. Specifically, when the following two conditions (condition 1) and (condition 2) are satisfied, the process may proceed to step S219. According to this determination, even if the current object detection position is closer to the past detection position, if the size of the detection region is unnaturally different, it is not determined that the objects are the same object (here, a person). It can be. As a result, transmissions (S115, S121) for which a collation request should not be necessary can be further reduced.
(Condition 1) The distance between the past object and the latest object is less than (or less than) a predetermined threshold (same as step S218)
(Condition 2) The absolute value of the area difference between the past object and the latest object is less than (or less than) a predetermined threshold value
The area difference under the condition 2 is a difference between the areas (width * height) calculated by the calculation unit 31 from the height and the width of the partial image.

  Further, in the above embodiment, the standby state mark m1 is represented by “!”, But is not limited to this. For example, it may be an hourglass icon or a comment such as "verifying" or "verifying".

  In addition, the captured image data has been described, but the invention is not limited thereto, and may be created image data created by a user without being photographed. In this case, the captured image data and the created image data are examples of “image data”. Also. The image acquisition terminal 2 is an example of a communication terminal, and the image acquisition terminal 2 may create the image data without acquiring the image data from the outside. Furthermore, the communication terminal may acquire sound data by sound collection, acquire temperature data by a temperature sensor, or acquire humidity data by a humidity sensor.

  The components such as the CPUs 301, 501, and 701 may be single or plural. Further, there may be a plurality of image acquisition terminals 2, distributed data processing terminals 6, and centralized data processing servers 7, respectively. Further, the distributed data processing terminal 6 may be a server.

Further, each function in the above embodiment can be realized by one or a plurality of processing circuits. Here, the `` processing circuit '' in the present embodiment is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or an ASIC designed to execute each function described above ( Application Specific Integrated
Circuit), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a GPU, and devices such as conventional circuit modules.

[Others]
This embodiment discloses the following invention.

[Appendix 1] (see S218)
A communication terminal that transmits image data of a predetermined portion of an object to cause another device to perform image collation, and transmits image data of a predetermined portion of the first object at a first time to the other device. Transmitting means for transmitting, the first position indicating the position of the first object at the first time, and the position of the second object at a second time after a predetermined time from the first time The communication terminal according to claim 1, wherein the transmission unit does not transmit image data of a predetermined portion of the second object when a distance between the second object and the second position is equal to or less than a predetermined value or less than a predetermined value.

[Appendix 2] (see S220, S221)
When the position of the second object at the second time point is out of the collationable range, the transmitting unit does not transmit image data of a predetermined portion of the second object. The communication terminal according to claim 1.

[Appendix 3] (see S231 and S232)
When the position of the first object at the first time point falls within a collationable range, the transmitting unit transmits image data of a predetermined portion of the first object. Item 3. The communication terminal according to item 1 or 2.

[Appendix 4] (see S234)
4. The communication terminal according to claim 1, wherein the transmitting unit transmits the image data obtained by encoding data of a predetermined portion of the object. 5.

2 Image acquisition terminal (example of communication terminal)
3a, 3b Real-time data processing terminal 5 Near-terminal data processing device 6 Distributed data processing terminal (an example of a display terminal)
7. Centralized data processing server (an example of another device)
31 Calculation unit (an example of calculation means)
34 Image processing unit (an example of image processing means)
35 Object detection unit (an example of object detection means)
38 Connection part (an example of acquisition means)
51 Transmission / Reception Unit (Example of Transmission Means)
61 transmitting / receiving section 62 receiving section 67 display controlling section 69 storage / reading section 71 transmitting / receiving section 74 feature quantity generating section 75 collating section 200 intranet 600 internet 3001 image sensor information management DB (an example of an image sensor information managing means)
3002 Cycle value management DB
3003 Image acquisition program DB
3004 Synthesis processing program management DB
3005 Distortion correction program management DB
3006 Service program management DB (an example of program management means)
3007 Object information management DB (an example of object information management means)
3008 Object displacement management DB (an example of object displacement management means)
6000 storage unit 7001 collation data management DB
8000 Storage unit 8001 Session management DB
8002 Authentication server management DB
9000 Storage unit 9001 Authentication management DB

JP-A-2017-204825

Claims (9)

A communication terminal that transmits image data of a predetermined portion of the object in order for another device to perform image matching,
Transmitting means for transmitting image data of a predetermined portion of the first object at a first time point so that the other device receives the image data;
Between a first position indicating the position of the first object at the first time and a second position indicating the position of the second object at a second time after a predetermined time from the first time When the distance is less than or equal to a predetermined value or less than a predetermined value, the transmitting unit does not transmit image data of a predetermined portion of the second object.   When the position of the second object at the second time point is out of the collationable range, the transmitting unit does not transmit image data of a predetermined portion of the second object. The communication terminal according to claim 1.   When the position of the first object at the first point in time falls within a collationable range, the transmission unit transmits image data of a predetermined portion of the first object. Item 3. The communication terminal according to item 1 or 2. The communication terminal according to claim 1, wherein:
Computing means for counting the number of times the object has entered the collation possible range,
The communication terminal according to claim 1, wherein the transmitting unit transmits the count number of the calculating unit so that the display terminal that displays the result of the object matching receives the count number. The communication terminal according to claim 1, wherein:
Light emission control means for controlling light emission for the light source;
A distance measuring unit that measures a distance from the communication terminal to the object by a time from when light is emitted from the light source 404 under the control of the light emission control unit to when a reflected light of the light is received,
Has,
When a difference of a predetermined distance or more occurs in the second object at the second time, the second object is regarded as a plurality of objects, and the positions of the plurality of objects at the second time are When at least one of the distances to the position of the object at the first time point of each of the plurality of objects is equal to or less than a predetermined value or less than a predetermined value, the transmitting unit transmits the second object related to the at least one of the plurality of objects. A communication terminal not transmitting image data of a predetermined portion.   The communication terminal according to claim 1, wherein the transmission unit transmits the image data obtained by encoding data of a predetermined portion of the object. A communication terminal according to any one of claims 1 to 6,
The other device,
A communication system having: A communication method performed by a communication terminal that transmits image data of a predetermined portion of an object in order for another device to perform image matching,
The communication terminal,
Performing a transmitting step of transmitting image data of a predetermined portion of the first object at a first time point so that the other device receives the image data;
Between a first position indicating the position of the first object at the first time and a second position indicating the position of the second object at a second time after a predetermined time from the first time If the distance is less than or equal to or less than a predetermined value, the transmitting step includes a process of not transmitting image data of a predetermined portion of the second object.   A program for causing a computer to execute the method according to claim 8.