JP7310105B2 - Communication terminal, communication system, image processing method, and program - Google Patents

Description

The present disclosure relates to communication terminals, communication systems, image processing methods, and programs.

In recent years, IoT (Internet of Things) has attracted attention. Among them, communication systems for edge computing are particularly attracting attention. In this communication system, the terminal side acquires image and voice data, transmits the data to a server via the Internet, and performs recognition processing on the server side. In Patent Document 1, a camera for face recognition is installed on the terminal (edge node) side for the purpose of executing processing for recognizing a person's face, and the terminal transmits image data acquired from the camera to a central server (face recognition A cloud service platform is disclosed that transmits to a facial recognition application server) and performs face authentication processing on the server.

Also, in order to realize a series of image processing, the terminal executes a program for executing processing only on the terminal side and a program for executing processing in cooperation with the server side. In addition, a series of image processing may include programs and services that require consideration of privacy, and authentication procedures are required when adding a new camera or system.

However, the series of image processing includes processes that do not require authentication, so as new cameras and services were added, authentication procedures were required to acquire all programs. Therefore, there is a problem that it takes time to start a series of image processing.

The invention according to claim 1 is a communication terminal that realizes a service provided by a service providing server through cooperation between processing in a service providing server that provides services via a communication network and processing in the terminal itself, , to perform the first image processing on the image data by executing a first program that does not require authentication for executing the processing in the own terminal, and to execute the processing in the own terminal; The terminal has image processing means for performing second image processing on the image data by executing a second program that requires authentication, and the terminal is configured to implement the service to be provided. a search unit for searching whether the first program and the second program exist in the computer; a determination unit for determining the existence of the first and second programs based on the search result; a first processing unit; and two processing units, and when the determination unit determines that at least one program among the first and second programs does not exist in the first processing unit, the first processing unit request information indicating a request for the nonexistent program is transmitted to the second processing unit, the second processing unit transmits the received request information so that the service providing server receives the request information, The processing unit receives a program based on the request information transmitted by the service providing server, transmits the received program to the first processing unit, and the first processing unit receives the received program. A communication terminal characterized by installing the program .

As described above, according to the present invention, it is possible to reduce the time required to start a series of image processing.

1 is a schematic diagram of a communication system according to an embodiment; FIG. 3 is a hardware configuration diagram of a real-time data processing terminal; FIG. 3 is a hardware configuration diagram of an imaging unit; FIG. FIG. 4(a) is a hemispherical image (front side) taken by the imaging unit 40b, FIG. 4(b) is a hemispherical image (back side) taken by the imaging unit 40b, and FIG. 4(c) is an equirectangular projection. FIG. 10 is a diagram showing a rendered image; 2 is a hardware configuration diagram of a near-terminal data processing device or a distributed data processing terminal; FIG. 3 is a hardware configuration diagram of a centralized data processing server; FIG. 3 is a software configuration diagram of a real-time data processing terminal and a near-terminal data processing device in an image acquisition terminal; FIG. 1 is a functional block diagram of a communication system; FIG. 1 is a functional block diagram of a communication system; FIG. (a) is a conceptual diagram of an image sensor information management table, and (b) is a conceptual diagram of a cycle value management table. (a) is a conceptual diagram of an image acquisition program management table, (b) is a conceptual diagram of a composition processing program management table, (c) is a conceptual diagram of a distortion correction program management table, and (d) is a conceptual diagram of a service program management table. is. 4 is a conceptual diagram of a matching data management table; FIG. 4 is a conceptual diagram of a session management table; FIG. 4 is a conceptual diagram of terminal IDs; FIG. 4 is a conceptual diagram of an authentication server management table; FIG. 4 is a conceptual diagram of an authentication management table; FIG. 4 is a sequence diagram showing authentication processing; FIG. 4 is a sequence diagram showing authentication processing; FIG. It is a figure which shows the example of a screen of a distributed data processing terminal. It is a figure which shows the example of a screen of a distributed data processing terminal. FIG. 10 is a sequence diagram showing processing of an image recognition start request; FIG. 10 is a sequence diagram showing preparatory processing for real-time processing of the real-time data processing terminal; FIG. 10 is a sequence diagram showing program acquisition processing; 4 is a sequence diagram showing image recognition processing; FIG. 10 is a flowchart showing object detection processing in real-time processing. FIG. 10 is a flowchart showing a first example of event generation processing in real-time processing; FIG. FIG. 10 is a diagram showing a display example of a captured image in a distributed data processing terminal; FIG. 10 is a sequence diagram showing a first example of matching processing; (a) is a display example of a verification data file selection screen in the distributed data processing terminal, and (b) is a display example of a verification data registration screen in the distributed data processing terminal. It is a display example of a collation result message in a distributed data processing terminal. FIG. 4 is a diagram showing a situation in which an image acquisition terminal is arranged in a certain room; FIG. 10 is a sequence diagram showing a second example of event generation processing in real-time processing; FIG. 11 is a sequence diagram showing a second example of matching processing;

[Overall overview]
An overall outline of an embodiment of the present invention will be described below with reference to the drawings. This embodiment discloses a communication system that implements edge computing. That is, the present embodiment discloses an invention in which the service provided by the service providing server 2 is realized by the image obtaining terminal 2 through cooperation between the processing in the service providing server 8 and the processing in the image obtaining terminal 2 .

<<Overall configuration of the embodiment>>
FIG. 1 is a schematic diagram of a communication system according to this embodiment. As shown in FIG. 1, the communication system of this embodiment includes a real-time data processing terminal 3, a near-terminal data processing device 5, a distributed data processing terminal 6, a centralized data processing server 7, a service providing server 8, and It is constructed by authentication servers 9a, 9b and 9c. A distributed processing system 100 is constructed by the real-time data processing terminal 3 , the near-terminal data processing device 5 , and the distributed data processing terminal 6 . Although there are a plurality of distributed processing systems 100, only one is shown in FIG. 1 to simplify the drawing.

The near terminal data processing device 5 is communicably connected to the distributed data processing terminal 6 via the intranet 200 . The distributed data processing terminal 6 is communicably connected to the centralized data processing server 7, the service providing server 8, and the authentication servers 9a, 9b, and 9c via the Internet 600. FIG. The authentication server 9 is a general term for the authentication servers 9a, 9b, and 9c.

Among them, the real-time data processing terminal 3 is a terminal that performs real-time processing (real-time processing) for obtaining captured image data. The real-time data processing terminal 3 is detachably connected to an imaging unit 40 having an image sensor for imaging an object, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. As a result, the real-time data processing terminal 3 digitizes the captured image data (hereinafter referred to as "captured image data") input from the imaging unit 40, and processes the object (herein Now, let's detect the image of the face). The real-time data processing terminal 3 transmits data of a partial image (hereinafter referred to as “partial image data”), which is the area of the object portion in the captured image, to the near-terminal data processing device 5 . In addition, although the face is mentioned as a target for detecting the image of the object, there are a wide variety of targets such as the human head, upper body, and lower body, and the object is not limited to people, but also cars and animals. . In the case of a car, the driver, fellow passenger, license plate, and in the case of an animal, the pattern of the face and body can be used.

The near-terminal data processing device 5 is arranged closest to the real-time data processing terminal 3, and is closely connected to the real-time data processing terminal 3 on a one-to-one basis, for example, through a data bus, USB (Universal Serial Bus), or the like. The near-terminal data processing device 5 encodes the partial image data received from the real-time data processing terminal 3 into a general-purpose format such as JPEG (Joint Photographic Experts Group), and then converts the data into a target image for face image matching. It is transmitted to the distributed data processing terminal 6 via the intranet 200 as collation data. Note that the real-time data processing terminal 3 and the near-terminal data processing device 5 are usually connected and integrated. Configure.

The distributed data processing terminal 6 is a computer used by a user at a position relatively close to the near-terminal data processing device 5 and capable of receiving various user operations. The distributed data processing terminal 6 registers and stores in advance verification data for verification of face images. The distributed data processing terminal 6 can request the centralized data processing server 7 via the Internet 600 to match the matching data with the data to be matched. In this case, the distributed data processing terminal 6 transmits to the centralized data processing server 7 the to-be-verified data received from the near-terminal data processing device 5 and the previously registered verification data. Furthermore, the distributed data processing terminal 6 can receive matching result information indicating the matching result from the centralized data processing server 7 as a response. Also, the distributed data processing terminal 6 can display the collation result via a graphic interface.

The centralized data processing server 7 is installed at a position relatively far from the near-terminal data processing device 5, and can communicate with the distributed data processing terminals 6 via a communication network such as the Internet 600 or the like. When the centralized data processing server 7 receives matching request information indicating a matching request and matching data and data to be matched, the centralized data processing server 7 compares the matching data and the data to be matched, and calculates the degree of similarity. The centralized data processing server 7 then transmits collation result information indicating the collation result including the calculated similarity to the distributed data processing terminal 6 . The matching result information includes, for example, the device identification ID of the distributed data processing terminal 6 that sent the data, the device identification ID of the image processing terminal 2 that sent the data to be checked, the calculated similarity, the calculated date and time, and the It includes information such as the name and number of the person who received it.

The service providing server 8 is a server that provides various services to the image acquisition terminal 2 .

The authentication server 9a performs ID authentication to determine whether the image acquisition terminal 2 is authorized to receive services from the service providing system 8 or not. The same applies to the authentication servers 9b and 9c.

<<Hardware configuration>>
Next, each hardware configuration of the communication system of this embodiment will be described with reference to FIGS. 2 to 6. FIG.

<Hardware configuration of real-time data processing terminal>
FIG. 2 is a hardware configuration diagram of a real-time data processing terminal. The real-time data processing terminal 3 includes a CPU 301 , a ROM 302 , a RAM 303 , an EEPROM 304 , a CMOS sensor 305 , an acceleration/direction sensor 306 , a media I/F 308 and a GPS receiver 309 .

Among them, the CPU 301 controls the operation of the real-time data processing terminal 3 as a whole. The ROM 302 stores programs used to drive the CPU 301 . A RAM 303 is used as a work area for the CPU 301 . Under the control of the CPU 301, the EEPROM 304 reads or writes various data such as programs for real-time data processing terminals. The CMOS sensor 305 captures an image of a subject (mainly a blind spot of the imaging unit 40) under the control of the CPU 301 to obtain captured image data. The acceleration/azimuth sensor 306 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. A media I/F 308 controls reading or writing (storage) of data to a recording medium 307 such as a flash memory. A GPS receiver 309 receives GPS signals from GPS satellites.

The real-time data processing terminal 3 also includes an imaging unit I/F 313 , a microphone 314 , a speaker 315 , a sound input/output I/F 316 , a display 317 , an external device connection I/F 318 and a touch panel 321 .

Among these, the image pickup unit I/F 313 is a circuit that controls driving of the image pickup unit 40 when the external image pickup unit 40 is connected. The microphone 314 is a type of built-in sound collecting means for inputting sound. A sound input/output I/F 316 is a circuit that processes input/output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301 . A display 317 is a kind of display means such as a liquid crystal or an organic EL that displays an image of a subject, various icons, and the like. The external device connection I/F 318 is an interface for connecting various external devices. The touch panel 321 is a kind of input means for operating the real-time data processing terminal 3 when the user presses the display 317 .

The real-time data processing terminal 3 also has a bus line 310 . A bus line 310 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 shown in FIG.

<Hardware configuration of imaging unit>
FIG. 3 is a hardware configuration diagram of the imaging unit. In particular, FIG. 3A is a hardware configuration diagram of a monocular imaging unit 40a among the imaging units 40. FIG. FIG. 3B is a hardware configuration diagram of a compound-eye imaging unit 40 b of the imaging unit 40 . Note that the imaging unit 40 is a generic term for a plurality of types of imaging units (imaging units 40a, 40b, etc.) that differ in the number of imaging elements or the type of lens.

As shown in FIG. 3A, the imaging unit 40a is electrically connected to an imaging device 401a such as a CMOS or CCD, a lens 402a, and an imaging unit I/F 313 of the real-time data processing terminal 3. connection I/F 408a. When the imaging unit 40a is connected to the imaging unit I/F 313 of the real-time data processing terminal 3, the imaging element 401a receives the imaging control signal sent from the imaging unit I/F 313 via the connection I/F 408a. , and transmits captured image data to the imaging unit I/F 313 via the connection I/F 408a.

In addition, as shown in FIG. 3B, the imaging unit 40b includes imaging elements 401b1 and 401b2 such as CMOS and CCD, lenses 402b1 and 402b2, and an imaging unit I/F 313 of the real-time data processing terminal 3. It has a connection I/F 408b for electrical connection. The lenses 402b1 and 402b2 are, for example, fisheye lenses. When the image pickup unit 40a is connected to the image pickup unit I/F 313 of the real-time data processing terminal 3, the image pickup elements 401b1 and 401b2 receive images sent from the image pickup unit I/F 313 via the connection I/F 408a. An image is captured by a control signal, and a plurality of captured image data captured by a plurality of image sensors are transmitted to the image capturing unit I/F 313 via the connection I/F 408b.

The image pickup unit 40a shown in FIG. 3(a) obtains a general planar image, while the image pickup unit 40b shown in FIG. If the image pickup device is arranged so as to capture the Here, the outline of the process until the equirectangular projection image EC is created from the image captured by the imaging unit 40b will be described with reference to FIG. 4(a) is a hemispherical image (front side) taken by the imaging unit 40b, FIG. 4(b) is a hemispherical image (rear side) taken by the imaging unit 40b, and FIG. 4(c) is an equirectangular image. FIG. 2 is a diagram showing an image represented by projection (hereinafter referred to as an “equidrectangular projection image”).

As shown in FIG. 4A, the image obtained by the imaging element 401b1 is a hemispherical image (front side) curved by the lens 402b1. Also, as shown in FIG. 4B, the image obtained by the imaging element 401b2 is a hemispherical image (rear side) curved by the lens 402b2. Then, the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) are synthesized to create an equirectangular projection image EC as shown in FIG. 4(c). Here, in order to obtain such a hemispherical image, an image such as that obtained in FIG. be able to.

<Hardware Configuration of Near Terminal Data Processing Device and Distributed Data Processing Terminal>
FIG. 5 is a hardware configuration diagram of a near-terminal data processing device or a distributed data processing terminal. Here, since the near-terminal data processing device 5 and the distributed data processing terminal 6 have the same hardware configuration, the near-terminal data processing device 5 will be explained first, and then the distributed data processing terminal 6 will be explained. are omitted.

As shown in FIG. 5 , the near-terminal data processing device 5 includes a CPU 501 , ROM 502 , RAM 503 , EEPROM 504 , CMOS sensor 505 , acceleration/direction sensor 506 , media I/F 508 and GPS receiver 509 .

Among these, the CPU 501 controls the operation of the near-terminal data processing device 5 as a whole. The ROM 502 stores programs used to drive the CPU 501 . A RAM 503 is used as a work area for the CPU 501 . Under the control of the CPU 501, the EEPROM 504 reads out or writes various data such as a near-terminal data processing device program. Under the control of the CPU 501, the CMOS sensor 505 captures an image of a subject (mainly a self-portrait of the user operating the near-terminal data processing device 5) to obtain captured image data. The acceleration/direction sensor 506 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. A media I/F 508 controls reading or writing (storage) of data to a recording medium 507 such as a flash memory. A GPS receiver 509 receives GPS signals from GPS satellites.

In addition, the near-terminal data processing device 5 includes a long-distance communication circuit 511, an antenna 511a of the long-distance communication circuit 511, a camera 512, an image sensor I/F 513, a microphone 514, a speaker 515, a sound input/output I/F 516, a display 517, An external device connection I/F 518 , a short-range communication circuit 519 , an antenna 519 a of the short-range communication circuit 519 , and a touch panel 521 are provided.

Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via the intranet 200 . The camera 512 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 and obtains captured image data. An imaging device I/F 513 is a circuit that controls driving of the camera 512 . The microphone 514 is a type of built-in sound collecting means for inputting sound. A sound input/output I/F 516 is a circuit for processing input/output of sound signals between the microphone 514 and the speaker 515 under the control of the CPU 501 . A display 517 is a kind of display means such as a liquid crystal or an organic EL that displays an image of a subject, various icons, and the like. The external device connection I/F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as NFC (Near Field radio Communication) or Bluetooth (registered trademark). The touch panel 521 is a kind of input means for operating the near-terminal data processing device 5 by the user pressing the display 517 .

The near-terminal data processing device 5 also has a bus line 510 . A bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

<Hardware configuration of centralized data processing server, service providing server, and authentication server>
FIG. 6 is a hardware configuration diagram of the centralized data processing server, service providing server, or authentication server. Here, since the centralized data processing server 7, the service providing server 8, and the authentication server 9 have the same hardware configuration, the centralized data processing server 7 will be explained, and the service providing server 8 and the authentication server 9 will be described. A description of the server 9 is omitted.

FIG. 6 is a hardware configuration diagram of the centralized data processing server. The centralized data processing server 7 is constructed by a computer, and as shown in FIG. 708 , network I/F 709 , keyboard 711 , mouse 712 , drive 714 compatible with CD-RW, DVD, and Blu-ray (registered trademark) discs, and bus line 710 . Since the centralized data processing server 7 functions as a server, input devices such as the keyboard 711 and mouse 712 and output devices such as the display 708 may be omitted.

Among these, the CPU 701 controls the operation of the centralized data processing server 7 as a whole. A ROM 702 stores programs used to drive the CPU 701 . A RAM 703 is used as a work area for the CPU 701 . The HD 704 stores various data such as programs. The HDD 705 controls reading and writing of various data to and from the HD 704 under the control of the CPU 701 . A media I/F 707 controls reading or writing (storage) of data to a recording medium 706 such as a flash memory. A display 708 displays various information such as a cursor, menus, windows, characters, or images. A network I/F 709 is an interface for data communication using the Internet 600 . The keyboard 711 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. A mouse 712 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. A drive 714 compatible with CD-RW, DVD, and Blu-ray discs reads various data from a CD-RW (Compact Disc-ReWritable), DVD, and Blu-ray disc 713 as an example of a removable recording medium. etc.

The centralized data processing server 7 also has a bus line 710 . A bus line 710 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 701 shown in FIG. Thus, the service providing server 8 and the authentication server 9 also have similar hardware configurations.

<<Software configuration>>
FIG. 7 is a software configuration diagram of the real-time data processing terminal and near-terminal data processing device in the image acquisition terminal 2 (communication terminal).

As shown in FIG. 7, the real-time data processing terminal 3 has an OS 300 and an image recognition application (hereinafter "application" is referred to as "application") AP1. The image recognition application AP1 operates on the work of the RAM 303 of the real-time data processing terminal 3. FIG. Of these, the OS 300 is basic software that provides basic functions and manages the real-time data processing terminal 3 as a whole. The image recognition application AP1 is an application for recognizing the face of a person, an animal, or the like from a captured image.

Also, the near-terminal data processing device 5 has an OS 500 and a communication application AP2. The communication application AP2 operates on the work of the RAM 503 of the near-terminal data processing device 5 . Of these, the OS 500 is basic software that provides basic functions and manages the entire near-terminal data processing device 5 . The communication application AP2 is an application for communicating with other terminals (apparatuses) such as the distributed data processing terminal 6 .

As described above, in the image acquisition terminal 2, the real-time data processing terminal 3 performs image recognition, while the near-terminal data processing device 5 communicates with the distributed data processing terminal 6 via the intranet 200. Distributed processing can be performed.

Note that the real-time data processing terminal 3 and the near-terminal data processing device 5 have not only an OS but also a driver, a software development kit (SDK), or an application programming interface (API). good.

Next, the functional configuration and processing of one embodiment of the present invention will be described with reference to the drawings.

<<Function Configuration of Embodiment>>
First, functional configurations of terminals, devices, and servers that configure the communication system according to the present embodiment will be described with reference to FIGS. 8 to 16. FIG. FIG. 8 is a functional block diagram of the communication system according to the embodiment; FIG. 8 particularly shows functional blocks of the image acquisition terminal 2 .

<Functional configuration of real-time data processing terminal>
As shown in FIG. 8, the real-time data processing terminal 3 includes a determination unit 33, an image processing unit 34, an event generation unit 36, a display control unit 37, a connection unit 38, a storage/readout unit 39, and a communication unit. 48. Each of these units is a function or means implemented by any one of the components shown in FIG.

The real-time data processing terminal 3 also has a storage unit 3000 constructed by the ROM 302, RAM 303, and EEPROM 304 shown in FIG. The storage unit 3000 stores geometric model data, which will be described later. Furthermore, in the storage unit 3000, an imaging device information management DB 3001, a cycle value management DB 3002, an image acquisition program management DB 3003, a composition processing program management DB 3004, a distortion correction program management DB 3005, and a service program management DB 3006 are constructed.

The imaging device information management DB 3001 is composed of an imaging device information management table, which will be described later. The cycle value management DB 3002 is composed of a cycle value management table which will be described later. The image acquisition program management DB 3003 is composed of an image acquisition program management table, which will be described later. The composite processing program management DB 3004 is composed of a composite processing program management table which will be described later. The distortion correction program management DB 3005 is composed of a distortion correction program management table which will be described later. The service program management DB 3006 is composed of a service program management table, which will be described later.

(Image sensor information management table)
FIG. 10A is a conceptual diagram showing an imaging device information management table. In this image pickup device information management table, the model number of the image pickup unit 40, the number of image pickup devices included in the image pickup unit 40, and the types of lenses used for these image pickup devices are associated and managed. Note that the model number is an example of type information indicating the type of imaging unit according to the number of imaging elements and the type of lens used for the imaging elements. Further, as long as the number of imaging elements and the type of lens can be specified, identification information such as a product number, bar code, etc. may be used instead of the model number.

(cycle value management table)
FIG. 10(b) is a conceptual diagram showing a cycle value management table. In this cycle value management table, the number of image pickup elements of the image pickup unit 40 and the cycle value (Frames Per Second) of object recognition processing (to be described later) are associated and managed.

Each table shown in FIG. 11 is a table for managing whether each program is installed in the real-time data processing terminal 3 or not.

(Image Acquisition Program Management Table)
FIG. 11A is a conceptual diagram showing an image acquisition program management table. In this image acquisition program management table, the number of image pickup devices of the image pickup unit 40, the image acquisition program (or its name), and information on the presence/absence of installation are associated and managed. For example, when the image acquisition program for the case where the number of imaging devices is "1" is installed in the real-time data processing terminal 3, the program "ProgC01 (1 system image acquisition)" and the information on whether or not it is installed "1 ” is associated and managed. If the image acquisition program for the case where the number of image pickup devices is "2" is installed in the real-time data processing terminal 3, the program "ProgC02 (acquisition of two systems of images)" and the information "1" for the presence or absence of installation are displayed. managed in association. When the software is installed, the information on whether or not it is installed is indicated by "1", and when it is not installed, the information on whether or not it is installed is indicated by "0".

(Synthesis processing program management table)
FIG. 11B is a conceptual diagram showing a composition processing program management table. In this compositing processing program management table, the number of imaging elements of the imaging unit 40, the name of the compositing processing program, and information on whether or not it is installed are associated and managed. For example, if an image synthesizing program for the case where the number of imaging devices is "1" is installed in the real-time data processing terminal 3, the program "ProgS01 (image synthesizing)" and the information "0" on whether or not it is installed are associated. managed by In this case, images obtained by capturing images a plurality of times with one image sensor are synthesized. When the image program for the case where the number of imaging devices is "2" is installed in the real-time data processing terminal 3, the program "ProgS02 (image synthesis)" and the information "1" regarding the presence or absence of installation are managed in association with each other. ing.

(Distortion correction program management table)
FIG. 11(c) is a conceptual diagram showing a distortion correction program management table. In this distortion correction program management table, the type of lens included in the imaging unit 40, the distortion correction program (or its name), and information on whether or not it is installed are associated and managed. For example, if a distortion correction program for the lens type "wide-angle" is installed in the real-time data processing terminal 3, the program "ProgW01 (wide-angle distortion correction)" and the information "1" indicating whether or not it is installed are associated. managed by When the distortion correction program for the lens type "fisheye" is installed in the real-time data processing terminal 3, the program "ProgW02 (fisheye distortion correction)" and the installation information "1" are associated. managed by

(service program management table)
FIG. 11(d) is a conceptual diagram showing a service program management table. In this service program management table, the authentication server ID for identifying the authentication server, the name of the service program for executing the service provided by the image acquisition terminal 2, and information on the presence or absence of installation are associated and managed. . For example, when the service program for the authentication server ID "a01" is installed in the real-time data processing terminal 3, the program "ProgD01 (object detection)" and the installation information "1" are associated and managed. It is In addition, when the service program for the authentication server ID "a02" is installed in the real-time data processing terminal 3, the program "ProgD02 (object counting)" and the information "1" regarding the presence or absence of installation are managed in association with each other. It is

(Each functional configuration of the real-time data processing terminal)
Next, with reference to FIG. 8, each functional configuration of the real-time data processing terminal 3 will be explained in more detail.

A determination unit 33 of the real-time data processing terminal 3 is realized by processing of the CPU 301 and performs various determinations. For example, the determination unit 33 determines the number of image sensors based on the model number sent from the imaging unit 40 and with reference to the image sensor information management DB 3001 .

The image processing unit 34 is realized by the processing of the CPU 301, and performs various image processing by executing each program (image acquisition program, image synthesis program, distortion correction program, and service program) shown in FIG. Specifically, the image processing unit 34 executes a first program (for example, an image acquisition program, an image synthesis program, and a distortion correction program) that does not require authentication to acquire the image data. First image processing (for example, image acquisition, image synthesis, distortion correction) is performed. Further, the image processing unit 34 executes a second program (for example, a service program) that requires authentication to obtain the image data, thereby performing second image processing (for example, object detection processing, object counting process).

For example, the image processing unit 34 detects, in the captured image data acquired by the image processing unit 34, feature points that are candidates for a predetermined specific object such as a face, and determines a predetermined specific object such as a face. The position of the object in the captured image is detected by referring to the shape model data representing the shape model of the object. Here, a known method can be used for the detection method.

The event generation unit 36 is implemented by the processing of the CPU 301 and generates detection information (event information) indicating that the position of the object has been detected by the image processing unit 34 .

The display control unit 37 is implemented by the processing of the CPU 301 and causes the display 317, which is an example of display means, to display various screens.

The connection unit 38 is realized by the processing of the imaging unit I/F 313 and the CPU 301 , and is an interface for mechanically and electrically connecting the imaging unit 40 to the real-time data processing terminal 3 .

The storage/readout unit 39 is implemented by the processing of the CPU 301 , stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000 .

The communication unit 48 is implemented by the external device connection I/F 318 and the processing of the CPU 301, and transmits and receives various data (or information) to and from the communication unit 58 of the near-terminal data processing device 5 via one-to-one communication. conduct. Note that this communication may be wireless as well as wired.

<Functional configuration of near-terminal data processing device>
As shown in FIG. 8 , the near-terminal data processing device 5 has a transmission/reception section 51 , a data detection section 56 , a display control section 57 , a communication section 58 and a storage/readout section 59 . Each of these units is realized by operating one of the components shown in FIG. function or means to be performed.

The near-terminal data processing device 5 also has a storage unit 5000 constructed by the ROM 502, RAM 503, and EEPROM 504 shown in FIG.

(Each functional configuration of the near-terminal data processing device)
Next, with reference to FIG. 8, each functional configuration of the near-terminal data processing device 5 will be described in more detail.

The transmitting/receiving unit 51 of the near-terminal data processing device 5 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501, and communicates with the distributed data processing terminal 6 via the communication network (here, the intranet 200). to send and receive various data (or information).

The data detection unit 56 is implemented by the processing of the CPU 501 and detects whether an event for receiving data from the real-time data processing terminal 3 has occurred and whether data reception has been completed.

The display control unit 57 is implemented by the processing of the CPU 501 and causes the display 517, which is an example of display means, to display various screens.

The communication unit 58 is realized by the processing of the external device connection I/F 518 and the CPU 501, and transmits and receives various data (or information) to and from the communication unit 48 of the real-time data processing terminal 3 via one-to-one communication. Note that this communication may be wireless as well as wired.

The storage/readout unit 59 is realized by the processing of the CPU 501 , stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

<Functional Configuration of Distributed Data Processing Terminal>
As shown in FIG. 9, the distributed data processing terminal 6 has a transmitting/receiving section 61 , a receiving section 62 , a determining section 63 , a display control section 67 and a storage/reading section 69 . Each of these units is realized by operating one of the components shown in FIG. function or means to

The distributed data processing terminal 6 also has a storage unit 6000 constructed by the ROM 502, RAM 503, and EEPROM 504 shown in FIG. The storage unit 6000 stores collation data. Furthermore, a collation data management DB 6001 is constructed in the storage unit 6000 . The collation data management DB 6001 is composed of a collation data management table which will be described later. Note that the collation data may be stored in a data management server or the like other than the distributed data processing terminal 6 .

(matching data management table)
FIG. 12 is a conceptual diagram showing a matching data management table. In this collation data management table, an image file indicating the file name of the collation data and the name of the person indicated by the collation image are associated and managed.

(Each functional configuration of the distributed data processing terminal)
The transmission/reception unit 61 of the distributed data processing terminal 6 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501 in the distributed data processing terminal 6, and transmits centralized data via a communication network (here, the Internet 600). It transmits and receives various data (or information) to and from the processing server 7 . For example, the transmitting/receiving unit 61 transmits to the centralized data processing server 7 a request for collation of collation data and data to be collated, and processes the collation results sent from the centralized data processing server 7 . The transmission/reception unit 61 also transmits/receives data to be compared and various data (or information) to/from the image acquisition terminal 2 via a communication network (here, the intranet 200).

The reception unit 62 is realized by processing of the touch panel 521 and the CPU 501 in the distributed data processing terminal 6, and receives various operations of the user.

The determination unit 63 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and makes various determinations.

The display control unit 67 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and displays various screens on the display 517, which is an example of display means.

The storage/readout unit 69 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, stores various data (or information) in the storage unit 6000, and reads out various data (or information) from the storage unit 6000. For example, the storage/readout unit 69 stores collation data (here, face image data) in the storage unit 6000 in response to the registration request accepted by the acceptance unit 62 and registers it.

<Functional configuration of centralized data processing server>
As shown in FIG. 9 , the centralized data processing server 7 has a transmitter/receiver 71 , a receiver 72 , a feature amount generator 74 , a collator 75 , and a memory/reader 79 . Each of these units is realized by operating one of the components shown in FIG. function or means to

The centralized data processing server 7 also has a storage section 7000 constructed by the ROM 702, RAM 703, and HD 704 shown in FIG. The storage unit 7000 stores feature amount data on the collation side, which will be described later.

(Each functional configuration of the centralized data processing server)
The transmission/reception unit 71 of the centralized data processing server 7 is realized by the processing of the network I/F 709 and the CPU 701, and exchanges various data (or information) with the distributed data processing terminals 6 via the communication network (here, the Internet 600). Send and receive. The transmitting/receiving unit 71 receives from the distributed data processing terminal 6 a request for matching matching data and data to be matched, and transmits matching result information indicating a matching result to the distributed data processing terminal 6 .

The reception unit 72 is implemented by the keyboard 711, the mouse 712, and the processing of the CPU 701, and receives various user operations.

The feature amount generation unit 74 is realized by the processing of the CPU 701 and generates a feature amount parameter from the matching data (partial image data) and the matching data received by the transmission/reception unit 71 .

The matching unit 75 is implemented by the processing of the CPU 701, uses the feature amount generated by the feature amount generating unit 74 to match the feature amount on the matching data side with the feature amount on the side of the data to be matched, and generates a score indicating the degree of similarity. (point) is calculated.

The storage/readout unit 79 is implemented by the processing of the CPU 701 , stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000 .

<Functional configuration of service providing server>
Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIGS. 9 and 13 to 15. FIG. As shown in FIG. 9, the service providing server 8 has a transmitting/receiving section 81, a determining section 82, an extracting section 87, and a storing/reading section 89. FIG. Each of these units is a function realized by any one of the components shown in FIG. It is a means.

The service providing server 8 also has a storage unit 8000 constructed by the RAM 703 and HD 704 shown in FIG. Various data sent from the distributed data processing terminal 6 or the authentication server 9 are stored in the storage unit 8000 . 11 are stored in the storage unit 8000, and the service providing server 8 can transmit the requested program in response to a request from the real-time data processing terminal 3. can be done.

Furthermore, a session management DB 8001 and an authentication server management DB 8002 are constructed in the storage unit 8000 . Of these, the session management DB 8001 is composed of a session management table, which will be described later. The authentication server management DB 8002 is composed of an authentication server management table which will be described later. Each table will be described in detail below.

(session management table)
FIG. 13 is a conceptual diagram showing a session management table. In this session management table, in order to provide services to the distributed data processing terminal 6, a session ID for identifying a communication session to be established with the distributed data processing terminal 6, a terminal ID for identifying the distributed data processing terminal 6, and the IP address of the distributed data processing terminal 6 of the user indicated by the terminal ID are stored and managed in association with each other.

FIGS. 14A, 14B, and 14C show an e-mail address as an example of a terminal ID (identification), each of which is composed of an authentication target part and an authentication non-target part. The authentication target part is a user ID used when being authenticated by the authentication server 9 . The non-authentication target portion is a portion that is not used when authenticated by the authentication server 9 .

Among them, in the first pattern shown in FIG. 14(a), the part to be authenticated consists of the account name "asai", the host name "myhost", and the front part of the domain name "ricoo.com". ing. On the other hand, the non-authentication target part is composed of the latter part of the domain name, "theta1". In this case, the retrieving unit 87 distinguishes between the authenticated part and the non-authenticated part by "/".

FIG. 14(b) also shows the first pattern, but differs from FIG. 14(a) in the non-authentication target portion. That is, the authentication server 9 authenticates the terminal ID shown in FIG. 14A and the terminal ID shown in FIG.

The terminal ID may be the second pattern shown in FIG. 14(c). In this second pattern, the part to be authenticated consists of the front part "asai" of the account name. On the other hand, the non-authenticated part is composed of the latter half of the account name "theta2", the host name "myhost", and the domain name "ricoo.com". In this case, the retrieving unit 87 distinguishes between the authenticated part and the non-authenticated part by "+".

(Authentication server management table)
FIG. 15 is a conceptual diagram showing an authentication server management table. In this authentication server management table, a URL (Uniform Resource Locator) for accessing each authentication server 9 is stored and managed in association with each authentication server ID for identifying each authentication server 9 .

(Each functional configuration of the service providing server)
Next, with reference to FIG. 9, each functional configuration of the service providing server 8 will be described in detail.

The transmission/reception unit 81 of the service providing server 8 is mainly realized by commands from the CPU 701 shown in FIG. and send and receive various data (or information).

The determination unit 82 is mainly realized by commands from the CPU 701 shown in FIG. .

Extraction unit 87 is mainly implemented by commands from CPU 701 shown in FIG. 6, and performs processing for extracting a user ID (authentication target part) as shown in FIG. 14 from a terminal ID.

The storage/readout unit 89 is mainly implemented by commands from the CPU 701 and the HDD 705 shown in FIG.

<Functional configuration of the authentication server>
Next, the functional configuration of the authentication server 9 will be described in detail with reference to FIGS. 9 and 11. FIG. The authentication server 9 has a transmission/reception section 91 , an authentication section 92 , and a storage/readout section 99 . Each of these units is a function or means realized by any one of the components shown in FIG. is.

The authentication server 9 also has a storage unit 9000 constructed by the RAM 703 and the HD 704 shown in FIG. Various data sent from the distributed data processing terminal 6 or the service providing server 8 are stored in the storage unit 9000 .

Furthermore, an authentication management DB 9001 is constructed in the storage unit 9000 . The authentication management DB 9001 is composed of an authentication management table which will be described later. This table will be described in detail below.

(Authentication management table)
FIG. 16(a) is a conceptual diagram showing an authentication management table possessed by the authentication server 9a, FIG. 16(b) is a conceptual diagram showing an authentication management table possessed by the authentication server 9b, and FIG. 16(c) is an authentication authorization table possessed by the authentication server 9c. 4 is a conceptual diagram showing a management table; FIG.

In each authentication management table, a user ID (part to be authenticated) of the terminal ID and a password are stored and managed in association with each other.

(Each functional configuration of the authentication server)
Next, each functional configuration of the authentication server 9 will be described in detail with reference to FIG.

The transmission/reception unit 91 of the authentication server 9 is mainly implemented by commands from the CPU 701 and the network I/F 709 shown in FIG. Send and receive data (or information).

The authentication unit 92 is mainly implemented by commands from the CPU 701 shown in FIG. 6, and determines whether or not the image acquisition terminal 2 that has transmitted the authentication request has legitimate authority to receive services. , the ID of the distributed data processing terminal 6 is authenticated. Such authentication is performed because the distributed data processing terminal 6 manages the image acquisition terminal 2 .

The storage/readout unit 99 is mainly realized by instructions from the CPU 701 and the HDD 705 shown in FIG. or information).

<<Processing or operation of the present embodiment>>
Next, the processing or operation of this embodiment will be described with reference to FIGS. 17 to 33. FIG.

<Authentication process>
First, the authentication process will be described with reference to FIGS. 17 to 20. FIG. 17 and 18 are sequence diagrams showing authentication processing. 19 and 20 are diagrams showing screen examples of the distributed data processing terminal.

As shown in FIG. 17, the transmission/reception unit 61 of the distributed data processing terminal 6 transmits a request for an authentication destination selection screen to the service providing server 8 (step S21). This request contains the terminal ID of the distributed data processing terminal 6 . At this time, the transmission/reception unit 61 also transmits the IP address of its own terminal. As a result, the transmission/reception unit 81 of the service providing server 8 receives the request for the authentication destination selection screen and the IP address of the distributed data processing terminal 6 .

Next, the determination unit 82 of the service providing server 8 determines whether or not the terminal ID of the distributed data processing terminal 6 received in step S21 is managed in the session management table (see FIG. 13) in association with a predetermined session ID. (step S22). Hereinafter, a case where terminal IDs are not managed will be described.

The transmission/reception unit 81 of the service providing server 8 transmits the data of the authentication destination selection screen to the distributed data processing terminal 6 (step S23). As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the data of the authentication destination selection screen.

Next, the display control unit 63 of the distributed data processing terminal 6 causes the display 517 to display an authentication destination selection screen s1 as shown in FIG. 19 (step S24). FIG. 19 shows a screen example of the distributed data processing terminal 6 . The authentication destination selection screen s1 displays a terminal ID input field b1, a password input field b2, and a login button b3 for requesting login (authentication request). Further, authentication server selection buttons a1, a2 and a3 for selecting the authentication servers 9a, 9b and 9c are displayed on the authentication destination selection screen s1. For example, the authentication server selection button a1 is a button used when the user receives an object detection service. The authentication server selection button a2 is a button used when the user receives the object counting service.

Here, the user inputs his/her (distributed data processing terminal 6) terminal ID in the entry field b1 and his/her own password in the entry field b2, and presses a desired button among the authentication server selection buttons a1, a2, and a3. Then, when the login button b3 is pressed, the reception unit 62 receives each input and selection (step S25). Here, a case will be described where the object detection processing is executed by the object detection processing program ProgD01 in the image recognition processing by selecting the authentication server selection button a1.

The transmitter/receiver 61 transmits an ID (here, terminal ID or user ID) authentication request to the service providing server 8 (step S26). This authentication request includes the terminal ID and password accepted in step S25, the selection result of the authentication server 9, and the URL of the distributed data processing terminal 6. FIG. This selection result indicates an authentication server ID for identifying the authentication server 9 . As a result, the transmission/reception unit 81 of the service providing server 8 receives the ID authentication request.

Next, the storage/readout unit 89 of the service providing server 8 searches the authentication server management table (see FIG. 15) using the authentication server ID as the selection result received in step S26 as a search key, thereby performing a corresponding The URL of the authentication server is read (step S27).

Next, the extraction unit 87 extracts only the user ID (authentication target part) from the terminal ID received in step S26 (step S28). Then, the transmitting/receiving section 81 transmits an ID authentication request to the authentication server 9 indicated by the URL read out in step S27 (step S29). This ID authentication request includes the user ID (part to be authenticated) extracted in step S28, the password received in step S26, and the URL of the distributed data processing terminal 6 received in step S26. . Thereby, the transmission/reception unit 71 of the authentication server 9 receives the user's authentication request.

Next, the storage/readout unit 99 of the authentication server 9 uses the set of the user ID (authentication target part) and password received in step S29 as a search key, and uses the same set of authentication information in the authentication management table (see FIG. 16). The authentication unit 92 performs authentication using the search result of the target portion and password (step S30). If the same group is managed, the authentication unit 92 determines that the distributed data processing terminal 6 is a legitimate terminal for receiving services from the service providing server 8. If the same group is not managed, the authentication unit 92 determines that the distributed data processing terminal 6 is not a valid terminal for receiving services from the service providing server 8 .

In step S28, the extractor 87 extracts the part to be authenticated from the terminal ID, but it is not limited to this. For example, the distributed data processing terminal 6 may extract the part to be authenticated from the terminal ID before the process of step S26. In this case, in step S26, the user ID (part to be authenticated) is transmitted instead of the terminal ID. Also, the authentication server 9 may extract the part to be authenticated from the terminal ID after the process of step S29. In this case, in step S29, the terminal ID is transmitted instead of the user ID (part to be authenticated).

Subsequently, as shown in FIG. 18, the authentication unit 92 of the authentication server 9 encrypts the token (transmission right) (step S41). Then, the transmitting/receiving unit 91 transmits the authentication result to the distributed data processing terminal 6 based on the URL of the distributed data processing terminal 6 received in step S29 (step S42). This authentication result indicates whether the distributed data processing terminal 6 is valid and includes the token encrypted in step S41. Thereby, the transmitting/receiving unit 61 of the distributed data processing terminal 6 receives the authentication result of the user. Hereinafter, a case where the user has legitimate authority will be described.

The transmission/reception unit 61 of the distributed data processing terminal 6 transmits a session establishment request to the service providing server 8 (step S43). This session establishment request contains the terminal ID and the encrypted token received in step S42. Thereby, the transmission/reception unit 81 of the service providing server 8 receives the session establishment request.

Next, in order to confirm that the distributed data processing terminal 6 that has transmitted the session establishment request is valid in step S30, the service providing server 8 sends the token to the authentication server 9. An authentication request is sent (step S44). This token authentication request includes the encrypted token received in step S43. As a result, the transmitter/receiver 91 of the authentication server 9 receives the token authentication request.

Next, the authentication unit 92 decrypts the encrypted token received in step S44 (step S45). The authentication unit 92 then authenticates the token by comparing the token before encryption in step S41 and the token after decryption in step S45 (step S46). Then, the transmission/reception unit 91 of the authentication server 9 transmits the authentication result of step S46 to the service providing server 8 (step S47). Thereby, the transmission/reception unit 81 of the service providing server 8 receives the authentication result. Hereinafter, the case where the token is determined to be valid in step S46 will be described.

Next, the storage/reading unit 89 of the service providing server 8 assigns a new session ID to the session management table (see FIG. 13), and assigns the terminal ID and IP received in step S26 to this session ID. Addresses are associated and managed (step S48). Then, the transmitting/receiving unit 81 transmits the data of the service providing screen to the distributed data processing terminal 6 (step S49). Thereby, the transmitting/receiving unit 61 of the distributed data processing terminal 6 receives the data of the service providing screen.

Next, the display control unit 63 of the distributed data processing terminal 6 causes the display 517 to display a service providing screen s2 as shown in FIG. 20 (step S50). FIG. 20 shows a screen example of the distributed data processing terminal 6. As shown in FIG. Here, as an example of service provision, a remote operation service for realizing remote operation of the image acquisition terminal 2 from the distributed data processing terminal 6 will be described. On the service providing screen s2 shown in FIG. 20, an IP address input field c1 for specifying a remote operation target and a "remote operation start" button c2 are displayed. Here, the terminal ID for identifying the remote control target is the terminal ID for identifying the image acquisition terminal 2 . The distributed data processing terminal 6 designates the IP address of the image acquisition terminal 2 to be remotely operated by user input, or uses the connection address of the image acquisition terminal 2 stored in the distributed data processing terminal 6 in advance. can be set by

<Preparation processing for image recognition>
Next, preparation processing for image recognition will be described with reference to FIGS. 21 to 23. FIG. FIG. 21 is a sequence diagram showing processing of an image recognition start request.

As shown in FIG. 21, in the distributed data processing terminal 6, the reception unit 62 receives a request to start image recognition from the user (step S61). In this case, the GUI (Graphical User Interface) of the distributed data processing terminal 6 is used. As a result, the transmitting/receiving unit 61 of the distributed data processing terminal 6 transmits image recognition start request information indicating a request to start image recognition to the near terminal data processing device 5 of the set image acquisition terminal 2 (step S62). This start request information includes an authentication server ID for identifying the authentication server 9 that authenticated in the authentication process described above (see FIGS. 17 and 18) and the session ID set in FIG. . As a result, the transmitting/receiving unit 51 of the near-terminal data processing device 5 receives the image recognition start request information. Then, the communication unit 58 transmits image recognition start request information to the real-time data processing terminal 3 (step S63). As a result, the communication unit 48 of the real-time data processing terminal 3 receives the image recognition start request information. By separating the user interface from the real-time data processing terminal 3 in this manner, remote operation from the distributed data processing terminal 6 becomes possible.

FIG. 22 is a sequence diagram showing preparatory processing for real-time processing of the real-time data processing terminal. For example, the real-time data processing terminal in the image acquisition terminal 2 that received the start request information shown in FIG. 21 executes the sequence shown in FIG.
As shown in FIG. 22, the connecting section 38 of the real-time data processing terminal 3 acquires the model number of the imaging unit 40 from the imaging unit 40 (step S71). In this case, the connecting section 38 requests the imaging unit 40 for the model number, and the imaging unit 40 transmits its own model number in response to this request.

Next, the storage/reading unit 39 searches the image pickup device information management DB 3001 (see FIG. 10A) using the model number acquired in step S71 as a search key, thereby obtaining the number of corresponding image pickup devices and the number of lenses. type is read out (step S72). Furthermore, the storage/readout unit 39 reads out the corresponding cycle value by searching the cycle value management DB 3002 (see FIG. 10B) using the number of image sensors read out in step S72 as a search key ( step S73).

Next, the storage/readout unit 39 searches for an image acquisition program managed in the image acquisition program management DB 3003 (see FIG. 11A) using the number of image sensors read out in step S72 as a search key. (Step S74). Next, the storage/readout unit 39 searches for a composition processing program managed in the composition processing program management DB 3004 (see FIG. 11B) using the number of image sensors read out in step S72 as a search key. (Step S75). Next, the storage/readout unit 39 searches for a distortion correction program managed in the distortion correction program management DB 3005 (see FIG. 11C) using the lens type read out in step S72 as a search key ( step S76). Next, the storage/readout unit 39 searches for service programs managed in the service program management DB 3006 (see FIG. 11D) using the lens type read out in step S72 as a search key (step S77). ).

Next, the determination unit 33 determines whether or not all the programs to be executed have been installed based on whether or not the programs to be executed are managed based on the search results of steps S74 to S77 (step S78). For example, in step S74, as a result of searching for an image processing program by the storage/reading unit 39, if the image processing program is managed, the determination unit 33 determines that the image processing program has been installed. , the determination unit 33 determines that the software has not been installed.

Then, when the judgment unit 33 judges that all four programs are managed (step S78; YES), the processing shown in FIG. 22 ends. On the other hand, if the determination unit 33 determines that at least one of the four programs is not installed (step S78; NO), the process proceeds to step S91, which will be described later.

FIG. 23 is a sequence diagram showing program acquisition processing. Here, there is shown a process for the real-time data processing terminal 3 to acquire from the service providing system 8 a program that has been found not to be installed by the process shown in FIG.

First, as shown in FIG. 23, the communication unit 48 of the real-time data processing terminal 3 sends request information indicating a request for a program that has not been installed to the communication unit 58 of the near-terminal data processing device 5. Send (step S91). This request information includes the requesting program name, authentication server ID, and session ID.

Next, the transmitting/receiving unit 51 of the near terminal data processing device 5 transmits the request information received by the communication unit 58 to the transmitting/receiving unit 61 of the distributed data processing terminal 6 (step S92). Then, the transmitting/receiving section 61 of the distributed data processing terminal 6 transmits the request information to the transmitting/receiving section 81 of the service providing server 8 (step S93).

Next, in the service providing server 8, the storage/readout unit 89 confirms the validity of the request based on the authentication server ID and the session ID, and reads out the program indicated by the program name included in the request information. (Step S94). Then, the transmitting/receiving section 81 transmits the read program to the transmitting/receiving section 61 of the distributed data processing terminal 6 (step S95). At this time, the program name, authentication server ID, and session ID are also transmitted.

Next, based on the authentication server ID and the session ID, the transmitting/receiving unit 61 of the distributed data processing terminal 6 transfers the program including the program name to the transmitting/receiving unit 51 of the near terminal data processing device 5 of the specified image acquisition terminal 2. is transmitted (step S96). Then, the communication unit 58 of the near-terminal data processing device 5 transmits the program including the program name to the communication unit 48 of the real-time data processing terminal 3 (step S97).

Next, the storage/reading unit 39 of the real-time data processing terminal 3 installs the program acquired by the communication unit 48, and also, among the tables shown in FIG. It is managed together with the ID (step S98).

Next, the storage/readout unit 39 activates all programs necessary for image recognition processing (step S99). As a result, the real-time data processing terminal 3 starts the real-time processing shown below by executing the activated program.

<Image recognition processing>
(Object detection processing)
Here, when the "login to object detection processing service" button b1 shown in FIG. 19 is pressed, the service providing server 8 gives permission to execute the object detection processing service among the image recognition processing services. will be explained. 17 to 20 shown above are executed, and the situation after the "remote operation start" button c2 in FIG. show. Therefore, the distributed data processing terminal 6 has transmitted the aforementioned start request information to the image acquisition terminal 2 . FIG. 24 is a sequence diagram showing image recognition processing. First, the real-time data processing terminal 3 performs real-time processing (step S111).
Real-time processing will now be described with reference to FIGS. 25 and 26. FIG. FIG. 25 is a flowchart showing object detection processing in real-time processing.

First, the determination unit 33 determines whether or not the number of imaging elements in the connected imaging unit 40 is one (step S201). In this case, the determination unit 33 makes a determination based on the number of image sensors read out by the process of step S72 described above. Then, if the number of imaging elements is one (step S201; YES), the image processing unit 34 sets the cycle value for repeating the process to 1/60 second (step S202). In this case, the image processing unit 34 sets the cycle value read by the process of step S73 described above.

Next, the connection unit 38 acquires one system of captured image data from the imaging unit 40 (step S203). This captured image data is digital image data, for example, data of a 4K image (width 3840 pixels×height 2160 pixels). In this case, the connection unit 38 executes processing by the image acquisition program (ProgC01 (1 system)) shown in FIG. 11(a). In executing this process, the real-time data processing terminal 3 does not need to use the authentication server ID included in the start request information and refer to FIG. 11(d) to determine whether the image acquisition program can be executed.

Next, the image processing unit 34 detects the object by detecting feature points that are candidates for the object in the captured image data (step S204). In this case, the image processing unit 34 uses the authentication server ID included in the start request information to determine from FIG. After determining whether the service program (ProgD01 (object detection processing)) can be executed, the processing is executed. The image processing unit 34 searches for a rectangle from the end of the captured image while comparing it with the shape model data of the object stored in advance in the storage unit 3000, and selects the position of the plausible feature point of the object. Regarding the processing of step S204, for example, "Face Recognition Technology and Its Applications: Special Feature on Elemental Technologies and Solutions Supporting Public Safety; Biometrics Authentication, NEC Technical Report, Vol.63, no.3, pp.26-30 , 2010-09” can be used.

Next, the image processing unit 34 corrects the distortion of the image of the detected object (step S205).
In this case, the image processing unit 34 executes processing according to the image processing program (ProgW01 (wide-angle distortion correction)) shown in FIG. 11(c). In executing this process, the real-time data processing terminal 3 does not need to use the authentication server ID included in the start request information and refer to FIG. 11(d) to determine whether the wide-angle distortion correction program can be executed.

On the other hand, in step S201, if the number of image sensors is not one (step S201; NO), the image processing unit 34 sets the cycle value for repeating real-time processing to 1/30 seconds (step S206). ). In this case, the image processing unit 34 sets the cycle value read by the process of step S73 described above. By setting the cycle value to 1/30 second, it is possible to prevent the later-described image synthesizing process from being delayed by setting the cycle value to be large compared to one input.

Next, the connection unit 38 acquires two captured image data of two systems from the imaging unit 40 (step S207). The two captured image data are hemispherical image data as shown in FIGS. 4(a) and 4(b). In this case, the connection unit 38 executes processing by the image acquisition program (ProgC02 (2 systems)) shown in FIG. 11(a). In executing this process, it is not necessary to determine whether or not the process can be executed, as described above.

Then, the image processing unit 34 synthesizes the two captured image data to create an equirectangular projection image EC as shown in FIG. 4(c) (step S208). In this case, the image processing unit 34 executes processing according to the synthesis processing program (ProgS02 (synthesis processing)) shown in FIG. 11(b). In executing this process, the real-time data processing terminal 3 does not need to use the authentication server ID included in the start request information to refer to FIG. None. Then, the process proceeds to step S204, and the image processing unit 34 detects the object by detecting feature points that are object candidates in the data of the equirectangular projection image EC. In this case, as described above, the image processing unit 34 uses the authentication server ID included in the start request information to execute the service program (ProgD01 (object detection processing)) shown in FIG. Execute the process after judging whether it can be executed or not.

Next, in step S205, the image processing unit 34 corrects the distortion of the image of the detected object. In this case, the image processing unit 34 executes processing according to the image processing program (ProgW02 (fisheye distortion correction)) shown in FIG. 11(c). In executing this process, the real-time data processing terminal 3 does not need to use the authentication server ID included in the start request information and refer to FIG. .

As described above, as shown in FIG. 24, the communication unit 48 of the real-time data processing terminal 3 transmits the captured image data to the communication unit 58 of the near-terminal data processing device 5 (step S112). Then, the transmitting/receiving unit 51 of the near terminal data processing device 5 transmits the captured image data received in step S112 to the transmitting/receiving unit 61 of the distributed data processing terminal 6 (step S113). As a result, the display control unit 67 of the distributed data processing terminal 6 causes the display 517 to display the captured image as shown in FIG. 27(a) in real time. FIG. 27 is a diagram showing a display example of a captured image on the distributed data processing terminal. In this captured image, a frame indicating a rectangle for which object detection (here, face detection) has been performed is displayed. Steps S112 to S114 are streaming processes.

Next, event generation processing in real-time processing will be described with reference to FIG. FIG. 26 is a flowchart showing event generation processing in real-time processing.

If the image processing unit 34 does not detect an object (here, a face) by the processing of step S204 (step S211; NO), whether or not a predetermined time (set cycle value) has elapsed is determined (step S212). A standby state is maintained until a predetermined period of time has elapsed (step S212; NO), and when the predetermined period of time has elapsed (step S212; YES), the process returns to step S211. On the other hand, if the image processing unit 34 detects an object (here, a face) through the process of step S204 (step S211; YES), the process proceeds to step S213. Then, when the image processing unit 34 detects a plurality of objects (faces in this case) through the processing of step S204 described above (step S213; NO), the display control unit 37 performs the processing shown in FIG. 27(b). , the captured image superimposed with the message m1 is displayed (step S214). This message m1 indicates the contents of prompting the user's action so that one face can be photographed. After the process of step S214, the process proceeds to step S212.

On the other hand, when the image processing unit 34 detects only one object (here, face) through the processing of step S205 described above (step S213; YES), the display control unit 37 performs the processing shown in FIG. is not displayed (step S215). In this case, the display control unit 37 does not display the message m1 when the message m1 is already displayed.

Next, the image processing unit 34 encodes the partial image data including the face at the detected position in a general-purpose format such as JPEG (step S216). Then, the event generator 36 generates an event message to notify the near-terminal data processing device 5 that the partial image data will be transmitted (step S217). Specifically, the event generator 36 generates an event message m2 such as "Send".

Thus, the process of step S111 shown in FIG. 24 is completed. Accordingly, the communication unit 48 transmits the event message m2 generated in step S217 and the partial image data detected in step S205 to the communication unit 58 of the near-terminal data processing device 5 (step S115). Note that when a plurality of objects (here, faces) are detected in the process of step S205, a plurality of partial image data are transmitted together with one event message m2. Also, when the communication system is used in a store or the like, the real-time data processing terminal 3 stores the event message m2 and the partial image data, and after the store closes (for example, at night), the real-time data processing terminal 3 stores the event message m2 and the partial image data. It may be transmitted to the terminal data processing device 5 .

Next, in the near-terminal data processing device 5, the data detection unit 56 detects whether or not the communication unit 58 has received the event message m2 of "Send" (step S116). Then, when the reception of the event message m2 is detected (step S116; YES), the communication section 58 receives the partial image data transmitted together with the event message m2 (step S117). Then, storage/readout unit 59 temporarily stores the partial image data in storage unit 5000 (step S118).

Next, the data detection unit 56 monitors whether or not the reception of the partial image data is completed (step S119). The process of step S119 is repeated until all partial image data are received for one event message m2 (step S119; NO). Then, when the reception of the partial image data is completed (step S119; YES), the storage/readout unit 59 is sent from the storage unit 5000 together with one event message m2, and is temporarily stored in the storage unit 5000. All partial image data are read out (step S120). After that, the transmitting/receiving section 51 transmits all the partial image data read in step S120 to the transmitting/receiving section 61 of the distributed data processing terminal 6 via the intranet 200 (step S121). As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives all the partial image data. The partial image data will be used later as collated data.

Next, referring to FIGS. 28 to 30, the matching process between the data to be matched, which is partial image data, and the registered matching data will be described. FIG. 28 is a sequence diagram showing the matching process. FIG. 29(a) is a display example of a verification data file selection screen in the distributed data processing terminal, and FIG. 29(b) is a display example of a verification data registration screen in the distributed data processing terminal. FIG. 30 is a display example of the collation result message m3 on the distributed data processing terminal.

First, the user operates the distributed data processing terminal 6 to register the verification data, and the display control unit 67 displays a file selection screen for verification data as shown in FIG. 29(a). . Accordingly, when the user selects an image to be registered (here, an image of a face) after pressing the "select file" button b11, the reception unit 62 receives selection of a file of the image to be registered (step S311 ). As a result, the display control unit 67 displays a registration screen for matching data as shown in FIG. 29(b). The selected image and file a21, a name input field a22, a "Register" button b21, and a "Cancel" button b22 are displayed in this registered image. Therefore, after confirming whether the selected image and file a21 are the desired image, the user enters the name of the image to be registered as verification data in the name input field a22 if it is the desired image, and then selects " When the "registration" button b21 is pressed, the reception unit 62 registers the image as collation data (step S312). Specifically, the storage/readout unit 69 associates and stores the file name a21 and the name input in the name input field a22 in the collation data management DB 6001 . If the user presses the "cancel" button b22 when the selected image and file a21 are not the desired image, the screen returns to the file selection screen shown in FIG. 29(a). In this way, by registering the verification data on the distributed data processing terminal 6 side without registering the verification data on the near terminal data processing device 5 side, the communication between the near terminal data processing device 5 and the distributed data processing terminal 6 It is possible to suppress the load of communication network communication.

Next, when the transmitting/receiving unit 61 of the distributed data processing terminal 6 receives the partial image data through the process of step S121 described above, the storage/reading unit 69 retrieves the registered data by searching the collation data management DB 6001. It is determined whether or not collation data remains (step S313). Then, when it is determined that there are remaining, the transmitting/receiving section 61 transmits collation request information indicating a collation request to the centralized data processing server 7 via the Internet 600 (step S314). This matching request information includes matching data and data to be matched. As a result, the transmission/reception unit 71 of the centralized data processing server 7 receives the collation request information. At this time, the collation data are sequentially transmitted from the first row in the collation data management table shown in FIG. That is, first, the read collation data and the to-be-collated data received in step S121 are transmitted. Then, when the next collation request information is transmitted, the collation data read out next and the to-be-collated data received in step S121 are transmitted.

Next, in the centralized data processing server 7, the feature amount generating unit 74 decodes both data (matching data and data to be matched) into bitmap data, and converts the unevenness of the nose and eyes of the face image related to both data. A feature amount parameter for identifying an individual, such as an inclination, is generated (step S315). Then, the matching unit 75 compares the parameters of the feature amounts of both data to calculate the degree of similarity (step S316). For example, "Deep Learning and Image Recognition: Fundamentals and Recent Trends (<Special Issue> Neuroscience and Mathematical Modeling) Operations Research: Management Science 60(4), 198-204, 2015 -04-01”, a deep neural network (DNN) is used. The parameter of the feature amount of the matching data is an example of the parameter of the first feature amount, and the parameter of the feature amount of the to-be-matched data is an example of the parameter of the second feature amount.

Next, the transmission/reception unit 71 of the centralized data processing server 7 transmits response information indicating a response corresponding to the collation request in step S314 to the transmission/reception unit 61 of the distributed data processing terminal 6 via the Internet 600 (step S317). This response information includes the degree of similarity calculated in step S316. As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the response information.

Next, in the distributed data processing terminal 6, the storage/readout unit 69 stores the "name" attached to the collation data included in the collation request information transmitted in step S314 and the "name" received in step S317. similarity" is temporarily stored in the storage unit 6000 (step S318). After that, in the collation data management table shown in FIG. 12, the collation data is changed to the next row, and the process of step S313 is performed again.

On the other hand, if it is determined in step S313 that no collation data remains (including the case where there is none at all), the process proceeds to step S319. Then, the storage/readout unit 69 reads out the "name" attached to the collation data with the highest similarity among the degrees of similarity temporarily stored in the storage unit 6000 (step S319). Then, the display control unit 67 displays a matching result message as shown in FIG. 30 on the display 517 of the distributed data processing terminal 6 on the real-time captured image shown in FIG. 27(a). (step S320). The collation result message m3 displays the collation result and the "name" given to the collation data with the maximum degree of similarity.

(object number processing)
The above is the object detection processing executed by the object detection processing program ProgD01 in the image recognition processing. Next, when the authentication server selection button a2 shown in FIG. 19 is selected in step S25, the execution of the object number processing by the object counting processing program ProgD02 in the image recognition processing is permitted. explain. 17 to 20 are executed in the same manner as the object detection processing program ProgD01 shown earlier, and the distributed data processing terminal 6 directs the image acquisition terminal 2 to the "remote operation start" button in FIG. Indicates that it is executed by pressing c2. The object number processing will be described with reference to FIGS. 31 to 33. FIG.

FIG. 31 is a diagram showing a situation in which image acquisition terminals are arranged in a certain room. This embodiment shows a case where the object detection range in the imaging range R1 of the image acquisition terminal 2 is limited to the left half of the captured image, rather than the entire captured image. For example, when the image acquisition terminal 2 is used as a surveillance camera, the right side is the entrance D2 exclusively for employees, and since the employees use IC cards or the like to enter and exit, it is difficult for suspicious persons to enter and exit. On the other hand, since the left side is the doorway D1 for ordinary people, a suspicious person can enter and exit easily. Therefore, in this embodiment, the object detection range is limited to the left half of the captured image. Note that the object detection range may be the right half, the upper half, and the lower half. Also, the target is not limited to two divisions, and the number of divisions may be changed as appropriate, such as four divisions, depending on the application.

FIG. 32 is a sequence diagram showing a modified example of event generation processing of the real-time processing shown in FIG. 33 is a sequence diagram showing another example of the matching process shown in FIG. 28. FIG.

First, as shown in FIG. 32, processing similar to that of step S211 shown in FIG. 26 is performed (step S221). However, in this case, the image processing unit 34 uses the authentication server ID included in the start request information to determine from FIG. After judging whether or not the service program (object counting processing program ProgD02) can be executed, the processing is executed. Then, when the image processing unit 34 does not detect an object (here, a face) by the processing of step S204 (step S221; NO), it is determined whether a predetermined time (set cycle value) has elapsed. It is determined whether or not (step S222). Then, it is in a standby state until a predetermined time elapses (step S222; NO), and when the predetermined time elapses (step S222; YES), the process returns to step S221 again. On the other hand, if the image processing unit 34 detects an object (here, a face) through the process of step S204 (step S221; YES), the process proceeds to step S223. If the position (x, y) of the object detected by the image processing unit 34 is not within the object of the captured image (here, within the left half) (step S223; NO), the process proceeds to step S222. move on. On the other hand, if the position (x, y) of the object detected by the image processing unit 34 is within the object of the captured image (here, within the left half) (step S223; YES), step The process proceeds to S224. Here, the position (x, y) of the object is coordinates in the width direction and the height direction with the upper left corner of the captured image as the origin. Specifically, if the width of the entire captured image is W, x<(W/2) (step S223; YES) and x≧(W/2) (step S223; NO ) and Thereafter, the same processing as in steps S216 and S217 described above is performed (steps S224 and S225).

Next, with reference to FIG. 33, the processing when the communication system is applied to the surveillance camera system will be described. Since the same processing as that of steps S311 to S319 shown in FIG. 28 is performed, the description thereof will be omitted.

After the process of step S319, the determination unit 63 of the distributed data processing terminal 6 determines whether the maximum similarity read out in step S319 is equal to or greater than a threshold value (for example, "0.5" when "1" is a perfect match). (step S331). In this case, if the maximum similarity is less than the threshold, the distributed data processing terminal 6 treats the object (face) as unknown. Then, the display control unit 67 causes the display 517 of the distributed data processing terminal 6 to display a message indicating "unknown" in the collation result column of the collation result message m3 shown in FIG. 30 (step S332). . In this case, the name of the collation result message m3 is not displayed.

Next, when the distributed data processing terminal 6 treats the object as an unknown person (step S333; YES), the storage/readout unit 69 increments the number of unknown persons managed by the storage unit 6000 by one ( step S334). Then, the transmitting/receiving unit 61 sends to the transmitting/receiving unit 71 of the centralized data processing server 7 one count of the unknown person managed by the storage unit 6000 and the image of the unknown person captured by the real-time data processing terminal 3. Photographing date and time information indicating the date and time of shooting is transmitted (step S335). As a result, the transmitting/receiving unit 71 of the centralized data processing server 7 receives one count of the unknown person and the shooting date/time information.

Next, in the centralized data processing server 7, the storage/readout unit 79 increments the number of unknown persons by 1 in the storage unit 7000, and stores the photographing date and time information (step S336).

On the other hand, in the distributed data processing terminal 6, the display control unit 67 causes the display 517 to display a message indicating that the judgment as to whether or not one person who has entered the room is an unknown person has been completed (step S337).

The shooting date/time information is also transmitted when the captured image data is transmitted from the real-time data processing terminal 3 to the distributed data processing terminal 6 in steps S112 and S113 shown in FIG. Also, in FIG. 24, the date and time the near-terminal data processing device 5 received the captured image data in step S112, or the date and time the near-terminal data processing device 5 transmitted the captured image data in step S113, instead of the date and time when the photograph was taken. There may be. Alternatively, in FIG. 24, the date and time when the distributed data processing terminal 6 received the captured image data in step S113 may be used instead of the date and time when the image was taken.

Further, the real-time data processing terminal 3 transmits position information indicating the position of the real-time data processing terminal 3 to the distributed data processing terminal 6 via the near-terminal data processing device 5 before the process of step S112. You can leave it. As a result, the distributed data processing terminal 6 can also transmit the location information in the process of step S335 described above. can be managed On the centralized data processing server 7 side, a terminal ID (Identification) for identifying the realtime data processing terminal 3 (or the image acquisition terminal 2) and a terminal ID (Identification) for identifying the realtime data processing terminal 3 (or the image acquisition terminal 2) are stored in the storage unit 7000. ) may be associated and managed, and the distributed data processing terminal 6 may transmit the terminal ID instead of the position information in the process of step S335. Thereby, the centralized data processing server 7 can also manage the installation positions of the distributed data processing terminals 6 .

<<main effects of the embodiment>>
As described above, according to the present embodiment, the image acquisition terminal 2 constituting the edge computing communication system performs processing only on its own terminal (image acquisition terminal 2) side in order to realize a series of image processing. A program for execution and a program for executing processing in cooperation with the service providing server 8 side may be executed. Even in this case, the image acquisition terminal 2 performs the first image processing on the image data by executing the first program that does not require authentication in order to execute the processing on its own terminal. By executing the second program that requires authentication to execute the processing on the own terminal and performing the second image processing on the image data, the time until the start of a series of image processing is reduced. The effect is that it can be shortened.

Further, as shown in FIG. 7, the real-time data processing terminal 3 extracts partial image data from the captured image data using the image recognition application AP1, and the near-terminal data processing apparatus 5 extracts partial image data using the communication application AP2. By transmitting to the distributed data processing terminal 6, load distribution can be performed. As a result, the real-time data processing terminal 3 can maintain a cycle value of 1/60 second for repeating real-time processing. In particular, even when a plurality of real-time data processing terminals are installed in the intranet 200, since the near-terminal data processing device 5 communicates with the distributed data processing terminal 6, each real-time data processing terminal This has the effect of maintaining a cycle value of 1/60 second for repeating time processing. In addition, by connecting the real-time data processing terminal 3 and the near-terminal data processing device 5 one-to-one at a position physically closer to the distributed data processing terminal 6 and the centralized data processing server 7, data can be exchanged. can be performed at high speed, the effect is that the processing time until finally obtaining the collation result can be shortened. In addition, even if the number of image sensors changes as shown in FIG. 3, the image acquisition terminal 2 can be prevented from being affected by the number of image sensors.

In addition, since the real-time data processing terminal 3 outputs partial image data instead of captured image data, the data transmitted from the image acquisition terminal 2 to the distributed data processing terminal 6 is not captured image data with a large amount of data. Only partial image data with a relatively small amount of data can be used. As a result, the amount of data flowing through the intranet 200 and the Internet 600 can be suppressed. Furthermore, the real-time data processing terminal 3 compresses the data volume by encoding the partial image data, thereby reducing the data size of the data transmitted from the near-terminal data processing device 5 to the distributed data processing terminal 6 via the intranet 200. can be reduced in quantity.

Also, when multiple faces are detected, as shown in FIG. Since it is possible to ensure that only one person's face is recognized, security can be enhanced by improving the accuracy of face recognition.

In the case of (processing the number of objects), the centralized data processing server 7 needs only half the object detection range. The frequency of transmitting partial image data can be reduced. As a result, the amount of data transmitted over the intranet 200 and the Internet 600 can be suppressed.

Further, since the number of unknown persons is stored in the centralized data processing server 7, the centralized data processing server 7 can be accessed from the distributed data processing terminal 6 or the like to confirm the number of unknown persons.

〔supplement〕
In each of the above embodiments, captured image data has been described, but the present invention is not limited to this, and may be created image data created by a user without being captured. In this case, the captured image data and the created image data are examples of "image data". again. The image acquisition terminal 2 is an example of a communication terminal, and image data may be created by itself without acquiring image data from the outside. Furthermore, the communication terminal may acquire sound data by sound collection, temperature data by a temperature sensor, and humidity data by a humidity sensor.

Further, each component such as each CPU 301, 501, 701 may be singular or plural. Further, there may be a plurality of image acquisition terminals 2, distributed data processing terminals 6, and centralized data processing servers 7, respectively. Further, distributed data processing terminal 6 may be a server.

Also, each function in the above-described embodiments can be realized by one or more processing circuits. Here, the "processing circuit" in this embodiment means a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or an ASIC designed to execute each function described above. Application Specific Integrated
Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU, and devices such as conventional circuit modules.

2 Image acquisition terminal (an example of a communication terminal)
3 Real-time data processing terminal 5 Near-terminal data processing device 6 Distributed data processing terminal 7 Centralized data processing server 34 Image processing unit (an example of image processing means)
35 Object detection unit 38 Connection unit (an example of acquisition means)
51 transmission/reception unit 61 transmission/reception unit 62 reception unit 67 display control unit 69 storage/readout unit 71 transmission/reception unit 74 feature amount generation unit 75 collation unit 200 intranet 600 Internet 3001 imaging device information management DB (an example of imaging device information management means)
3002 Cycle value management DB
3003 Image Acquisition Program DB
3004 Synthesis processing program management DB
3005 Distortion correction program management DB
3006 Service program management DB (an example of program management means)
6000 Storage unit 6001 Matching data management DB
8000 Storage unit 8001 Session management DB
8002 Authentication server management DB
9000 Storage unit 9001 Authentication management DB

Japanese translation of PCT publication No. 2016-519812

Claims (13)

A communication terminal that realizes a service provided by a service providing server through cooperation between processing by a service providing server that provides services via a communication network and processing by the terminal itself,
performing first image processing on image data by executing a first program that does not require authentication for executing processing in the own terminal, and performing authentication in order to execute processing in the own terminal; an image processing means for performing second image processing on the image data by executing a second program that requires
a search unit for searching whether a first program and a second program exist in the terminal for realizing the provided service;
a determination unit that determines existence of the first and second programs based on the searched result;
having a first processing unit and a second processing unit,
If the determining unit determines that at least one of the first and second programs does not exist in the first processing unit, the first processing unit determines the presence of the program to the second processing unit. Send request information indicating a request for a program that does not
The second processing unit transmits the received request information so that the service providing server receives it,
The second processing unit receives a program based on the request information transmitted by the service providing server, transmits the received program to the first processing unit,
the first processing unit installs the received program;
A communication terminal characterized by: The first program is at least one of an image acquisition program for acquiring predetermined image data, an image synthesizing program for synthesizing a plurality of image data, and a distortion correction program for correcting image distortion. and the second program is an object detection program for detecting a predetermined object from an image related to the image data or a predetermined object included in the image when receiving a predetermined service from a service providing server that provides the service. 2. The communication terminal according to claim 1, wherein the communication terminal is an object counting program for counting the number of objects. The image acquisition program is a first image acquisition program for acquiring image data obtained by imaging with a single imaging device, or a plurality of image acquisition programs obtained by imaging with a plurality of imaging devices. 3. The communication terminal according to claim 2, wherein the second image acquisition program is for acquiring the image data of the. The distortion correction program is a wide-angle distortion correction program for correcting image distortion associated with image data obtained by an image pickup device using a wide-angle lens, or an image associated with image data obtained by an image pickup device using a fisheye lens. 3. The communication terminal according to claim 2, wherein the program is a fish-eye distortion correcting program for correcting distortion of . image pickup device information management means for managing predetermined identification information for identifying a predetermined image pickup unit that captures an image of a subject and obtains image data, the number of image pickup devices included in the image pickup unit, and the type of lens in association with each other;
program management means for managing the number of image sensors or the type of the lens and the first program in association with each other, and managing the type of the lens and the second program in association with each other;
Acquisition means for acquiring specific identification information for identifying a specific imaging unit from a specific imaging unit that captures an image of a subject and obtains image data;
has
The image processing means includes a first program and a second program in which the number of image sensors and the type of lens corresponding to the specific identification information in the image sensor information management means correspond in the program management means. 2. The communication terminal according to claim 1, wherein the first image processing and the second image processing are performed by executing a communication terminal according to claim 1;
an authentication server that performs the authentication;
a service providing system that causes the authentication server to authenticate the communication terminal when providing the service to the communication terminal;
A communication system comprising:
receiving means for receiving, from the communication terminal, terminal identification information for identifying the communication terminal, including an authentication target portion used for user authentication and an authentication non-target portion not used for user authentication;
a transmission means for transmitting only the authentication target portion of the received terminal identification information to the authentication server;
A communication system characterized by comprising: A communication terminal that realizes a service provided by a service providing server through cooperation between processing by a service providing server that provides services via a communication network and processing by the terminal itself,
performing first image processing on image data by executing a first program that does not require authentication for executing processing in the own terminal, and performing authentication in order to execute processing in the own terminal; an image processing means for performing second image processing on the image data by executing a second program that requires
The first program is at least one of an image acquisition program for acquiring predetermined image data, an image synthesizing program for synthesizing a plurality of image data, and a distortion correction program for correcting image distortion. and
The second program is an object detection program for detecting a predetermined object from an image related to the image data or detecting a predetermined object included in the image when receiving a predetermined service from a service providing server that provides the service. A communication terminal characterized by an object counting program for counting numbers. A communication terminal that realizes a service provided by a service providing server through cooperation between processing by a service providing server that provides services via a communication network and processing by the terminal itself,
performing first image processing on image data by executing a first program that does not require authentication for executing processing in the own terminal, and performing authentication in order to execute processing in the own terminal; A predetermined image processing means for performing the second image processing on the image data by executing a second program that requires a predetermined image capturing unit for capturing an image of a subject and obtaining image data. image pickup device information management means for managing identification information, the number of image pickup devices included in the image pickup unit, and the type of lens in association with each other;
program management means for managing the number of image sensors or the type of the lens and the first program in association with each other, and managing the type of the lens and the second program in association with each other;
Acquisition means for acquiring specific identification information for identifying a specific imaging unit from a specific imaging unit that captures an image of a subject and obtains image data;
has
The image processing means includes a first program and a second program in which the number of image sensors and the type of lens corresponding to the specific identification information in the image sensor information management means correspond in the program management means. and performing the first image processing and the second image processing. a communication terminal that realizes the service provided by the service providing server through cooperation between the processing of the service providing server that provides the service via a communication network and the processing of the terminal itself;
an authentication server that performs authentication;
a service providing system that causes the authentication server to authenticate the communication terminal when providing the service to the communication terminal;
wherein the communication terminal performs first image processing on image data by executing a first program that does not require authentication in order to execute processing on the own terminal, having image processing means for performing second image processing on the image data by executing a second program that requires authentication in order to execute processing in the own terminal;
The service providing system receives, from the communication terminal, terminal identification information for identifying the communication terminal, including an authentication target portion used for user authentication and an authentication non-target portion not used for user authentication. receiving means for
a transmission means for transmitting only the authentication target portion of the received terminal identification information to the authentication server;
A communication system characterized by comprising: An image processing method executed by a communication terminal that obtains image data,
The communication terminal is
performing first image processing on the image data by executing a first program that does not require authentication for execution, and executing a second program that requires authentication for execution, performing second image processing on the image data;
The first program is at least one of an image acquisition program for acquiring predetermined image data, an image synthesizing program for synthesizing a plurality of image data, and a distortion correction program for correcting image distortion. and
The second program is an object detection program for detecting a predetermined object from an image related to the image data or detecting a predetermined object included in the image when receiving a predetermined service from a service providing server that provides the service. An image processing method characterized by being an object counting program for counting numbers. An image processing method executed by a communication terminal that obtains image data,
The communication terminal is
performing first image processing on the image data by executing a first program that does not require authentication for execution, and executing a second program that requires authentication for execution, performing second image processing on the image data;
Predetermined identification information for identifying a predetermined imaging unit that captures an image of a subject and obtains image data, the number of imaging elements possessed by the imaging unit, and the type of lens are associated and managed, and Associating and managing the lens type and the first program, associating and managing the lens type and the second program,
Acquiring specific identification information for identifying a specific imaging unit from a specific imaging unit that captures an image of a subject to obtain image data;
The first image processing and the An image processing method characterized by performing the second image processing. a communication terminal that realizes the service provided by the service providing server through cooperation between the processing of the service providing server that provides the service via a communication network and the processing of the terminal itself;
an authentication server that performs authentication;
a service providing system that causes the authentication server to authenticate the communication terminal when providing the service to the communication terminal;
A communication method by a communication system having
The communication terminal performs first image processing on image data by executing a first program that does not require authentication in order to execute processing in the own terminal, and performs processing in the own terminal. performing second image processing on the image data by executing a second program that requires authentication for execution;
The service providing system is
receiving, from the communication terminal, terminal identification information for identifying the communication terminal, including an authentication target portion used for user authentication and an authentication non-target portion not used for user authentication; A communication method, wherein only the part to be authenticated of the terminal identification information is transmitted to the authentication server. A program that causes a computer to perform the method according to claims 10-12.

Images