JP6686547B2 - Image processing system, program, image processing method - Google Patents

Description

  The present invention relates to an image processing system, a program, and an image processing method.

  It has been conventionally practiced that the manager of a retail store analyzes the behavior of the visitor and reflects it on the store operation and product planning. For example, an image of a camera installed in the store is used for the analysis, and the store manager or the like looks at the image in the store to understand how the visitor moves, where to stay, and the like. As a result, it is possible to understand the flow line of the visitor, the product that is easily attracted to the visitor, and the display method.

  A monitoring system has been devised that not only provides images of each location in the store but also processes images by combining a camera and image processing technology and outputs information that a store manager or the like can easily analyze. See 1.). In Patent Document 1, a person behavior analysis is performed in which the shop area is subdivided to form a plurality of sub-areas, and the flow line data of a person who has passed through the specified sub-area is extracted from each flow line data in the flow line database. A device is disclosed.

  If a person is detected from the image in this way, the detection result of the person is converted into a distribution expression such as a heat map (an analysis method that divides the target into squares and changes the display color of the square according to the degree of attention of each square). It is also possible to use it by adding it to the original video.

  By adopting a wide-angle camera with a wide angle of view as the surveillance camera, a single camera can cover a wide area, and thus an increase in cost can be suppressed. Therefore, a wide-angle camera is increasingly used as a surveillance camera.

  However, there are cases where desired information cannot be obtained even if a distribution expression such as a heat map is performed on a wide-angle image having a wide angle of view. For example, when there are no people on the ceiling but the ceiling is reflected in the wide-angle image, a heat map including the ceiling is created. In this case, people are much more likely to be detected in the aisle than in the ceiling (for example, only the aisle becomes bright red), so even if the heat map shows that there are no people on the ceiling, how many people are in the aisle I can't get the information I really want to know about

  Therefore, it is conceivable that the store manager or the like sets an area in the wide-angle image and detects a person or the like in this area. However, the wide-angle image is distorted three-dimensionally, and while store managers etc. look at this wide-angle image and analyze it, image analysis such as human detection is performed while the wide-angle image is converted into a flat image. However, there is a problem that it is difficult to apply the human detection result to the set area.

  In view of the above problems, it is an object of the present invention to provide an image processing system that can add additional information to a wide-angle image.

The present invention is an image processing system including one or more information processing devices, the display device displaying a wide-angle image having a wider angle of view than a predetermined value on a display device, and the wide-angle image displayed on the display device. Receiving means for accepting designation of a region, coordinate transforming means for transforming coordinates of the region in the display device into coordinates of a plane image when the wide-angle image is analyzed, coordinates included in an analysis result of the plane image, and Analysis means for creating information on the area based on the result of comparing the coordinates of the area converted by the coordinate conversion means .

  An image processing system that can add additional information to a wide-angle image can be provided.

It is an example of a diagram for explaining an outline of processing by the image processing system. It is an example of a schematic configuration diagram of an image processing system. It is an example of the hardware block diagram of an imaging device. It is an example of a hardware configuration diagram of a communication terminal in the case of a cradle having a wireless communication function. It is an example of a hardware configuration diagram of an image management device and an information terminal. It is an example of each functional block diagram of an imaging device, a communication terminal, an image management device, and an information terminal which constitute a part of image processing system. It is an example of a diagram for explaining the trajectory of the pointing device operated by the viewer Y to register the area. It is an example of the figure explaining the form of the image of each phase in which a spherical image is processed, and conversion of coordinates. It is an example of a flowchart showing a procedure for an information terminal to register an area designated by a viewer in an image management apparatus. FIG. 6 is an example of a diagram illustrating an area in which an end point straddles an end of a planar image. It is an example of a flow chart showing a procedure for an area inspection unit of the image management device to inspect an area. It is an example of the figure explaining an area and a person detection area. It is an example of a flow chart showing the processing procedure for the image management device to display the area. It is an example of an area registration screen displayed on the display by the information terminal when the area is registered. It is an example of a diagram illustrating an example of registration of the area. It is an example of a diagram for explaining the association of areas. It is a figure which shows an example of an area display screen. It is a figure which shows the example of a display of the area included in the other area. It is an example of a diagram for explaining the inspection result of the area.

Hereinafter, modes for carrying out the present invention will be described with reference to the drawings.
FIG. 1 is an example of a diagram for explaining an outline of processing by the image processing system of the present embodiment. The image processing system has an image pickup device capable of picking up an image of a 360 ° field of view. An image captured by this image capturing device will be referred to as a spherical image.

  (1) As shown in FIG. 1A, a viewer can operate the information terminal 7 to rotate, enlarge, or reduce the spherical image 10 to browse a desired display range 640 of the spherical image 10. . Since the spherical image 10 is a sphere, the display range 640 is strictly curved three-dimensionally, but since the display range 640 is a small part of the spherical image 10, it is hardly noticeable. The viewer designates an arbitrary area 602 from the omnidirectional image 10 while the omnidirectional image 10 has not been converted into a planar image. For example, diagonal vertices of this area 602 are specified by coordinates on the screen (hereinafter referred to as screen coordinates).

  (2) As shown in FIG. 1B, the information terminal 7 converts the screen coordinates indicating the area 602 into the world coordinate system (three-dimensional coordinates). This is because, although the screen is two-dimensional, the area 602 pointed to by the viewer Y is reflected on a curved surface, and therefore the position of the area 602 on this curved surface is accurately specified.

  (3) As shown in FIG. 1C, the information terminal 7 converts the three-dimensional coordinates of the area 602 into two-dimensional coordinates. The two-dimensional coordinates are coordinates in the coordinate system before the spherical image is converted into the plane image. Therefore, the position of the area 602 in the planar image 11 is specified.

  The plane image 11 is an image used when analyzing the person detection, and the person detection area (target area) in which the person is detected is also specified by the two-dimensional coordinates of the plane image 11. Therefore, it becomes possible to handle the human detection area and the area 602 set by the viewer in the same coordinate system, and it is determined whether or not a person exists in the area based on the relative position of the human detection area and the set area. .

  Further, when the information terminal 7 displays the area 602 again, the coordinates of the area 602 are converted into the same three-dimensional coordinates as the omnidirectional image 10, so that the area 602 set by the viewer Y is the same as that at the time of setting. Is displayed.

  As described above, the image processing system according to the present embodiment can suppress the cost increase by using the imaging device that captures the spherical image 10. Further, since the viewer arbitrarily sets the area 602 in wide image data such as the spherical image 10, the ceiling or the like can be excluded from the detection target. Even if the area 602 is set in the distorted image data like the omnidirectional image 10, the coordinates of the area 602 are converted to the coordinate system of the planar image 11 in which analysis (image processing) such as human detection is performed. It becomes easy to apply the analysis result such as human detection to the set area 602.

<About terms>
The wide-angle image having a wider angle of view than a predetermined angle means an image having a wider angle of view than at least the angle of view of a normal image pickup device (for example, a camera of 35 mm film). Alternatively, it refers to an image that is curved like an image captured by a fisheye lens. Preferably, the angle of view is 360 ° in the horizontal direction and 180 ° in the vertical direction, but the angle of view is not necessarily 360 ° in the horizontal direction and 180 ° in the vertical direction. In the present embodiment, a spherical image will be used as an example for description.

  Note that the imaging device 1 initially outputs a planar image, but the planar image is attached to the spherical image at the time of browsing to create the spherical image 10. In this embodiment, the area after designation of the area is converted into the coordinates of the planar image.

  The area is a part of the wide-angle image. It may also be referred to as an area, zone or range. The region may have any shape. In this embodiment, the term “area” will be used.

  The analysis of the wide-angle image means that some image analysis is performed on the wide-angle image. Image analysis is the detection and recognition of objects. Further, the target is a subject imaged in the spherical image. The subject that the viewer cancels may be the target. Further, the target is movable or portable because it may or may not be reflected in the spherical image. In addition, the object is something whose shape changes like a door, and whose state changes by turning on or off like a lamp. In the present embodiment, as an example, the target will be described as a person (pedestrian).

  The information on the area (area) means information on the area (area) created based on the analysis result of the wide-angle image. When the analysis result of the wide-angle image is the detection result of the target, the information on the area (area) is the information on the presence or absence of the target. Further, when the analysis result of the wide-angle image is the operation content of the target, the information about the area is information about how the target is performing. When the analysis result of the wide-angle image is the target state, the information about the area is the information about the target state. Further, the information regarding the area may be information obtained by processing the analysis result of the wide-angle image. For example, when there is a wide-angle image analysis result for a certain period of time, the information regarding the area is information obtained by processing the number of times the target is detected, the frequency, the number of times the state has changed, and the like.

<System configuration of image processing system>
FIG. 2 is an example of a schematic configuration diagram of the image processing system 200. The image processing system 200 includes an image management device 5, an imaging device 1, a communication terminal 3, and an information terminal 7, which are connected via a communication network 9. The image pickup apparatus 1 is installed in the store by the installer X. The information terminal 7 is operated by the viewer Y.

  The communication network 9 is constructed to include at least one of a LAN of a company to which the viewer Y belongs, a provider network of a provider connecting the LAN to the Internet, and a line provided by a line operator. There is. When the communication terminal 3 and the information terminal 7 are directly connected to the line telephone network or the mobile telephone network without going through the LAN, they can be connected to the provider network without going through the LAN. The communication network includes WAN and Internet. The communication network may be constructed by either wired or wireless, or may be a combination of wired and wireless.

  The image capturing apparatus 1 is a camera that captures 360 degrees of surroundings by one-time imaging as described above and creates a spherical image. It may be called a digital still camera or a digital video camera. When the communication terminal 3 has a camera, the communication terminal 3 can be a digital camera. In the present embodiment, in order to make the description easy to understand, the image pickup apparatus 1 will be described as a digital camera for obtaining a spherical image. The imaging device 1 periodically images the surrounding 360. The image management device 5 does not necessarily have to be periodic, and may be imaged irregularly, may be imaged by the operation of the installer X, or the viewer Y may request the image management device 5 so that the image management device 5 transmits the image. The image may be picked up by the command.

  Note that the image capturing apparatus 1 may automatically capture a number of landscapes with different lines of sight and create a spherical image by combining a plurality of image data.

  The communication terminal 3 has a communication function of connecting to the communication network 9 instead of the imaging device 1. The communication terminal 3 is a cradle for supplying power to the image pickup apparatus 1 and fixing it to a store. The cradle is an expansion device that expands the function of the image pickup apparatus 1. The communication terminal 3 has an interface for connecting to the imaging device 1, and thus the imaging device 1 can use the function of the communication terminal 3. The communication terminal 3 performs data communication with the imaging device 1 via this interface. Then, data communication is performed with the image management device 5 via the wireless router 9a and the communication network 9.

  If the imaging device 1 has a function of directly performing data communication with the wireless router 9a or the communication network 9, the communication terminal 3 may be omitted. Alternatively, the imaging device 1 and the communication terminal 3 may be integrally configured.

  The image management device 5 is, for example, an information processing device that functions as a server, and can perform data communication with the communication terminal 3 and the information terminal 7 via the communication network 9. The image management apparatus 5 has OpenGL ES (API: Application Interface for 3D graphics) installed. By calling OpenGL ES, you can create a spherical image from a Mercator image or a thumbnail image of a part of the spherical image (predetermined area image).

  Note that it is preferable that cloud computing is applied to the image management device 5. Although there is no strict definition in the physical configuration of cloud computing, the configuration of the information processing device is configured by dynamically connecting / disconnecting resources such as CPU, RAM, and storage that configure the information processing device according to the load. It is known that the installation location can be changed flexibly. In cloud computing, the image management device 5 is generally virtualized. One information processing device can provide the function as a plurality of image management devices 5 by virtualization, or a plurality of information processing devices can provide the function as one image management device 5 by virtualization. . The image management device 5 may be provided by a single information processing device instead of cloud computing.

  The information terminal 7 is, for example, a notebook PC (Personal Computer), and performs data communication with the image management device 5 via the communication network 9. The information terminal 7 may be a notebook PC, a tablet terminal, a PC, a PDA (Personal Digital Assistant), an electronic blackboard, a video conference terminal, a wearable PC, a game machine, a mobile phone, a car navigation system, a smartphone, or the like. Moreover, it is not limited to these.

  The imaging device 1, the communication terminal 3, and the wireless router 9a are installed at predetermined positions by the installer X at each sales base such as a store. The information terminal 7 is installed in a head office or the like that controls each sales base, and displays an image showing the status of each base sent via the image management device 5, so that the viewer Y can display the status of each base. You can browse the image showing. However, the information terminal 7 can communicate with the image management device 5 from a place other than the head office.

<Hardware configuration of the embodiment>
Next, the hardware configurations of the imaging device 1, the communication terminal 3, the information terminal 7, and the image management device 5 according to the present embodiment will be described with reference to FIGS. 3 to 5.

<< Imaging device >>
FIG. 3 is an example of a hardware configuration diagram of the image pickup apparatus 1. In the following, the image pickup apparatus 1 is an omnidirectional image pickup apparatus using two image pickup elements, but the number of image pickup elements may be three or more. Further, the device does not necessarily have to be a device dedicated to the omnidirectional image pickup, and may have the same function as that of the image pickup device 1 by attaching an omnidirectional image pickup unit which is attached later to an ordinary digital camera, a smartphone or the like. .

  As shown in FIG. 3, the image pickup apparatus 1 includes an image pickup unit 101, an image processing unit 104, an image pickup control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only Memory). ) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, and antenna 117a.

  Of these, the image pickup unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having a field angle of 180 ° or more for forming a hemispherical image, and two image pickups provided corresponding to the respective wide-angle lenses. It has elements 103a and 103b. The image pickup devices 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image formed by a fish-eye lens into image data of an electric signal and output the image data. It has a timing generation circuit for generating a vertical synchronizing signal, a pixel clock, etc., a register group for setting various commands and parameters necessary for the operation of this image sensor, and the like.

  The image pickup devices 103a and 103b of the image pickup unit 101 are each connected to the image processing unit 104 by a parallel I / F bus. On the other hand, the image pickup devices 103a and 103b of the image pickup unit 101 are connected by a serial I / F bus (I2C bus or the like) separately from the image pickup control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Further, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, electronic compass 118, and the like.

  The image processing unit 104 takes in image data output from the image pickup devices 103a and 103b through a parallel I / F bus, performs a predetermined process on each image data, and then performs a combining process on these image data. , Create so-called Mercator image data.

  The image capturing control unit 105 generally sets the image capturing control unit 105 as a master device and the image capturing elements 103a and 103b as slave devices, and sets a command or the like in a register group of the image capturing elements 103a and 103b using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the imaging control unit 105 also uses the I2C bus to fetch status data and the like of the register group of the imaging elements 103a and 103b and send it to the CPU 111.

  Further, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some image pickup apparatuses 1 may have a preview display function on a display and a function corresponding to moving image display. In this case, the image data is output from the image pickup devices 103a and 103b continuously at a predetermined frame rate (frames / minute).

  The image pickup control unit 105 also functions as a synchronization control unit that synchronizes the output timing of the image data of the image pickup devices 103a and 103b in cooperation with the CPU 111, as described later. In addition, in the present embodiment, the display unit is not provided in the imaging device 1, but the display unit may be provided.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs a predetermined process on the sound data.

  The CPU 111 controls the entire operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data in the process of being processed by the image processing unit 104 and data of a processed Mercator image.

  The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, and a touch panel having both display and operation functions. The user operates the operation buttons to input various imaging modes, imaging conditions, and the like.

  The network I / F 116 is a general term for interface circuits (USB I / F, etc.) with external media such as SD cards and personal computers. Further, the network I / F 116 may be a network interface regardless of whether it is wireless or wired. The data of the Mercator image stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or is stored in the communication terminal 3 or the like via the network I / F 116 which becomes the network I / F as necessary. It is sent to an external device.

  The communication unit 117 communicates with an external device such as the communication terminal 3 via an antenna 117a provided in the imaging device 1 by a wireless communication technology such as WiFi (wireless fidelity), NFC, or LTE (Long Term Evolution). To do. The communication unit 117 can also transmit the data of the Mercator image to the external device of the communication terminal 3.

  The electronic compass 118 calculates the azimuth and tilt (Roll rotation angle) of the imaging device 1 from the magnetism of the earth, and outputs the azimuth / tilt information. This azimuth / tilt information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. It should be noted that the related information also includes the data of the image capturing date and time and the data capacity of the image data.

<< communication terminal >>
Next, the hardware configuration of the communication terminal 3 will be described with reference to FIG. 4 is a hardware configuration diagram of the communication terminal 3 in the case of a cradle having a wireless communication function.

  As shown in FIG. 4, the communication terminal 3 includes a CPU 301 that controls the overall operation of the communication terminal 3, a ROM 302 that stores a basic input / output program, and a RAM (Random Access Memory) 304 that is used as a work area of the CPU 301. , A communication unit 305 for data communication by Wi-fi, NFC, LTE, etc., a USB I / F 303 for wired communication with the image pickup apparatus 1, and an RTC (Real Time Clock) 306 for holding calendar and time information. There is.

  Further, a bus line 310 such as an address bus or a data bus for electrically connecting the above-mentioned respective parts is provided.

  The ROM 302 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

  The communication unit 305 communicates with the wireless router 9a or the like by a signal of wireless communication using the antenna 305a.

  In addition to the drawing, a GPS (Global Positioning Systems) satellite or a GPS receiving unit for receiving a GPS signal including position information (latitude, longitude, and altitude) of the communication terminal 3 by an IMES (Indoor MEssaging System) as an indoor GPS is provided. May be.

<< Image management device, information terminal >>
The hardware configuration of the image management device 5 and the information terminal 7 in the case of the notebook PC will be described with reference to FIG. Note that FIG. 5 is a hardware configuration diagram of the image management device 5 and the information terminal 7. Since both the image management device 5 and the information terminal 7 are computers, the configuration of the image management device 5 will be described below. The configuration of the information terminal 7 is the same as that of the image management apparatus 5, and even if there is a difference, it does not hinder the description of the present embodiment.

  The image management apparatus 5 includes a CPU 501 that controls the overall operation of the image management apparatus 5, a ROM 502 that stores a program used to drive the CPU 501 such as an IPL, and a RAM 503 that is used as a work area of the CPU 501. Further, it has an HD 504 that stores various data such as a program for the image management device 5, and an HDD (Hard Disk Drive) 505 that controls reading or writing of various data to the HD 504 under the control of the CPU 501. Further, it has a media drive 507 for controlling the reading or writing (storage) of data with respect to the recording medium 506 such as a flash memory, and a display 508 for displaying various information such as a cursor, a menu, a window, characters, or an image. A touch panel is preferably attached to the display 508. In addition, a network I / F 509 for data communication using the communication network 9, a keyboard 511 having a plurality of keys for inputting characters, numerical values, various instructions, selection and execution of various instructions, processing target It has a mouse 512 for selecting and moving a cursor. Further, it has a CD-ROM drive 514 that controls reading or writing of various data from or to a CD-ROM (Compact Disc Read Only Memory) 513 as an example of a removable recording medium. Further, as shown in FIG. 11, a bus line 510 such as an address bus or a data bus for electrically connecting the above components is provided.

<Functions of image processing system>
FIG. 6 is a functional block diagram of the imaging device 1, the communication terminal 3, the image management device 5, and the information terminal 7, which form a part of the image processing system 200.

<< Functional configuration of the imaging device 1 >>
The image pickup apparatus 1 has a reception unit 12, an image pickup unit 13, a sound collection unit 14, a connection unit 15, and a storage / readout unit 19. Each of these units is a function or means realized by any one of the components shown in FIG. 3 being operated by an instruction from the CPU 111 according to a program for the image pickup device expanded from the SRAM 113 to the DRAM 114. Is.

  The image pickup apparatus 1 also includes a storage unit 1000 constructed by one or more of the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. The storage unit 1000 stores a program for the imaging device and a terminal ID.

  The accepting unit 12 of the imaging device 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 illustrated in FIG. 3, and accepts an operation input from the user (installer X in FIG. 2). It should be noted that the imaging device 1 automatically and regularly images the surroundings even if the installer X does not perform an operation for imaging. The regular intervals may be set by the installer X in the imaging device 1 or may be set by the viewer Y via the image management device 5.

  The image pickup unit 13 is mainly realized by the processing of the image pickup unit 101, the image processing unit 104, the image pickup control unit 105, and the CPU 111 shown in FIG. 3, picks up a landscape or the like, and creates image data.

  The sound collecting unit 14 is mainly realized by the processing of the microphone 108, the sound processing unit 109, and the CPU 111 shown in FIG. 3, and picks up the sound around the imaging device 1.

  The connection unit 15 is mainly realized by the processing of the network I / F 116 and the CPU 111, receives power supply from the communication terminal 3, and performs data communication with the communication terminal 3.

  The storage / readout unit 19 is mainly realized by the processing of the CPU 111 shown in FIG. 3, and stores various data in the storage unit 1000 and reads various data from the storage unit 1000. In the following, the description “via the storage / readout unit 19” may be omitted even when the imaging device 1 reads and writes from the storage unit 1000.

<< Functional configuration of communication terminal 3 >>
The communication terminal 3 includes a transmission / reception unit 31, a reception unit 32, a connection unit 33, and a storage / readout unit 39. Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 4 being operated by an instruction from the CPU 301 according to a program for a communication terminal expanded from the ROM 302 to the RAM 304. Is.

  Further, the communication terminal 3 has a storage unit 3000 constructed by the ROM 302 and the RAM 304 shown in FIG. The storage unit 3000 stores programs for communication terminals.

(Each functional configuration of the communication terminal 3)
The transmission / reception unit 31 of the communication terminal 3 is mainly realized by the processing of the communication unit 305 and the CPU 301 shown in FIG. 4, and transmits / receives various data to / from the image management device 5 via the wireless router 9a and the communication network 9. I do. Note that in the following, the description “via the transmission / reception unit 31” may be omitted even when the communication terminal 3 communicates with the image management device 5.

  The reception unit 32 is realized mainly by the processing of the CPU 301 shown in FIG. 4, and receives an operation input from a user (installer X in FIG. 2).

  The connection unit 33 is mainly realized by the processing of the USB I / F 303 and the CPU 301 illustrated in FIG. 4, supplies power to the imaging device 1 and performs data communication.

  The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 4, and stores various data in the storage unit 3000 and reads various data from the storage unit 3000. In the following, the description “via the storage / reading unit 39” may be omitted even when the communication terminal 3 reads and writes from the storage unit 3000.

<< Functional configuration of image management device >>
The image management device 5 includes a transmission / reception unit 51, a person detection unit 52, a thumbnail creation unit 53, a screen creation unit 54, an area registration unit 55, an area inspection unit 56, and a storage / readout unit 59. Each of these units is realized by any one of the components shown in FIG. 5 being operated by an instruction from the CPU 501 according to the device program 5010 for the image management device, which is expanded from the HD 504 on the RAM 503. Function or means.

  Further, the image management device 5 has a RAM 503 shown in FIG. 5 and a storage unit 5000 constructed by the HD 504. A base management DB 5001, an imaging management DB 5002, an image management DB 5003, a thumbnail management DB 5004, an area information management DB 5005, and an analysis information management DB 5006 are constructed in this storage unit 5000. Each database will be described below.

表１は、拠点管理ＤＢ５００１に記憶される各情報をテーブル状に示す拠点管理テーブルである。拠点管理テーブルでは、地域ＩＤ、地域名、拠点ＩＤ、拠点名、拠点レイアウトマップ、及び、装置ＩＤの各フィールドが関連付けて記憶されている。また、拠点管理テーブルの１つの行をレコードという場合がある。以下の各テーブルでも同様である。このうち、地域ＩＤは、地域を識別するための識別情報である。地域ＩＤの一例としては重複しない番号とアルファベットの組み合わせが挙げられる。

Table 1 is a base management table showing each information stored in the base management DB 5001 in the form of a table. In the base management table, fields of region ID, region name, base ID, base name, base layout map, and device ID are stored in association with each other. Further, one row of the site management table may be called a record. The same applies to the following tables. Of these, the area ID is identification information for identifying the area. An example of the area ID is a combination of a unique number and an alphabet.

  The area name indicates an area or range of land such as Kanto, Tokyo, Shibuya Ward, New York State, and New York City. It may be called the area name. The identification information refers to a name, a code, a character string, a numerical value, or a combination of two or more of these, which is used to uniquely distinguish a specific target from a plurality of targets. The same applies to the following IDs or identification information.

  The base ID is an example of identification information for identifying the base. The base ID is assigned to the base name so as not to overlap. It may be referred to as site-specific information. An example of the base ID is a combination of a unique number and an alphabet. The base refers to a place where the imaging device 1 is installed and where the surrounding area is not imaged. An example of a base is a store.

  The base name is a name of a base such as a store name such as a Shibuya store or a venue name such as a Shibuya venue. File names such as image data showing the layout and map of each site are registered in the site layout map. The location of the image pickup apparatus 1 and the products handled at the location is specified by the two-dimensional coordinates by the location layout map.

  The terminal ID is identification information for identifying the imaging device 1. It may be referred to as terminal-specific information. The terminal ID is, for example, a serial number, a manufacturing number, a numerical value that does not overlap with the model number, an IP address, a MAC address, or the like of the imaging device 1, but is not limited thereto. As shown in Table 1, one or more imaging devices (terminal IDs) are installed in one base, and their positions are managed by a layout map or the like. The site management table may be registered by the installer X or the viewer Y, or may be registered by the supplier of the image processing system 200.

表２は、撮像管理ＤＢ５００２に記憶される各情報をテーブル状に示す撮像管理テーブルである。撮像管理テーブルでは、拠点ＩＤごとに、撮像タイトル、撮像開始日時、及び撮像終了日時の各フィールドが関連付けて記憶されている。撮像タイトルは、閲覧者Ｙが入力したイベントのタイトルである。つまり、閲覧者Ｙが消費者の行動を監視したい何らかのイベントが店舗で催される場合に、このイベントの名称が撮像タイトルとなる。当然ながら、イベントの名称は閲覧者Ｙが任意に付与できるためイベントの名称でなくてもよい。例えば、単に撮像年月日とすることもできる。閲覧者Ｙは、画像データの複数のファイルから所望の画像データを抽出する際に、撮像タイトルを参照することができる。なお、１回の撮像イベントで複数の画像データが時系列に（定期的に）撮像される。撮像開始日時は、閲覧者Ｙによって入力された日時であり、撮像装置１が撮像を開始する（又は開始した）日時を示す。撮像終了日時は、閲覧者Ｙによって入力された日時であり、撮像装置１が撮像を終了する（又は終了した）日時を示す。閲覧者Ｙは撮像開始日時と撮像終了日時を事前に登録しておくこともできる（予約撮像）。撮像管理テーブルは、主に画像管理装置５が登録する。

Table 2 is an imaging management table showing each information stored in the imaging management DB 5002 in a table form. In the imaging management table, fields of an imaging title, an imaging start date and time, and an imaging end date and time are associated and stored for each base ID. The imaging title is the title of the event input by the viewer Y. That is, when some event that the viewer Y wants to monitor the behavior of the consumer is held at the store, the name of this event becomes the imaging title. Of course, the name of the event need not be the name of the event because the viewer Y can arbitrarily give it. For example, it may be simply the image pickup date. The viewer Y can refer to the image capture title when extracting desired image data from a plurality of files of image data. It should be noted that a plurality of image data is imaged in time series (periodically) in one imaging event. The image capturing start date and time is the date and time input by the viewer Y, and indicates the date and time when the image capturing apparatus 1 starts (or starts) image capturing. The image capturing end date and time is the date and time input by the viewer Y, and indicates the date and time when the image capturing apparatus 1 ends (or ends) image capturing. The viewer Y can also register the image capturing start date and time and the image capturing end date and time in advance (reserved image capturing). The image management device 5 mainly registers the imaging management table.

表３は、画像管理ＤＢ５００３に記憶される各情報をテーブル状に示す画像管理テーブルである。画像管理テーブルでは、端末ＩＤごとに、画像ＩＤ、画像データのファイル名、及び撮像日時が関連付けて記憶されている。画像ＩＤは、画像データを一意に識別するための識別情報の一例である。画像固有情報と称してもよい。画像データのファイル名は、画像ＩＤで特定される画像データのファイル名である。撮像日時は画像データが端末ＩＤで示される撮像装置１で撮像された日時である。画像データも、記憶部５０００に記憶されている。

Table 3 is an image management table showing each information stored in the image management DB 5003 in a table form. In the image management table, an image ID, a file name of image data, and a shooting date and time are stored in association with each other for each terminal ID. The image ID is an example of identification information for uniquely identifying the image data. It may be referred to as image-specific information. The file name of the image data is the file name of the image data specified by the image ID. The image capturing date and time is the date and time when the image data was captured by the image capturing device 1 indicated by the terminal ID. The image data is also stored in the storage unit 5000.

  For example, when browsing an image file, the viewer Y accesses the image management device 5 with the information terminal 7 and selects a shooting title from the shooting management table in Table 2. The image management device 5 can read the terminal ID associated with the site ID of the imaged title from the site management table in Table 1. Since the terminal ID is clarified, the image management apparatus 5 can specify the image data in which the image capturing date and time is included in the image capturing start date and time from the image capturing start date and time among the image data associated with the terminal ID in the image management table.

  Of course, the viewer Y can directly specify the terminal ID and the site ID. In the present embodiment, for simplicity, a mode in which the viewer Y specifies a terminal ID and browses will be mainly described. The image management table is mainly registered by the image management device 5.

表４は、サムネイル管理ＤＢ５００４に記憶される各情報をテーブル状に示すサムネイル管理テーブルである。サムネイルとは親指程度のという意味であり、サムネイル画像は縮小した、画素数を低減した又は一覧用のイメージデータという意味になる。

Table 4 is a thumbnail management table showing each information stored in the thumbnail management DB 5004 in the form of a table. A thumbnail means a thumb level, and a thumbnail image means a reduced size, a reduced number of pixels, or a list of image data.

  In the thumbnail management table, thumbnail IDs, file names of thumbnail images, and predetermined area information are stored in association with each other for each image ID. The thumbnail ID is an example of identification information for uniquely identifying the thumbnail image created based on the image data indicated by the image ID. It may be referred to as thumbnail specific information. The file name of the thumbnail image is the file name of the thumbnail image indicated by the thumbnail ID. The image management device 5 gives the file name of the thumbnail image. The predetermined area information indicates a predetermined area in which the thumbnail image is created in the image data indicated by the image ID. The thumbnail management table is mainly registered by the image management device 5.

表５は、エリア情報管理ＤＢ５００５に記憶される各情報をテーブル状に示すエリア情報テーブルである。エリア情報テーブルでは、端末ＩＤごとに、登録エリアＩＤ、登録エリアの名称、値、及び、関連づけの各フィールドが関連付けて記憶されている。端末ＩＤは上記と同じものである。登録エリアＩＤは登録されたエリアを一意に識別するための識別情報の一例である。エリア固有情報と称してもよい。登録エリアの名称は、閲覧者Ｙが任意に付与することができるエリアの名称である。値は、エリアを特定するための情報であり、エリアが矩形であれば例えば、左上のコーナーの座標（x,y）と幅（width）と高さ（height）である。対角の２点の座標でもよい。エリア情報テーブルの値は、平面画像の座標系に基づいて決定されている。関連づけのフィールドにはエリアに関連付けられた他のエリアの登録エリアＩＤが登録される。例えば、表５のarea_1はarea_3と関連付けられている。エリアの関連づけは閲覧者Ｙが任意に行うことができる。また、３つ以上のエリアを関連付けることもできる。

Table 5 is an area information table showing each information stored in the area information management DB 5005 in the form of a table. In the area information table, registration area IDs, registration area names, values, and association fields are stored in association with each other for each terminal ID. The terminal ID is the same as above. The registration area ID is an example of identification information for uniquely identifying the registered area. It may be referred to as area-specific information. The name of the registered area is the name of the area that the viewer Y can arbitrarily assign. The value is information for specifying the area, and if the area is a rectangle, for example, the coordinates (x, y), the width (width), and the height (height) of the upper left corner. The coordinates of two diagonal points may be used. The value of the area information table is determined based on the coordinate system of the plane image. The registered area ID of another area associated with the area is registered in the association field. For example, area_1 in Table 5 is associated with area_3. The viewer Y can arbitrarily associate the areas. Also, three or more areas can be associated.

  The coordinates of each area may overlap. Two areas (area_A, area_B) can be registered so as to have an inclusive relationship (area_A⊃area_B). As a result, when the viewer Y wants to analyze the presence / absence of a person in a specific area (area_B) in the area_A, by registering an area having an inclusive relationship, the area_B can be further added to the number of people detected in the area_A. The number of people detected in can be grasped. Further, the viewer Y registers a plurality of area_B with respect to area_A, so that a heat map can be created by excluding a ceiling or the like where there are no people. In this way, it is possible to limit the parameter to be analyzed for the entire image, and it is possible to perform analysis by removing unnecessary information.

表６は、解析情報管理ＤＢ５００６に記憶される各情報をテーブル状に示す解析情報テーブルである。解析情報テーブルでは、画像ＩＤごとに、人検知エリアＩＤ、及び値が関連づけて管理されている。画像ＩＤは上記と同じである。人検知エリアＩＤは人が検知されたエリアを一意に識別するための識別情報の一例である。人検知エリア固有情報と称してもよい。値は、人検知エリアを特定するための情報である。データの形式は登録エリアと同様である。例えば、画像ＩＤ＝au123456ifauyの全天球画像（全天球画像に変換される前の平面画像）には２つの人検知エリアがあり、それぞれの値が登録されている。

Table 6 is an analysis information table showing each information stored in the analysis information management DB 5006 in the form of a table. In the analysis information table, a human detection area ID and a value are managed in association with each other for each image ID. The image ID is the same as above. The person detection area ID is an example of identification information for uniquely identifying the area in which a person is detected. It may be referred to as person detection area specific information. The value is information for specifying the human detection area. The data format is the same as the registration area. For example, the celestial sphere image with the image ID = au123456ifauy (two-dimensional image before conversion into a celestial sphere image) has two human detection areas, and their respective values are registered.

(Functional configuration of image management device)
The transmission / reception unit 51 of the image management device 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 5, and the like, and communicates with the communication terminal 3 or the information terminal 7 and various data via the communication network 9. Send and receive. In the following description, the description “via the transmitting / receiving unit 51” may be omitted even when the image management device 5 communicates with the information terminal 7.

  The person detection unit 52 is mainly realized by the processing of the CPU 501 shown in FIG. 5 and the like, and detects a person from image data. The detection of this person is performed by setting an appropriate feature amount and setting the learning identification device. For example, Haar-like features, LBP (Local Binary Patterns) features, HOG (Histogram of Oriented Gradients) features, and the like are known as feature amounts. Known learning and identifying devices include SVM (Support Vector Machines) and cascade classification using AdaBoost. However, the present invention is not limited to these, as long as it can detect a person.

  The thumbnail creating unit 53 is realized mainly by the processing of the CPU 501 shown in FIG. 5 and the like, and creates a thumbnail image of a predetermined area image.

  The screen creation unit 54 is realized by the processing of the CPU 501 shown in FIG. 5, and when transmitting the image data to the information terminal 7, the information terminal 7 uses the HTML data, JavaScript (registered trademark), CSS, etc. Create screen information for displaying. Since the information for specifying the area is described in the coordinate system of the planar image, when transmitting the screen data for displaying the area, the screen creation unit 54 performs a process of converting the coordinates of the area into the coordinates of the world coordinate system. To do.

  The area registration unit 55 is realized by the processing of the CPU 501 shown in FIG. 5, and the like, acquires information to be registered in the area information table from the information terminal 7, and registers it in the area information management DB 5005.

  The area inspection unit 56 is realized by the processing of the CPU 501 shown in FIG. 5, and the like, inspects the area using the analysis information, and creates information about the area.

  The storage / readout unit 59 is mainly realized by the processing of the HDD 505 and the CPU 501 shown in FIG. 5, stores various data in the storage unit 5000, and reads various data from the storage unit 5000. In the following, the description “via the storage / readout unit 59” may be omitted even when the image management device 5 reads and writes from the storage unit 5000.

<Functional configuration of information terminal 7>
The information terminal 7 includes a transmission / reception unit 71, a reception unit 72, a display control unit 73, an area coordinate calculation unit 74, and a storage / readout unit 79. Each of these units is a function realized by any of the components shown in FIG. 5 being operated by an instruction from the CPU 501 according to a program for the information terminal 7 expanded from the HD 504 on the RAM 503, or It is a means.

  Further, the information terminal 7 has a RAM 503 shown in FIG. 5 and a storage unit 7000 constructed by the HD 504. The storage unit 7000 stores a terminal program 7010 for the information terminal 7. The terminal program 7010 for the information terminal 7 is, for example, browser software, but it may be application software having a communication function such as browser software. Further, since the browser software executes the screen data to display the screen and accept operations, the terminal program 7010 includes screen data (HTML, JavaScript (registered trademark), etc.) transmitted by the image management device 5. Be done.

(Each functional configuration of the information terminal 7)
The transmission / reception unit 71 of the information terminal 7 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 5, and transmits / receives various data to / from the image management device 5 via the communication network 9. In the following, even when the information terminal 7 communicates with the image management device 5, the description “via the transmitting / receiving unit 71” may be omitted.

  The reception unit 72 is realized mainly by the processing of the keyboard 511 and the mouse 512 shown in FIG. 5, the CPU 501, and the like, and receives the operation input from the user (browser Y in FIG. 2).

  The display control unit 73 is mainly realized by the processing of the CPU 501 shown in FIG. 5 and the like, interprets the screen information transmitted from the image management device 5, and causes the display 508 of the information terminal 7 to display various screens. Control for.

  The area coordinate calculation unit 74 converts the coordinates on the screen that specify the area received by the reception unit 72 into the coordinates of the planar image. That is, conversion to the world coordinate system and conversion of a plane image into two-dimensional coordinates are performed.

  The storage / readout unit 79 is mainly realized by the processing of the HD 504 and the CPU 501 shown in FIG. 5, stores various data in the storage unit 7000, and reads various data from the storage unit 7000. In the following, the description “via the storage / readout unit 79” may be omitted even when the information terminal 7 reads and writes from the storage unit 7000.

<Registration of area>
Area registration will be described with reference to FIG. 7. FIG. 7 is an example of a diagram for explaining the trajectory of the pointing device operated by the viewer Y to register the area. The viewer Y operates the keyboard 511, the mouse 512, or the pointing device such as the touch panel of the information terminal 7 and drags from a point to a point. The accepting unit 72 of the information terminal 7 accepts this operation and acquires the locus represented by the arrow as shown in FIG. The coordinates shown in FIG. 7 are screen coordinates on the screen when the spherical image is displayed on the display 508 as it is.

  An example of an operation for registering an area is an operation of dragging an arbitrary place using a pointing device. When the display 508 has a touch panel, the registration may be accepted by touch dragging or pinch operation. In these cases, the reception unit 72 acquires the coordinates of the drag start point 651 and the drag end point 652 during the operation. In FIG. 7A, the start point 651 is the upper left apex (upperLeft) of the area 602 and the end point 652 is the lower right apex (lowerRight) of the area 602, but the start point 651 and the end point 652 are the area 602 depending on the dragging direction. Which vertex of the is different.

  Further, when the viewer Y drags from the lower right to the upper left as shown in FIG. 7B, the coordinates are smaller at the end point than at the start point. In this case, the area coordinate calculation unit 74 determines the drag direction from the coordinates of the locus up to the end point 652 with the start point 651 as a reference. When it is determined that the dragging direction is the upper left on the trajectory, the width of the negative value and the height of the negative value are detected with respect to the start point 651.

  When it is determined that the dragging direction is the lower left as shown in FIG. 7C, the width of the negative value and the height of the positive value with respect to the start point 651 are detected. When it is determined that the dragging direction is the upper right as shown in FIG. 7D, the width of the positive value and the height of the negative value with respect to the start point 651 are detected.

  The reception unit 72 detects the dragging direction of the viewer Y, and detects the origin (in the case of the start point, in the case of the end point, in other cases), the width (lrx1-ulx1) or (urx1-llx1), and the height ( lry1-uly1) (lly1-ury1) is calculated. The origin is assumed to be the top left corner of the area. For example, if the width of the negative value and the height of the negative value are added to the starting point 651 in FIG. 7B, the coordinates of the origin (upper left apex) are obtained. The same applies to FIGS. 7C and 7D.

  By recording the width and height with respect to the origin instead of the coordinates of the two points of the diagonal vertices, the area can be appropriately designated even in the omnidirectional image. That is, since the spherical image has no boundaries, there are two types of area designation in the area designation by two points (the area from the upper left apex to the lower right apex through the area, the lower right apex to outside the area to the upper left apex). This is because the area leading to In this way, as processing unique to the omnidirectional image, the reception unit 72 can appropriately calculate the information for identifying the area by absorbing the drag direction even if the drag direction is various.

  Note that the shape of the area does not have to be rectangular, and the area may be specified by a region surrounded by an arbitrary locus of the pointing device. In this case, the area is specified by time series coordinates. Also, the area may be specified from the coordinates when the viewer Y points any two points with the pointing device. In this case, it is preferable that the viewer Y follows a rule such as pointing to the lower right vertex after the upper left vertex.

<About coordinate conversion>
FIG. 8 is an example of a diagram for explaining the conversion of image form and coordinates in each phase in which a spherical image is processed. FIG. 8A shows a conversion procedure.
-Screen coordinate system to world coordinate system This transformation uses view transformation and projective transformation. In general, since the world coordinate system is converted to the screen coordinate system, the view transform and the inverse transform of the projective transform are applied in this embodiment. The view conversion is a conversion for rotating or translating the coordinates of the virtual camera, and corresponds to a process for converting a three-dimensional object into an image corresponding to the orientation of the camera. The transformation matrix Mv for view transformation is as follows.

ここでＸｘ、Ｙｘ、ＺｘはカメラのＸ軸を、Ｘｙ、Ｙｙ、ＺｙはカメラのＹ軸を、Ｘｚ、Ｙｚ、ＺｚはカメラのＺ軸を示し、Ｐはカメラの位置ベクトル、Ｘ，Ｙ，ＺはＸ軸、Ｙ軸、Ｚ軸の方向余弦を示す。

Here, Xx, Yx, and Zx represent the X axis of the camera, Xy, Yy, and Zy represent the Y axis of the camera, Xz, Yz, and Zz represent the Z axis of the camera, P represents the position vector of the camera, and X, Y, and Z indicates the direction cosine of the X axis, the Y axis, and the Z axis.

  Next, in the projective transformation, a three-dimensional object is transformed into two-dimensional coordinates when it is projected on a screen at a predetermined distance from the camera. FIG. 8B is a diagram illustrating the projective transformation. The distance from the camera to the screen m2 is N, and the range in which an object can exist is the distance F to the surface m1. θ is an angle of view in the width direction of the screen of the information terminal 7. From these, the projective transformation matrix Mp is as follows. It should be noted that the distances N and F are set to predetermined values.

したがって、スクリーン座標における全天球画像の任意の点の座標（ｘ_ｃ、ｙ_ｃ、ｚ_ｃ）はワールド座標系の座標（ｘ_ｗ、ｙ_ｗ、ｚ_ｗ）に以下のように変換される。ｚ_ｃは以下の式で算出される。このＺはオブジェクトまでの距離である。
z_c＝｛（Ｎ＋Ｆ）Ｚ−２ＮＦ｝／（Ｎ−Ｆ）

Therefore, the coordinates (x _c , y _c , z _c ) of an arbitrary point in the spherical image in the screen coordinates are converted into the coordinates (x _w , y _w , z _w ) in the world coordinate system as follows. z _c is calculated by the following formula. This Z is the distance to the object.
z _c = {(N + F) Z-2NF} / (NF)

・ワールド座標系からｕｖ座標系
この変換はワールド座標系の全天球画像を正規化することで行われる。正規化とは座標の範囲が（0, 0) から (1, 1)に含まれるようにすることをいう。ｕｖ座標系の座標を（u, v）とする。また、仮想的なカメラからの距離ｒ、距離ｒに対するｘ_ｗ、ｙ_ｗ、ｚ_ｗの比をｄｘ、ｄｙ、ｄｚとする。
r = sqrt(x ^2, + y^2 + z^2)
dx = x/r
dy = y/r
dz = z/r
以上からｕ座標は以下のようになる。
u = 1/2 + arctan(dz/dx) / 2*PI + 1/4
但し、u ≧ 1のとき u=ｕ-1とする。

-World coordinate system to uv coordinate system This conversion is performed by normalizing the spherical image in the world coordinate system. Normalization means that the range of coordinates is included from (0, 0) to (1, 1). The coordinates of the uv coordinate system are (u, v). Further, the distance r from the virtual camera and the ratio of x _w , y _w , and z _w to the distance r are dx, dy, and dz.
r = sqrt (x ^ 2, + y ^ 2 + z ^ 2)
dx = x / r
dy = y / r
dz = z / r
From the above, the u coordinate is as follows.
u = 1/2 + arctan (dz / dx) / 2 * PI + 1/4
However, when u ≧ 1, u = u−1.

Further, the v coordinate is as follows.
v = 1/2-arcsin (dy) / PI
Calculate the product of the coordinate system normalized uv coordinates of the plane image from the uv coordinate system and the resolution of the plane image.
The resolution of the plane image in the x direction is resolutionWidth, and the resolution in the y direction is resolutionHeight.
Planar image coordinates (x, y) = (u * resolutionWidth, v * resolutionHeight)
As described above, the screen coordinates of the display 508 in the area designated by the information terminal 7 can be converted into the coordinates of the plane image.

  It should be noted that when the plane image is displayed on the display 508, the reverse conversion may be performed.

<Registration of area>
Subsequently, an operation procedure of the image processing system 200 will be described with reference to FIGS.

  FIG. 9 is an example of a flowchart showing a procedure in which the information terminal 7 registers the area designated by the viewer in the image management device 5. The process of FIG. 9 starts when the viewer Y performs an operation to start registration of the area. Further, description will be made with reference to FIG. 10 as appropriate. FIG. 10 is an example of a diagram illustrating an area in which the lower right apex straddles the edge of the planar image.

  Image data is downloaded to the information terminal 7. Since the image data is associated with the terminal ID, the information terminal 7 holds the terminal ID. The viewer Y starts a drag operation with the pointing device. The reception unit 72 of the information terminal 7 receives the drag operation (S10). The reception unit 72 acquires, for example, a mouse on-mouse down event or a touch panel touch event. At this time, it is possible to acquire screen coordinates with the origin at the upper left corner of the display 508.

  Next, the area coordinate calculation unit 74 records the coordinates of the start point 651 in the storage unit 7000 or the like (S20).

  Next, the area coordinate calculation unit 74 determines whether the drag operation is continuing (S30). For example, when the reception unit 72 detects an on-mouse-up event, the determination in step S30 is No.

  While the determination in step S30 is Yes, the area coordinate calculation unit 74 records the current coordinates of the pointing device while updating the area rectangle as long as dragging continues (S40).

  If the determination in step S30 is No, the area coordinate calculation unit 74 determines that the current coordinates at the time when the drag operation is completed are the coordinates of the end point 652 (S50). As described with reference to FIG. 7, since the coordinates of the pointing device are always detected, the drag direction is determined, and the coordinates of the area (upper left vertex and lower right vertex) independent of the drag direction are obtained.

  Next, the area coordinate calculation unit 74 performs coordinate conversion between the coordinates of the upper left vertex and the coordinates of the lower right vertex. As described above, the screen coordinate system-> world coordinate system-> uv coordinate system-> plane image coordinate system is converted (S60).

  Since the coordinates finally obtained by the conversion process are the plane coordinates in which the spherical image is two-dimensionally expressed, the upper left vertex and the lower right vertex may straddle the end of the planar image. In this case, unless the width and height of the area are calculated in consideration of straddling the end portion, the width and height have negative values, and the area cannot be correctly specified.

  In FIG. 10, an upper left apex and a lower right apex that straddle the end portion in the planar image are shown. As shown in FIG. 10, there can be four patterns in the arrangement of the areas in the planar image. Pattern 1 is a case where the lower right apex does not straddle the end in the x direction and the end in the y direction. In the case of pattern 1, the area is not divided. Straddling the edge means that the lower right apex exceeds the resolution in the x direction or the resolution in the y direction of the plane image. Alternatively, it means that the coordinates of the lower right vertex are smaller than the coordinates of the upper left vertex.

Pattern 2 is a case where the lower right apex spans only the end in the y direction. In this case, the area is divided into upper and lower parts. Pattern 3 is a case where the lower right apex spans only the end in the x direction. In this case, the area is divided into left and right. Pattern 4 is a case where the lower right apex straddles the end in both the x and y directions. In this case, the area is divided into four areas, top, bottom, left and right. The area coordinate calculation unit 74 judges this pattern and appropriately calculates the width and height of the area. Areas 1 to 4 are adapted to patterns 1 to 4, respectively, and the coordinates of the upper left vertex and the lower right vertex of areas 1 to 4 are shown below.
Area 1: Upper left apex (ulx1, uly1), lower right apex (lrx1, lry1)
Area 2: Upper left vertex (ulx2, uly2), lower right vertex (lrx2, lry2)
Area 3: Upper left vertex (ulx3, uly3), lower right vertex (lrx3, lry3)
Area 4: Upper left vertex (ulx4, uly4), lower right vertex (lrx4, lry4)
As is clear from FIG. 10, the width and height of each area are calculated as follows.
Area 1: width = lrx1-ulx1
Height = lry1-ulx1
Area 2: Width = lrx2-ulx2
Height = resolution of height-uly2 + lry2
Area 3: width = width resolution-ulx3 + lrx3
Height = lry3-ulx3
Area 4: Width = Width resolution-ulx4 + lrx4
Height = resolution of height-uly4 + lry4
Returning to FIG. 9, description will be made.
Next, it is determined whether the x coordinate of the lower right vertex is larger than the x coordinate of the upper left vertex (S70). That is, it is determined whether the pattern is 3 or 4.

  If the determination in step S70 is Yes, since it is pattern 1 or 2, the area coordinate calculation unit 74 calculates the difference between the x coordinate of the lower right vertex and the x coordinate of the upper left vertex (S80). In other words, the difference between the lower right apex and the upper left apex is used as it is as the width of the area.

  If the determination in step S70 is No, the area coordinate calculation unit 74 calculates the width in consideration of the resolution of the width because it is pattern 3 or 4, (S90). Specifically, it is calculated by the above formula for calculating the widths of the areas 3 and 4.

  Next, it is determined whether the y coordinate of the lower right vertex is larger than the y coordinate of the upper left vertex (S100). That is, it is determined whether the pattern is 2 or 4.

  If the determination in step S100 is Yes, since it is pattern 1 or 3, the area coordinate calculation unit 74 calculates the difference between the y coordinate of the lower right vertex and the y coordinate of the upper left vertex (S110). That is, the difference between the lower right apex and the upper left apex is used as it is as the height of the area.

  If the determination in step S100 is No, since it is pattern 2 or 4, the area coordinate calculation unit 74 calculates the height in consideration of the resolution of the height (S120). Specifically, it is calculated by the above formula for calculating the height of the areas 2 and 4.

  In this way, it is determined whether the lower right vertex crosses the edge of the planar image, and when it crosses, the width and height are calculated in consideration of the resolution. Can be calculated correctly. Even if the viewer Y designates the area as the spherical image, the area can be correctly specified in the planar image.

  When the calculation of the width and the height is completed, the area coordinate calculation unit 74 receives an input of an arbitrary name of the area (S130). Details will be described with reference to FIG.

  The area coordinate calculation unit 74 transmits the terminal ID, the area start point, the width, the height, and the name to the image management device 5 (S140). The above is the registration of one area, and the viewer Y performs the same processing for each area.

<Area inspection>
Since the area is set in the plane image, the area can be inspected using the analysis information. A human detection result is used as an example of analysis information.

  FIG. 11 is an example of a flowchart showing a procedure for the area inspection unit 56 of the image management apparatus 5 to inspect an area. Note that description will be made with reference to FIG. 12 as appropriate. FIG. 12 is an example of a diagram illustrating an area and a person detection area.

  For example, if it is possible to detect the presence or absence of people in each area, it is possible to quantitatively check how many people were present in each area during a certain period. FIG. 11 shows an example of outputting the count result of the number of people in each area as a graph or a number as an example of the area inspection. The process of FIG. 11 is executed, for example, when the viewer Y performs an operation to start the inspection of the area by designating the terminal ID, the image ID and the like. Alternatively, the image pickup apparatus 1 automatically starts when the image pickup is finished and the analysis of the person detection is finished. Further, FIG. 11 is executed for each image data (image frame).

  First, the area inspection unit 56 acquires the resolution and analysis information of the plane image (S10). The resolution of the plane image is the same as the resolution of the plane image in FIG. 10 and is known in advance or stored in the storage unit 5000. The analysis information is stored in the analysis information management DB 5006.

  Next, the area inspection unit 56 acquires information specifying the area from the area information management DB 5005 (S20). Since the area is associated with the terminal ID, the terminal ID associated with the image data to be inspected for the area is identified, and the value of the registered area ID associated with this terminal ID (the area is identified Information) to be read in order.

  If the next area is not registered (No in S30), the process of FIG. 11 ends.

  When the paired area is registered (Yes in S30), the position of the area with respect to the planar image is determined and the area is divided (S40 to S80).

  The process of dividing the area will be described with reference to FIG. Areas 1 to 4 in FIG. 12 are the same as areas 1 to 4 in FIG. 10, respectively. An area that straddles the edge of the two-dimensional image, such as areas 2 to 4, cannot be determined in relation to the human detection area without specifying where the area after straddling the edge is located. Therefore, the area straddling the end of the plane image is divided at the end of the plane image as follows.

  Area 1: Area 1 is not divided because it does not straddle the edge of the planar image.

  Area 2: Area 2 is divided into area 2 and area 2 ′ because it straddles the end of the planar image in the y direction.

Area 2 top left corner (x2, y2), width = width2, height = width-direction resolution-y2
Area 2'top left corner (x2,0), width = width2, height = height2- (height resolution-y2)
Area 3: Area 3 is divided into area 3 and area 3 ′ because it straddles the end of the planar image in the x direction.

Area 3 Upper left corner (x3, y3), width = width resolution−x3, height = height
Area 3'top left corner (0, y3), width = width3− (width resolution−x3), height = height3
Area 4: Area 4 is divided into areas 4, 4 ′, 4 ″, 4 ′ ″ because it straddles the end portions of the planar image in the x direction and the y direction.

Area 4 Upper left corner (x4, y4), width = width resolution-x4, height = height resolution-y4
Area 4'top left corner (0,0), width = width4− (width resolution−x4), height = height4− (height resolution−y4)
Area 4 '' Upper left corner (x4,0), width = width resolution-x4, height = height4- (height resolution-y4)
Area 4 '''Upper left vertex (0, y4), width = width4− (width resolution−x4), height = height resolution−y4
Returning to FIG. 11, description will be made. The area inspection unit 56 calculates the coordinates (lrx, lry) of the lower right apex from the origin (x, y) and the width of the information specifying the area (S40). That is, the width is added to the x coordinate of the origin to calculate the x coordinate lrx of the coordinate of the lower right vertex.

  It is determined whether the x coordinate lrx of the lower right apex is larger than the resolution of the plane image in the x direction (S50). In the case of “resolution of the plane image in the x direction <x coordinate lrx”, the coordinates of the lower right apex straddle the end of the plane image in the x direction. That is, the state of area 3 in FIG. 12 is set.

  If the determination in step S50 is Yes, the area inspection unit 56 divides the area into two in the x direction and creates two areas, area 3 and area 3 '(S60). Areas 3 and 3 '(coordinates of upper left apex, width, height) have been described with reference to FIG.

  Next, the area inspection unit 56 determines whether the y coordinate lry of the lower right apex is larger than the resolution of the planar image in the y direction (S70). In the case of “resolution of the plane image in the y direction <y coordinate lry”, the coordinates of the lower right apex straddle the end of the plane image in the y direction. That is, the state of area 2 in FIG. 12 is set.

  If the determination in step S70 is Yes, the area inspection unit 56 divides the area into two in the y direction and creates two areas, area 2 and area 2 '(S80). Area 2 and area 2 ′ (coordinates of upper left apex, width, height) are shown in FIG.

  Further, the area 4 determined as Yes in both steps S50 and S70 is divided into four areas 4, 4 ', 4' ', and 4' ''. Areas 4, 4 ′, 4 ″, and 4 ′ ″ (coordinates of upper left apex, width, height) are shown in FIG.

  Next, a process of counting up the number of times a person is detected is performed for each of the divided areas. The area inspection unit 56 determines whether or not the image data has a human detection area (S90). If the determination in step S90 is No, the process returns to step S30.

  If the determination in step S90 is Yes, the area inspection unit 56 extracts one divided area (S100).

  Then, it is determined whether or not the center of the person detection area is included in the divided areas (S110).

  When the determination in step S110 is Yes, the area inspection unit 56 increases the number of times a person is detected in the area by one (S120). The number of times is counted for each area, and even if the area is divided, it is counted for each area before division. In this way, the information about the area can be created for each area using the analysis result. Since it is premised that the areas do not overlap each other, the process thereafter returns to step S90 and the next person detection area is processed. If the areas overlap each other, the process may return to step S100.

  If the determination in step S110 is no, the process returns to step S100 to take out the next divided area. Therefore, it is determined whether or not the area divided for one person detection area includes the center of the person detection area in a brute force manner.

  Thus, the number of people detected in each area was measured. If the same process is performed on the next image data (image frame), the information terminal 7 can display the number of times a person is detected and the frequency in a certain period for each area in a graph or the like.

  Further, the information terminal 7 can specify the inclusion area and the inclusion area based on the coordinates, and can calculate the ratio of the number of times of detection of the person in the inclusion area to the number of times of detection of the person in the inclusion area. If there are a plurality of included areas, it is possible to easily create a heat map or the like in the included areas.

  Note that the number of people in the area may be counted when a part of the person detection area overlaps with the area instead of comparing the centers of the person detection areas. In this case, the same person may be counted in a plurality of areas, but a person approaching the area can be counted. Further, the number of people in the area having the largest overlapping area between the human detection area and the area may be counted. In this case, it is possible to suppress counting of the same person in a plurality of areas and count the number of persons approaching the area. Further, the number of people in the area may be counted when the distance between the person detection area and the area is within a threshold value.

<Display area>
A process in which the information terminal 7 displays the area registered in the image management device 5 will be described. The information (value) for identifying the area registered in the image management device 5 is the coordinates in the plane image, and since the information terminal 7 displays the spherical image, the information for identifying the area is set in the world coordinate system. Needs to be converted to coordinates.

  FIG. 13 is an example of a flowchart showing a processing procedure for the information terminal 7 to display the area. The process of FIG. 13 starts when the viewer Y operates the information terminal 7 to request the image management device 5 to view image data. Since the image data is specified, the terminal ID is specified.

  First, the display control unit 73 of the information terminal 7 acquires the resolution of the plane image (S10). The resolution is known. When the person detection area is also displayed, the analysis information is acquired from the analysis information management DB 5006.

  The display control unit 73 acquires the information specifying the area from the area information management 5005 of the image management device 5 (S20). Since the area is associated with the terminal ID, the terminal ID associated with the image data to be displayed may be specified.

  Next, the display control unit 73 creates a rendering base having the same resolution as the plane image (S30). This is because the rendering base is pasted as a new layer on the spherical image. For example, the canvas tag which is an HTML tag is used.

  The display control unit 73 determines whether or not the next area is registered in the imaging device 1 (S40). When the next area is registered in the imaging device 1, the display control unit 73 performs area division processing (S50). This division processing has been described with reference to FIG.

  Then, all the divided areas are rendered based on the rendering base (S60, S70). Since the coordinates of the divided areas are known, it is sufficient to draw a rectangle on the rendering base. That is, a quadrangle specified by the coordinates of the divided areas is drawn. The area is processed to be displayed semi-transparently. Also, a different color (semi-transparent) is added to each area. The semi-transparent image allows the viewer Y to view the spherical image.

  When the processing is completed for all areas (No in S40), the display control unit 73 layers the rendering base (S70). Layering means generating the entire canvas tag as one image data. Places without areas are completely transparent.

  The display control unit 73 attaches the layered area to the spherical image (S80). That is, the process of attaching the layered planar image to the celestial sphere is performed as in the conventional case. As a result, the layer is attached to the spherical image.

  The viewer Y can confirm the area by superimposing it on the spherical image. Note that the image management device 5 may perform the processing of FIG. The viewer Y can operate whether or not to display the area while displaying the image data.

<Screen example>
14 to 18 show examples of screens from area registration to display.
FIG. 14A is an example of the area registration screen 601 displayed on the display 508 by the information terminal 7 when the area is registered.

  The viewer Y selects the menu for displaying the inside of the spherical image and registering the area. Then, as described above, the area is designated by dragging an arbitrary position with a pointing device such as a mouse. In FIG. 14A, the dragged area 602 is shown by a dotted line. The position of the area 602 is registered. As a process, the reception unit 72 of the information terminal 7 detects the coordinates of the start point and the end point of the drag, and the display control unit 73 displays a rectangle whose diagonal vertices are the coordinates.

  The viewer Y can give a name to the area as shown in FIG. In FIG. 14B, a name registration field 603 is displayed. The viewer Y inputs the name and presses the OK button 604. The reception unit 72 of the information terminal 7 receives this operation, and the transmission / reception unit 71 transmits the terminal ID, the information (value) for specifying the area, and the name to the image management apparatus 5. Although the name is arbitrary, for example, since the area in FIG. 14B is designated as a refrigerator, “refrigerator” or the like is input. Therefore, a name can be given to a specific product or the like to be displayed.

  FIG. 15A is an example of a diagram illustrating an example of registering an area with a plurality of coordinate points. The viewer Y repeats inputting the coordinate points 605 so as to surround the area to be designated (for example, clicking or touching with the pointing device 650). The reception unit 72 of the information terminal 7 receives the input of the coordinate point 605, and the display control unit 73 of the information terminal 7 clearly indicates the area 602 by surrounding the coordinate point 605 and the coordinate point 605 with a dotted line. Such area registration is effective when it is desired to register an area other than a rectangle.

  FIG. 15B is an example of a diagram illustrating an example of freehand area registration. In this case, the viewer Y drags the pointing device 650 along the outer edge of the area. The receiving unit 72 of the information terminal 7 receives the coordinates of the locus 606, and the display control unit 73 of the information terminal 7 clearly indicates the area 602 by displaying the locus 606 of the pointing device 650 with a line. You can specify an area of any shape in a short time.

  FIG. 16A is an example of a diagram for explaining the association of areas. The viewer Y specifies the area 602B after the area 602A. For example, assume that the area A is given the name "refrigerator" and the area B is given the name "before cash register". When the user Y wants to associate the area 602B with the area 602A, the viewer Y displays the menu for association by right-clicking in the area 602B while selecting the area 602A. A mark 660 indicating that selection is being performed is displayed in the area 602A. The reception unit 72 receives the selection operation of this menu, and the display control unit 73 causes the area association column 610 as shown in FIG. In the area association column 610, a message 611 "Please select an area to be associated" and "Before checkout" are displayed as the selected area 612.

  When the viewer Y selects the area 602A with the pointing device 650, the reception unit 72 detects the position of the pointing device 650 and determines that the area 602A has been selected. The reception unit 72 displays the name (refrigerator) of the area A registered in the area 602A in the associated area 613 of the area association field 610. In this way, the viewer Y can associate the areas with each other by a simple operation of selecting the areas. Further, since the already designated area is displayed, the relative position with respect to the area the viewer Y is about to register is easy to understand.

  FIG. 16B is an example of a diagram for explaining areas that are further registered in the area. In FIG. 16B, the area 602A is registered in the area 602C. The information terminal 7 can determine that the area 602A is included in the area 602C by comparing the coordinates of the area 602C with the coordinates of the area 602A. Alternatively, the viewer Y may explicitly specify. Therefore, the image management apparatus 5 can easily calculate the ratio of the number of people detected in the area 602A to the total number of people detected in the area 602C.

  FIG. 17 is a diagram showing an example of the area display screen 621. FIG. 17A is a diagram showing a display example of the registered area 602. Although two areas 602A and 602D are displayed in FIG. 17A, they are displayed in different colors. The screen data displayed on the display 508 by the information terminal 7 is created by the display controller 73 of the information terminal 7 as described above, but can also be created by the screen creator 54 of the image management apparatus 5. Although it is black and white in the figure, for example, the area 602A is displayed in red and the area 602D is displayed in blue. This makes it easier for the viewer Y to understand that the different area 602 is displayed. The displayed color is an example, and the viewer Y may be able to specify it.

  On the other hand, the display control unit 73 of the information terminal 7 displays the associated areas in the same color. In FIG. 17B, the associated “before cashier” area 602B and the associated “refrigerator” area 602A are displayed in the same color. The field of association is read out by referring to the display control unit 73 of the information terminal 7 and the area information management DB 5005, and a plurality of associated areas are set to the same color. Although it is black and white in the figure, it is displayed in red, for example. The colors displayed are examples. The viewer Y can easily determine the associated area by seeing the color of the area.

  Since the color of each area is semi-transparent, the viewer Y can easily determine which subject in the spherical image is designated for each area.

  FIG. 18A shows a display example of an area included in the other area. Since the areas 602A and 602C are displayed in different colors, the viewer Y can easily determine the included area. If the included area 602A is associated with the included area 602C, the color is the same, which makes it difficult to determine. The display control unit 73 of the information terminal 7 refers to the area information management DB 5005, reads the information (value) that specifies the area, and specifies the area in which one is completely included in the other. Then, a frame line of a color different from the area color is formed in the included area. As a result, when the information terminal 7 displays the omnidirectional image, the overlapping areas can be discriminated. Alternatively, one of them may be blinked, the brightness may be reduced, or the same color may be used for display.

  FIG. 18B is a diagram showing a display example of the human detection area. The display control unit 73 of the information terminal 7 previously forms a person detection area read from the analysis information management DB 5006 on a planar image. The forming method is the same as that of the area. In this way, the information terminal 7 can display the person detection area 630 together with the area 602 or separately from the area 602. The information terminal 7 can switch the display and non-display of the person detection area by the operation of the viewer Y. For example, in the example of FIG. 18B, since the center of the person detection area 630 overlaps with the area 602A, it is determined that a person is detected in the area 602A.

<Area inspection results>
FIG. 19 is an example of a diagram for explaining the inspection result of the area. In FIG. 19A, the horizontal axis is the date and the vertical axis is the number of detected persons. The number of people is counted by area. That is, the number of people detected per day is displayed for each area. The viewer Y can select the area to be displayed. Therefore, it is easy to determine the area with many people and the area with few people. Also, it is easy to understand what kind of change has occurred depending on the date.

  FIG. 19B shows a screen on which the total number of people in the omnidirectional image is displayed. The lower graph of FIG. 19B is the same as that of FIG. In the upper part of FIG. 19B, the bar count is displayed as the count number of persons detected in the entire spherical image for each date. Note that FIG. 19B shows the count numbers of the two imaging devices A and B. If the number of people detected in the whole celestial sphere image is large, the number of people detected in the area tends to be large. Therefore, the relationship between the whole and the part (area) can be easily grasped.

  The information terminal 7 can print the inspection result as shown in FIG. 19 or can send the inspection result to another device or terminal.

<Summary>
As described above, in the image processing system according to the present embodiment, even if the area 602 is set in the distorted image data like the spherical image 10, the area 602 is analyzed (image processing) such as human detection. Since it is converted into the coordinate system of the plane image 11, it is easy to apply the analysis result such as the person detection to the set area 602. Further, even if there is no boundary in the omnidirectional image but a boundary (edge) occurs in the planar image, the area is divided, so that the analysis result of person detection or the like can be correctly applied to each area.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments, but the present invention is not limited to these embodiments, and various modifications can be made without departing from the gist of the present invention. And substitutions can be added.

  For example, although a person is detected in the present embodiment, the information terminal 7 may determine the gender and display the number of times each of the male and female is detected. Alternatively, the number of times each operation is performed may be displayed by discriminating an operation such as reaching, sitting, or returning a product. Further, the presence / absence of products, the number of products, the volume, whether or not the arrangement of products is disturbed may be displayed.

  Further, in the above-described embodiment, the area is registered in the information terminal 7 and the image management device 5 creates information about the area. However, the information terminal 7 may create information about the area, or the image management device 5 creates the area. May be accepted. Further, an information processing device in which the information terminal 7 and the image management device 5 are integrated may execute the processing described in the present embodiment by itself. The information terminal 7 may have all or one or more of the functions of the image management apparatus 5, and the image management apparatus 5 may have all or one or more of the functions of the information terminal 7.

  In addition, in order to facilitate the understanding of the processes of the imaging device 1, the communication terminal 3, the image management device 5, and the information terminal 7, the configuration examples such as FIG. It is divided. However, the present invention is not limited by the division method or name of each processing unit. The processing of the image pickup apparatus 1, the communication terminal 3, the image management apparatus 5, and the information terminal 7 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing. Further, the image processing system 200 may include a plurality of image management devices 5.

  Further, the database of the storage unit 5000 of the image management apparatus 5 may be provided directly on the image management apparatus 5 or on the communication network 9 in which the image management apparatus 5 can read and write.

  The display 508 is an example of a display device, the reception unit 72 is an example of a reception unit, the area coordinate calculation unit 74 is an example of a coordinate conversion unit, and the area inspection unit 56 is an example of an analysis unit. The control unit 73 is an example of a display unit. The image management device 5 is an example of a first information processing device, and the information terminal 7 is an example of a second information processing device. The process performed by the image processing system 200 described in the present embodiment is an example of an image processing method.

1: Imaging device 3: Communication terminal 5: Image management device 7: Information terminal 31: Transmission / reception unit 51: Transmission / reception unit 52: Human detection unit 54: Screen creation unit 55: Area registration unit 56: Area inspection unit 200: Image processing system

JP 2009-048229 A

Claims (15)

An image processing system including one or more information processing devices,
Display means for displaying a wide-angle image having a wider angle of view than a predetermined value on a display device;
Receiving means for accepting designation of the area of the wide-angle image displayed on the display device,
Coordinate conversion means for converting the coordinates of the area in the display device into the coordinates of a plane image when the wide-angle image is analyzed;
Based on the result of comparing the coordinates of the analysis result of the plane image and the coordinates of the region converted by the coordinate conversion unit , an analysis unit that creates information about the region,
Image processing system having a.   The image processing system according to claim 1, wherein the coordinate conversion means converts the coordinates of the area on the display device into three-dimensional coordinates, and converts the three-dimensional coordinates into coordinates of the planar image. The wide-angle image is a spherical image in which 360 ° is captured,
When the area converted into the coordinates of the plane image exceeds the resolution of the plane image, the coordinate conversion unit calculates the width and height of the area in consideration of the resolution of the plane image,
The analysis means, when the lower right apex of the region in the coordinates of the plane image calculated using the width and height straddles the end of the plane image, divides the region at the end,
The image processing system according to claim 1, wherein the coordinates of the respective divided areas are compared with the coordinates of the analysis result.   The image processing system according to any one of claims 1 to 3, wherein information regarding the area created by the analysis unit is output for each area. The analysis result of the planar image has coordinates of a target area including the target when a predetermined target is detected in the planar image,
The image processing system according to claim 4, wherein when at least a part of the target area overlaps with the area, the analysis unit creates information about the area based on detection of the target in the area. The analyzing unit determines whether at least a part of the target region overlaps with the region for each wide-angle image,
When overlapping, counting the number of times the object is detected for each area,
The image processing system according to claim 5, wherein the display unit displays the number of times for each area.   The image processing system according to claim 6, wherein the display unit displays the number of times the target is detected in each of the wide-angle images as well as the number of times the target is detected in each of the regions. The reception unit is a rectangular range specified by coordinates of two points of a pointing device in the display device,
A range surrounded by a plurality of points of the pointing device in the display device, or
The image processing system according to claim 1, wherein the designation of the area is accepted by a range surrounded by a trajectory of a pointing device in the display device.   The image processing system according to claim 8, wherein the accepting unit accepts designation of the next area in a state where the area that has already been accepted is displayed. The image processing system according to any one of claims 1 to 9, wherein the display unit displays the area on the display device by making the area semi-transparent and superimposing the area on the wide-angle image. The receiving unit receives association of any two or more of the specified areas,
The image processing system according to claim 10, wherein the display unit displays the two associated areas in the same color on the display device. The accepting unit accepts designation of the region in which the entire region is included in the other region,
The analysis means calculates a ratio of the number of times the object is detected in the included area to the number of times the object is detected in the included area,
The image processing system according to claim 7, wherein the display unit displays the ratio. An image processing system including a first information processing device and a second information processing device,
The second information processing device,
Display means for displaying a wide-angle image having a wider angle of view than a predetermined value on a display device;
Receiving means for accepting designation of the area of the wide-angle image displayed on the display device,
Coordinate conversion means for converting the coordinates of the area in the display device into the coordinates of a plane image when the wide-angle image is analyzed,
The first information processing device is
An image processing system comprising: an analysis unit that creates information about the area based on a result of comparing the coordinates of the analysis result of the plane image with the coordinates of the area converted by the coordinate conversion unit . Information processing equipment,
Display means for displaying a wide-angle image having a wider angle of view than a predetermined value on a display device;
Receiving means for accepting designation of the area of the wide-angle image displayed on the display device,
Coordinate conversion means for converting the coordinates of the area in the display device into the coordinates of a plane image when the wide-angle image is analyzed;
A program for functioning as an analysis unit that creates information about the region based on the result of comparing the coordinates of the analysis result of the plane image with the coordinates of the region converted by the coordinate conversion unit . An image processing method performed by an image processing system including one or more information processing devices, comprising:
A step of displaying a wide-angle image having a wider angle of view than a predetermined value on the display device by the display means;
Receiving means receives the designation of the area of the wide-angle image displayed on the display device,
Coordinate transformation means transforming the coordinates of the region in the display device into the coordinates of a plane image when the wide-angle image is analyzed;
Analyzing means, based on the result of comparing the coordinates of the analysis result of the planar image and the coordinates of the area converted by the coordinate conversion means , creating information about the area,
An image processing method including.

Images