JP2021125145A - Information processing apparatus, image processing system, and program - Google Patents

Abstract

To provide an information processing apparatus that can perform distribution expression according to the size of a person.SOLUTION: There is provided an information processing apparatus 5 having a person detection unit 52 that detects a person from image data picked up by an imaging apparatus 1, and a distribution expression creation unit 55 that creates the distribution expression of the person by using the size of the person detected by the person detection unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an image processing system, and a program.

Traditionally, retail store managers analyze the behavior of visitors and staff and reflect it in store operations and product planning. For example, the image of a camera installed in the store is used for the analysis, and the store manager or the like sees the image in the store to understand how the visitor moves and where he / she stays. As a result, it is possible to understand the flow lines of visitors, products that are easy to attract interest, and exhibition methods, which can be reflected in store management and product planning.

A monitoring system has been devised that not only provides images of each location in the store, but also processes the images by combining a camera and image processing technology to output information that is easy for the store manager to analyze (for example, patent documents). See 1.). Patent Document 1 discloses a system that identifies the same person by matching in the face area, recognizes the entire human body in a rectangular area, divides the whole human body into parts of the face and upper body, and tracks the behavior of the specified person. ..

However, in the conventional technique, there is a problem that the size of the distribution expression of people cannot be adjusted. One of the display methods of the person detection result is a distribution expression such as a heat map (a display method in which the target is divided into squares and the display color of the square is changed according to the detection result of each square). There is a way to express it. Conventionally, the size of the heat map has been fixed.

In view of the above problems, an object of the present invention is to provide an information processing device or the like capable of expressing a distribution based on the size of a person.

In view of the above problems, the present invention creates a distribution representation of the person using a person detection unit that detects a person from image data captured by the image pickup apparatus and the size of the person detected by the person detection unit. Provided is an information processing apparatus characterized by having an expression creation unit.

It is possible to provide an information processing device or the like that can express a distribution based on the size of the distribution expression of a person.

It is a figure explaining the heat map generated by an image processing system. This is an example of a schematic configuration diagram of an image processing system. This is an example of the hardware configuration diagram of the image pickup device. This is an example of a hardware configuration diagram of a communication terminal in the case of a cradle having a wireless communication function. This is an example of a hardware configuration diagram of an image management device and a terminal device. This is an example of a functional block diagram of an image pickup device, a communication terminal, an image management device, and a terminal device that form a part of an image processing system. It is a figure explaining the image in which a person is detected. It is a figure which shows typically the information obtained by the detection result of a person. It is a figure explaining the method of adjusting the size of a heat map when a person is detected by a rectangle. It is a figure explaining the method of adjusting the size of a heat map when a person is detected by a polygon. It is a figure explaining the method of adjusting the size of the heat map when a person is detected at the apex of the body. It is a figure which shows an example of the heat map size selection screen. It is a figure which shows an example of the heat map creation screen displayed by a terminal device. It is a figure which shows typically the method of making a heat map. This is an example of a flowchart for explaining a method of creating a heat map. It is an example of the figure explaining the method of determining the density of a heat map. This is an example of a flowchart showing a method of determining the density of the heat map. It is a figure which shows some variation of the shape of a heat map. It is a figure which shows an example of the heat map displayed in a different color for each person. It is a figure which shows the display example of the heat map at a certain moment. It is a figure which shows typically superposition of the original image data and a heat map image. This is an example of a diagram for explaining the correspondence between the coordinates of the equirectangular image and the three-dimensional coordinates. It is an example of a figure showing an actual original image data and a heat map image. This is an example of a sequence diagram for explaining the processing performed by the image processing system.

Hereinafter, as an example of the embodiment for carrying out the present invention, the image processing system and the image processing method performed by the image processing system will be described with reference to the drawings.

<Outline of processing>
First, the outline of the processing performed by the image processing system will be described with reference to FIG. FIG. 1 is a diagram illustrating a heat map generated by an image processing system.

(1) The image processing system performs image processing to detect a person in the image data and detects the person. For example, it is possible to generate an circumscribing rectangle that is the maximum range of a person. FIG. 1A shows a rectangle 201 of a person with an open hand detected by image processing.

In the present embodiment, the heat map can be generated as it is from the size of the rectangle 201 which is the detection result of the person. Further, depending on the user who uses the image processing system, the size of only the body of the person may be reflected in the heat map, or the maximum width of the person may be reflected in the heat map. For example, a user who wants to know the stay of a visitor often wants to know where the visitor is with higher accuracy (narrow range), but in a facility such as a gym where a person moves a lot, the visitor moves a lot. In some cases, the user wants to know if he / she will collide with another person (whether there is enough space to exercise).

Therefore, in the present embodiment, the size of the heat map can be adjusted according to the user who uses the image processing system and the purpose of the person detection result.

(2) For a user or an application in which the maximum width length is used as a heat map, as shown in FIG. 1 (b), the image processing system sets the maximum width length (maximum width) as the size of the heat map.

(3) As shown in FIG. 1 (c), the image processing system obtains the size of the body 202 and sets this width as the size of the heat map for the user or the application in which the body is used as the heat map.

(4) (5) Since the image processing system creates a heat map of the size adjusted in this way, the size of the heat map can be flexibly changed. In FIG. 1D, the heat map 203 having the maximum width of the person is displayed, and in FIG. 1E, the heat map 204 having the same size as the width of the body of the person is displayed.

As described above, the image processing system of the present embodiment can link the heat map, which has been fixed regardless of the size of the person, to the size of the person, and can further adjust the size of the person. For example, since the size of the heat map is adjusted according to the user or the application, the size of the heat map can be changed.

<Terminology>
A person is a human being. This embodiment can be applied not only to humans but also to animals. Further, it may be applied to a moving body such as a vehicle.

Adjusting means making it moderate. For example, it means that the size of the heat map is set to an appropriate size according to the user and the application.

The distribution expression is, for example, an expression method for expressing how people are distributed by changing the color and density. Sometimes called a heat map.

The information regarding the size of the distribution expression is information indicating how much the size of the distribution expression is adjusted. In this embodiment, it is described by the term of heat map size information.

<System configuration of image processing system>
FIG. 2 is an example of a schematic configuration diagram of the image processing system 200. The image processing system 200 includes an image management device 5, an image pickup device 1, a communication terminal 3, and a terminal device 7 connected via a communication network 9. The image pickup device 1 is installed in the store by the installer X. The terminal device 7 is operated by the viewer Y.

The communication network 9 is assumed to be a general network such as the Internet. It may include a LAN in a company or a home, or may include a wide area Ethernet (registered trademark) or a WAN composed of a plurality of LANs. The communication network may be partially or wholly wireless. The wireless network may include a mobile phone network such as 3G, 4G, and 5G in addition to wireless LAN and WiMAX.

The image pickup device 1 is a camera that captures a 360-degree image of the surroundings with one image capture and creates a spherical image. Sometimes referred to as a digital still camera or digital video camera. Further, when the communication terminal 3 is equipped with a camera, the communication terminal 3 can be a digital camera. In the present embodiment, in order to make the explanation easy to understand, the image pickup apparatus 1 will be described as a digital camera for obtaining a spherical image. The image pickup apparatus 1 periodically images the surrounding 360. It does not necessarily have to be periodic, and may be imaged irregularly, may be imaged by the operation of the installer X, or may be imaged from the image management device 5 by the viewer Y requesting the image management device 5. You may take an image by the command of.

The image pickup device 1 may automatically capture several landscapes having different line-of-sights and combine a plurality of image data to create a spherical image.

The communication terminal 3 has a communication function of connecting to the communication network 9 instead of the image pickup device 1. The communication terminal 3 is a cradle for supplying electric power to the image pickup device 1 and fixing it to a store. The cradle is an expansion device that expands the functions of the image pickup apparatus 1. The communication terminal 3 has an interface for connecting to the image pickup device 1, whereby the image pickup device 1 can use the functions of the communication terminal 3. The communication terminal 3 performs data communication with the image pickup apparatus 1 via this interface. Then, data communication is performed with the image management device 5 via the wireless router 9a and the communication network 9.

If the image pickup device 1 has a function of directly communicating data with the wireless router 9a or the communication network 9, the communication terminal 3 may not be provided. Alternatively, the image pickup device 1 and the communication terminal 3 may be integrally configured.

The image management device 5 is, for example, an information processing device that functions as a server, and can perform data communication with the communication terminal 3 and the terminal device 7 via the communication network 9. OpenGL ES (API: Application Interface for 3D graphics) is installed in the image management device 5. By calling OpenGL ES, it is possible to create a spherical image from a regular-distance cylindrical image, or to create a thumbnail image of a part of the spherical image (predetermined area image).

The image management device 5 may be compatible with cloud computing. There is no strict definition for the physical configuration of cloud computing, but the configuration of the information processing device is achieved by dynamically connecting and disconnecting resources such as the CPU, RAM, and storage that make up the information processing device according to the load. It is known that the installation location can be changed flexibly. The image management device 5 may be on-premises (inside the firewall) or may exist on the Internet. Further, the image management device 5 may be provided by a plurality of information processing devices, or may be realized by one information processing device.

The terminal device 7 is, for example, a PC (Personal Computer), and performs data communication with the image management device 5 via the communication network 9. The terminal device 7 may be a tablet terminal, a PDA (Personal Digital Assistant), an electronic blackboard, a video conference terminal, a wearable PC, a game machine, a mobile phone, a car navigation system, a smartphone, or the like, in addition to a PC. In this embodiment, since the terminal device 7 can communicate with the image management device 5 via the Web browser, the terminal device 7 may be a device on which the Web browser operates. However, the terminal device 7 may execute a native application dedicated to the image processing system 200.

The image pickup device 1, the communication terminal 3, and the wireless router 9a are installed at predetermined positions by the installer X at each sales base such as a store. The terminal device 7 is installed at the head office or the like that controls each sales base, and by displaying an image showing the status of each base sent via the image management device 5, the viewer Y can display the status of each base. You can browse the image showing. However, the terminal device 7 can communicate with the image management device 5 from a place other than the head office.

<Hardware configuration of the embodiment>
Next, the hardware configurations of the image pickup device 1, the communication terminal 3, the terminal device 7, and the image management device 5 of the present embodiment will be described with reference to FIGS. 3 to 5.

<< Imaging device >>
FIG. 3 is an example of a hardware configuration diagram of the image pickup apparatus 1. In the following, the image pickup device 1 is an omnidirectional image pickup device using two image pickup elements, but the number of image pickup devices may be three or more. Further, the device does not necessarily have to be a device dedicated to omnidirectional imaging, and by attaching a retrofitted omnidirectional imaging unit to a normal digital camera, smartphone, or the like, it may have substantially the same function as the imaging device 1. ..

As shown in FIG. 3, the imaging device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only Memory). ) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, and antenna 117a.

Of these, the image pickup unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two image pickups provided corresponding to each wide-angle lens. It includes elements 103a and 103b. The image sensors 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images from fisheye lenses into image data of electrical signals and output them. It has a timing generation circuit that generates a vertical sync signal, a pixel clock, and the like, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The image sensors 103a and 103b of the image pickup unit 101 are each connected to the image processing unit 104 by a parallel I / F bus. On the other hand, the image pickup elements 103a and 103b of the image pickup unit 101 are connected by a serial I / F bus (I2C bus or the like) separately from the image pickup control unit 105. The image processing unit 104 and the image pickup control unit 105 are connected to the CPU 111 via the bus 110. Further, a ROM 112, a SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

The image processing unit 104 takes in the image data output from the image pickup elements 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data. , So-called equirectangular projection format image data.

The image pickup control unit 105 generally uses the image pickup control unit 105 as a master device and the image pickup elements 103a and 103b as slave devices, and sets commands and the like in the register group of the image pickup elements 103a and 103b by using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the image pickup control unit 105 also uses the I2C bus to take in the status data and the like of the register group of the image pickup elements 103a and 103b and send them to the CPU 111.

Further, the image pickup control unit 105 instructs the image pickup devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some image pickup devices 1 may have a preview display function on a display and a function corresponding to a moving image display. In this case, the image data is continuously output from the image sensors 103a and 103b at a predetermined frame rate (frames / minute).

Further, as will be described later, the image pickup control unit 105 also functions as a synchronization control means for synchronizing the output timings of the image data of the image pickup elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the image pickup apparatus 1 is not provided with a display unit, but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the image pickup apparatus 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data in the process of being processed by the image processing unit 104 and image data in the equirectangular projection format that has been processed.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. By operating the operation buttons, the user inputs various imaging modes and imaging conditions.

Network I / F116 is a general term for interface circuits (USB I / F, etc.) with external media such as SD cards and personal computers. Further, the network I / F116 may be a network interface regardless of whether it is wireless or wired. The equirectangular projection format image data stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or is recorded on an external medium via the network I / F 116, or if necessary, via the network I / F 116 which becomes a network I / F. It is transmitted to an external device such as a communication terminal 3.

The communication unit 117 is connected to an external device such as a communication terminal 3 via a remote radio technology such as Wi-Fi (wireless fidelity), NFC, or LTE (Long Term Evolution) via an antenna 117a provided in the image pickup device 1. Communicate. The communication unit 117 can also transmit image data in equirectangular projection format to an external device of the communication terminal 3.

The electronic compass 118 calculates the orientation and tilt (Roll rotation angle) of the image pickup device 1 from the magnetism of the earth, and outputs the orientation / tilt information. This orientation / tilt information is an example of related information (metadata) along with Exif, and is used for image processing such as image correction of captured images. The related information also includes each data of the image capturing date and time and the data capacity of the image data.

<< Communication terminal >>
Next, the hardware configuration of the communication terminal 3 will be described with reference to FIG. Note that FIG. 4 is a hardware configuration diagram of the communication terminal 3 in the case of a cradle having a wireless communication function.

As shown in FIG. 4, the communication terminal 3 includes a CPU 301 that controls the operation of the entire communication terminal 3, a ROM 302 that stores basic input / output programs, and a RAM (Random Access Memory) 304 that is used as a work area of the CPU 301. It has a communication unit 305 for data communication via Wi-fi, NFC, LTE, etc., a USB I / F 303 for wired communication with the image pickup device 1, and an RTC (Real Time Clock) 306 for holding a calendar and time information. There is.

Further, a bus line 310 such as an address bus or a data bus for electrically connecting each of the above parts is provided.

The ROM 302 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

The communication unit 305 communicates with the wireless router 9a and the like by using the antenna 305a and using the wireless communication signal.

In addition to the illustration, it is equipped with a GPS receiver that receives GPS signals including the position information (latitude, longitude, and altitude) of the communication terminal 3 by GPS (Global Positioning Systems) satellite or IMES (Indoor MEssaging System) as indoor GPS. You may be.

<< Image management device, terminal device >>
The hardware configuration of the image management device 5 and the terminal device 7 will be described with reference to FIG. Note that FIG. 5 is a hardware configuration diagram of the image management device 5 and the terminal device 7. Since both the image management device 5 and the terminal device 7 are computers, the configuration of the image management device 5 will be described below. It is assumed that the configuration of the terminal device 7 is the same as that of the image management device 5, and even if there is a difference, there is no problem in the description of the present embodiment.

The image management device 5 has a CPU 501 that controls the operation of the entire image management device 5, a ROM 502 that stores a program used for driving the CPU 501 such as an IPL, and a RAM 503 that is used as a work area of the CPU 501. Further, it has an HDD (Hard Disk Drive) 505 that controls reading or writing of various data to the HD 504 according to the control of the HD 504 and the CPU 501 that store various data such as a program for the image management device 5. It also has a media drive 507 that controls reading or writing (storage) of data to a recording medium 506 such as a flash memory, and a display 508 that displays various information such as cursors, menus, windows, characters, or images. It is preferable that the display 508 is equipped with a touch panel. In addition, a network I / F 509 for data communication using the communication network 9, a keyboard 511 equipped with a plurality of keys for inputting characters, numerical values, various instructions, etc., and a target for selecting, executing, and processing various instructions. It has a mouse 512 for selecting, moving a cursor, and the like. Further, it has a CD-ROM drive 514 that controls reading or writing of various data to a CD-ROM (Compact Disc Read Only Memory) 513 as an example of a detachable recording medium. Further, as shown in FIG. 5, each of the above components is provided with a bus line 510 such as an address bus or a data bus for electrically connecting the components.

<About the function of the image processing system>
FIG. 6 is a functional block diagram of the image pickup device 1, the communication terminal 3, the image management device 5, and the terminal device 7, which form a part of the image processing system 200.

<< Functional configuration of imaging device >>
The image pickup device 1 includes a reception unit 12, an image pickup unit 13, a sound collection unit 14, a connection unit 15, and a storage / reading unit 19. Each of these parts has a function realized by operating any of the components shown in FIG. 3 by a command from the CPU 111 according to a program for the image pickup apparatus 1 developed on the DRAM 113 from the SRAM 113. It is a means.

Further, the image pickup apparatus 1 has a storage unit 1000 constructed by one or more of the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. The storage unit 1000 stores the program for the image pickup apparatus 1 and the terminal ID.

The reception unit 12 of the image pickup apparatus 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 shown in FIG. 3, and receives an operation input from the user (installer X in FIG. 2). The imaging device 1 automatically and periodically images the surroundings even if there is no operation for imaging by the installer X. The periodic interval may be set by the installer X in the image pickup device 1 or by the viewer Y via the image management device 5.

The image pickup unit 13 is realized mainly by the processing of the image pickup unit 101, the image processing unit 104, the image pickup control unit 105, and the CPU 111 shown in FIG. 3, and images a landscape or the like to create image data.

The sound collecting unit 14 is mainly realized by the processing of the microphone 108, the sound processing unit 109, and the CPU 111 shown in FIG. 3, and collects the sound around the image pickup device 1.

The connection unit 15 is mainly realized by processing the network I / F 116 and the CPU 111, receives power supply from the communication terminal 3, and performs data communication with the communication terminal 3.

The storage / reading unit 19 is mainly realized by the processing of the CPU 111 shown in FIG. 3, and stores various data in the storage unit 1000 and reads various data from the storage unit 1000. In the following, even when the image pickup apparatus 1 reads and writes from the storage unit 1000, the description "via the storage / reading unit 19" may be omitted.

<< Functional configuration of communication terminal >>
The communication terminal 3 has a communication unit 31, a reception unit 32, a connection unit 33, and a storage / reading unit 39. Each of these parts has a function realized by operating any of the components shown in FIG. 4 by an instruction from the CPU 301 according to a program for the communication terminal 3 developed on the RAM 304 from the ROM 302. It is a means.

Further, the communication terminal 3 has a storage unit 3000 constructed by the ROM 302 and the RAM 304 shown in FIG. The program for the communication terminal 3 is stored in the storage unit 3000.

(Each function configuration of communication terminal)
The communication unit 31 of the communication terminal 3 is mainly realized by the processing of the communication unit 305 and the CPU 301 shown in FIG. 4, and transmits / receives various data to / from the image management device 5 via the wireless router 9a and the communication network 9. I do. In the following, even when the communication terminal 3 communicates with the image management device 5, the description "via the communication unit 31" may be omitted.

The reception unit 32 is mainly realized by the processing of the CPU 301 shown in FIG. 4, and receives an operation input from the user (installer X in FIG. 2).

The connection unit 33 is mainly realized by the processing of the USB I / F 303 and the CPU 301 shown in FIG. 4, and supplies power to the image pickup apparatus 1 and performs data communication.

The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 4, and stores various data in the storage unit 3000 and reads various data from the storage unit 3000. In the following, even when the communication terminal 3 reads / writes from the storage unit 3000, the description "via the storage / reading unit 39" may be omitted.

<< Functional configuration of image management device >>
The image management device 5 includes a communication unit 51, a person detection unit 52, an adjustment unit 53, a screen creation unit 54, a distribution expression creation unit 55, and a storage / reading unit 59. Each of these parts is realized by operating any of the components shown in FIG. 5 by an instruction from the CPU 501 according to the device program 5010 for the image management device 5 developed on the RAM 503 from the HD 504. Function or means.

Further, the image management device 5 has a RAM 503 shown in FIG. 5 and a storage unit 5000 constructed by the HD 504. The base information storage unit 5001, the imaging information storage unit 5002, the image data storage unit 5003, the analysis information storage unit 5004, the distribution expression size information storage unit 5005, and the user information storage unit 5006 are constructed in the storage unit 5000. Has been done. Each database will be described below.

表１は、拠点情報記憶部５００１に記憶される各情報をテーブル状に示す拠点管理テーブルである。拠点管理テーブルでは、地域ＩＤ、地域名、拠点ＩＤ、拠点名、拠点レイアウトマップ、及び、装置ＩＤの各フィールドが関連付けて記憶されている。また、拠点管理テーブルの１つの行をレコードという場合がある。以下の各テーブルでも同様である。このうち、地域ＩＤは、地域を識別するための識別情報である。地域ＩＤの一例としては重複しない番号とアルファベットの組み合わせが挙げられる。

Table 1 is a base management table showing each information stored in the base information storage unit 5001 in a table shape. In the base management table, each field of the area ID, the area name, the base ID, the base name, the base layout map, and the device ID is stored in association with each other. In addition, one row of the base management table may be called a record. The same applies to each of the following tables. Of these, the area ID is identification information for identifying the area. An example of a region ID is a unique combination of numbers and alphabets.

The area name indicates an area or range of land such as Kanto, Tokyo, Shibuya Ward, New York State, New York City, etc. It may be called a region name. The identification information refers to a name, a code, a character string, a numerical value, or a combination of two or more of these, which is used to uniquely distinguish a specific object from a plurality of objects. The same applies to the following ID or identification information.

The base ID is an example of identification information for identifying the base. The base ID is assigned to the base name so as not to be duplicated. It may be referred to as site-specific information. An example of a base ID is a combination of a unique number and an alphabet. The base is a place where the image pickup apparatus 1 is installed and serves as a base for photographing the surroundings. An example of a base is a store.

The base name is a store name such as the Shibuya store, a venue name such as the Shibuya venue, etc., and is the name of the base. File names such as the layout of each base and image data showing the map are registered in the base layout map. The location of the image pickup device 1 and the products handled at the base is specified by the base layout map in two-dimensional coordinates.

The terminal ID is identification information for identifying the image pickup apparatus 1. It may be referred to as terminal-specific information. The terminal ID is, for example, a serial number, a serial number, a numerical value that does not overlap with the model number, an IP address, a MAC address, or the like of the image pickup apparatus 1, but is not limited thereto. As shown in Table 1, one or more image pickup devices 1 (terminal IDs) are installed at one base, and their positions are managed by a layout map or the like. The base management table may be registered by the installer X or the viewer Y, or may be registered by the supplier of the image processing system 200.

表２は、撮像情報記憶部５００２に記憶される各情報をテーブル状に示す撮像管理テーブルである。撮像管理テーブルでは、拠点ＩＤごとに、撮像タイトル、撮像開始日時、及び撮像終了日時の各フィールドが関連付けて記憶されている。撮像タイトルは、閲覧者Ｙが入力したイベントのタイトルである。つまり、閲覧者Ｙが消費者の行動を監視したい何らかのイベントが店舗で催される場合に、このイベントの名称が撮像タイトルとなる。当然ながら、イベントの名称は閲覧者Ｙが任意に付与できるためイベントの名称でなくてもよい。例えば、単に撮像年月日とすることもできる。閲覧者Ｙは、画像データの複数のファイルから所望の画像データを抽出する際に、撮像タイトルを参照することができる。なお、１回の撮像イベントで複数の画像データが時系列に（定期的に）撮像される。撮像開始日時は、閲覧者Ｙによって入力された日時であり、撮像装置１が撮像を開始する（又は開始した）日時を示す。撮像終了日時は、閲覧者Ｙによって入力された日時であり、撮像装置１が撮像を終了する（又は終了した）日時を示す。閲覧者Ｙは撮像開始日時と撮像終了日時を事前に登録しておくこともできる（予約撮像）。撮像管理テーブルは、主に画像管理装置５が登録する。

Table 2 is an imaging management table showing each information stored in the imaging information storage unit 5002 in a table shape. In the image pickup management table, each field of the image pickup title, the image pickup start date and time, and the image pickup end date and time is stored in association with each base ID. The captured title is the title of the event input by the viewer Y. That is, when some event that the viewer Y wants to monitor the consumer's behavior is held in the store, the name of this event becomes the imaging title. As a matter of course, the name of the event does not have to be the name of the event because the viewer Y can arbitrarily assign the name of the event. For example, it may be simply the imaging date. Viewer Y can refer to the captured title when extracting desired image data from a plurality of files of image data. It should be noted that a plurality of image data are imaged in time series (regularly) in one imaging event. The imaging start date and time is the date and time input by the viewer Y, and indicates the date and time when the imaging device 1 starts (or starts) imaging. The imaging end date and time is a date and time input by the viewer Y, and indicates a date and time when the imaging device 1 ends (or ends) imaging. Viewer Y can also register the imaging start date and time and the imaging end date and time in advance (reserved imaging). The image management device 5 mainly registers the image capture management table.

表３は、画像データ記憶部５００３に記憶される各情報をテーブル状に示す画像管理テーブルである。画像管理テーブルでは、端末ＩＤごとに、画像ＩＤ、画像データのファイル名、及び撮像日時が関連付けて記憶されている。画像ＩＤは、画像データを一意に識別するための識別情報の一例である。画像固有情報と称してもよい。画像データのファイル名は、画像ＩＤで特定される画像データのファイル名である。撮像日時は画像データが端末ＩＤで示される撮像装置１で撮像された日時である。画像データも、記憶部５０００に記憶されている。

Table 3 is an image management table showing each information stored in the image data storage unit 5003 in a table shape. In the image management table, the image ID, the file name of the image data, and the imaging date and time are stored in association with each terminal ID. The image ID is an example of identification information for uniquely identifying the image data. It may be referred to as image-specific information. The file name of the image data is the file name of the image data specified by the image ID. The imaging date and time is the date and time when the image data was captured by the imaging device 1 indicated by the terminal ID. Image data is also stored in the storage unit 5000.

For example, when viewing an image file, the viewer Y accesses the image management device 5 with the terminal device 7 and selects an imaging title from the imaging management table in Table 2. The image management device 5 can read the terminal ID associated with the base ID of the imaging title from the base management table in Table 1. Since the terminal ID is clarified, the image management device 5 can specify the image data in which the imaging date and time is included in the imaging start date and time to the imaging end date and time among the image data associated with the terminal ID in the image management table.

Of course, the viewer Y can also directly specify the terminal ID and the base ID. In the present embodiment, for the sake of simplicity, a mode in which the viewer Y specifies a terminal ID for browsing will be mainly described. The image management table is mainly registered by the image management device 5.

表４は、解析情報記憶部５００４に記憶される各情報をテーブル状に示す。表４（ａ）は人物検知結果テーブルの一例を示す。重複しないＩＤ（識別情報）に対応付けて、画像ＩＤ、人物ＩＤ、人物Ｘ座標、人物Ｙ座標が登録されている。
画像ＩＤ：画像データ記憶部の画像の画像ＩＤと同じ者である。
人物ＩＤ：検知された人物の識別情報である。同じ画像ＩＤに同じ人物ＩＤがあるのは、検知領域の大きさを変えて人物検知部５２が人物を検知する場合があるからである。
人物Ｘ座標：検知領域のＸ座標である。
人物Ｙ座標：検知領域のＹ座標である。

Table 4 shows each information stored in the analysis information storage unit 5004 in a table form. Table 4 (a) shows an example of the person detection result table. An image ID, a person ID, a person X coordinate, and a person Y coordinate are registered in association with a unique ID (identification information).
Image ID: The same person as the image ID of the image in the image data storage unit.
Person ID: Identification information of the detected person. The same person ID exists in the same image ID because the person detection unit 52 may detect a person by changing the size of the detection area.
Person X coordinate: The X coordinate of the detection area.
Person Y coordinate: The Y coordinate of the detection area.

When the detection area of a person is a rectangle, the coordinates of the diagonal vertices of the rectangle are registered. In addition, the items (mainly the number of person X coordinates and person Y coordinates) of the person detection result table change depending on the person detection method such as a polygon described later.

Table 4 (b) is an example of a person range storage table. An image ID, a person ID, coordinates 1, and coordinates 2 are registered in association with a unique ID (identification information). The image ID and person ID may be the same as in Table 4 (a). Coordinates 1 and 2 represent the width of the heat map. That is, the size of the heat map adjusted based on the size information of the heat map described later is shown.

表５（ａ）は、分布表現大きさ情報記憶部５００５に記憶される各情報をテーブル状に示す分布表現大きさ情報テーブルである。分布表現大きさ情報テーブルには、ヒートマップの大きさ情報が登録されている。大きさ情報は、例えば胴体、最大幅長、頭部などである。閲覧者Ｙは端末装置７に表示された所定の画面から任意のヒートマップの大きさ情報を選択できる。ヒートマップの大きさ情報が選択されると、調整部５３は選択された大きさ情報でヒートマップの大きさを調整する。なお、ヒートマップの大きさ情報は幅方向の大きさを意味しており、高さ方向は登録されていない。これは、ヒートマップの大きさに人物の高さが反映されないためである。しかし、ヒートマップの大きさに人物の高さが反映される場合は、分布表現大きさ情報テーブルにヒートマップの高さ方向の大きさ情報が登録されることが好ましい。

Table 5 (a) is a distribution expression size information table showing each information stored in the distribution expression size information storage unit 5005 in a table shape. The size information of the heat map is registered in the distribution expression size information table. The size information is, for example, the torso, the maximum width, the head, and the like. Viewer Y can select arbitrary heat map size information from a predetermined screen displayed on the terminal device 7. When the heat map size information is selected, the adjusting unit 53 adjusts the heat map size with the selected size information. The size information of the heat map means the size in the width direction, and the height direction is not registered. This is because the height of the person is not reflected in the size of the heat map. However, when the height of the person is reflected in the size of the heat map, it is preferable that the size information in the height direction of the heat map is registered in the distribution expression size information table.

Further, although it has been explained in Table 5 (a) that the viewer Y can select from a plurality of size information, one size information may be stored in advance. For example, when the analysis performed by the viewer Y is fixed, the image management device 5 may create a heat map with one size information prepared by the service provider.

Table 5 (b) also shows the distribution expression size information table, but in Table 5 (b), the size information of the heat map is associated with the application. For example, the fuselage is associated with the application of retention analysis, and the maximum width is associated with the application of space analysis. The retention analysis is selected when the viewer Y analyzes how the visitor moves and where the visitor stays. Space analysis is selected when viewer Y analyzes whether there is sufficient space in the facility where the visitor exercises. Viewer Y can select any purpose from a predetermined screen displayed on the terminal device 7. When the application is selected, the adjusting unit 53 acquires the size information of the heat map associated with the application, and adjusts the size of the heat map with this size information.

表６は、ユーザ情報記憶部５００６に記憶される各情報をテーブル状に示す。表６（ａ）は用途別設定項目テーブルの一例を示す。用途別設定項目テーブルには、重複しないＩＤ（識別情報）に対応付けて社名と選択可能領域が登録されている。用途別設定項目テーブルは各会社が選択できるヒートマップの大きさのリストである。例えば、
・一般的な店舗のＡ社は最大幅長、肩幅及び頭幅からヒートマップの大きさを選択できる。
・レストランであるＢ社は肩幅と頭幅からヒートマップの大きさを選択できる。
・必ずしも決まった幅としたくないＣ社は、自由に選択する（自由選択長）。
・ジムを経営するＤ社は最大幅長をヒートマップの大きさとする。

Table 6 shows each information stored in the user information storage unit 5006 in a table shape. Table 6 (a) shows an example of the setting item table for each application. In the setting item table for each application, the company name and selectable area are registered in association with a unique ID (identification information). The setting item table for each application is a list of heat map sizes that can be selected by each company. for example,
・ Company A, a general store, can select the size of the heat map from the maximum width, shoulder width, and head width.
・ Company B, a restaurant, can select the size of the heat map from the shoulder width and head width.
・ Company C, which does not necessarily have a fixed width, can freely select (free selection length).
・ Company D, which runs the gym, sets the maximum width as the size of the heat map.

Note that the viewer Y may register one size instead of registering the list of heat map sizes that can be selected in this way.

Table 6 (b) is an example of a setting method table for each area. In the area-specific setting method table, the recognition area and the selection method are registered in association with the unique ID (identification information). The area-specific setting method table is a table in which when the viewer Y selects the selectable area in Table 6 (a) from a predetermined screen, how the selected items are displayed on the screen is registered. For example, when the maximum width, head width, and shoulder width are selectable areas, it is shown that they can be selected with the radio buttons.

(Each function configuration of the image management device)
The communication unit 51 of the image management device 5 is mainly realized by processing of the network I / F 509 and the CPU 501 shown in FIG. 5, and various data with the communication terminal 3 or the terminal device 7 via the communication network 9. (An example of the first communication unit). In the following, even when the image management device 5 communicates with the terminal device 7, the description "via the communication unit 51" may be omitted.

The person detection unit 52 detects a person from the image data. This person is detected by a classifier obtained by machine learning. For example, there is a method of detecting an area in which a person appears from image data using a CNN (Convolutional Neural Network), which is a form of deep learning. Machine learning is a technology for making a computer acquire learning ability like a human being, and the computer autonomously generates an algorithm necessary for judgment such as data identification from learning data taken in advance, and newly It is a technology that applies this to various data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-teacher learning, enhanced learning, and deep learning, and may be a learning method that combines these learning methods. Machine learning It doesn't matter how you learn for.

The adjusting unit 53 adjusts the size of the heat map based on the size information of the heat map. That is, the size of the person used for the distribution expression is reduced, or the width of the person used for the distribution expression is made smaller than the width of the person detected by the person detection unit 52. Specifically, the size of the heat map to be created, such as the maximum width, the body, or the head, is acquired from the distribution expression size information storage unit 5005, and this size is detected among the detected persons. The width corresponding to the above is set to the coordinates 1 and 2 of the person range storage table. Details will be described later.

When transmitting the image data to the terminal device 7, the screen creation unit 54 creates screen information for the terminal device 7 to display the image data using HTML data, JavaScript (registered trademark), CSS, or the like. Since the image data is a three-dimensional spherical image, it cannot be displayed on the two-dimensional display 508. Therefore, the screen creation unit 54 includes a program for projecting the spherical image onto the display 508 by perspective projection conversion in the screen information. As a result, the viewer Y can operate the terminal device 7 to rotate the spherical image and display a part of the spherical image from an arbitrary direction with less distortion.

The screen creation unit 54 includes the heat map created by the distribution expression creation unit 55 in the screen information and transmits the heat map to the terminal device 7. In the terminal device 7, the spherical image and the heat map are superimposed and displayed.

The storage / reading unit 59 is mainly realized by the processing of the HDD 505 and the CPU 501 shown in FIG. 5, and stores various data in the storage unit 5000 and reads various data from the storage unit 5000. In the following, even when the image management device 5 reads / writes from the storage unit 5000, the description “via the storage / reading unit 59” may be omitted.

<Functional configuration of terminal device>
The terminal device 7 has a communication unit 71, a reception unit 72, a display control unit 73, and a storage / reading unit 79. Each of these parts has a function realized by operating any of the components shown in FIG. 5 by an instruction from the CPU 501 according to a program for the terminal device 7 developed on the RAM 503 from the HD 504. It is a means.

Further, the terminal device 7 has a RAM 503 shown in FIG. 5 and a storage unit 7000 constructed by the HD 504. The terminal program 7010 for the terminal device 7 is stored in the storage unit 7000. The terminal program 7010 for the terminal device 7 is, for example, a Web browser, but may be application software having a communication function such as a Web browser. Further, since the Web browser executes screen data to display the screen and accept operations, the terminal program 7010 includes screen data (HTML, JavaScript (registered trademark), etc.) transmitted by the image management device 5. Is done.

(Each function configuration of the terminal device)
The communication unit 71 of the terminal device 7 is realized mainly by the processing of the network I / F 509 and the CPU 501 shown in FIG. 5, and transmits / receives various data to / from the image management device 5 via the communication network 9 ( An example of the second communication unit). In the following, even when the terminal device 7 communicates with the image management device 5, the description "via the communication unit 71" may be omitted.

The reception unit 72 is realized mainly by the processing of the keyboard 511 and the mouse 512 shown in FIG. 5 and the CPU 501, and receives an operation input from the user (viewer Y in FIG. 2).

The display control unit 73 is realized mainly by the processing of the CPU 501 shown in FIG. 5, interprets the screen information transmitted from the image management device 5, and causes the display 508 of the terminal device 7 to display various screens. Control for

The storage / reading unit 79 is mainly realized by the processing of the HD 504 and the CPU 501 shown in FIG. 5, and stores various data in the storage unit 7000 and reads various data from the storage unit 7000. In the following, even when the terminal device 7 reads / writes from the storage unit 7000, the description “via the storage / reading unit 79” may be omitted.

<Example of human detection result>
FIG. 7 is a diagram illustrating an image in which a person is detected. Since the spherical image is captured at a circumference of 360 degrees, the entire image cannot be displayed on a flat display (it can be displayed, but the distortion becomes large). Further, although the image data after imaging is stored as an equirectangular image, distortion becomes large at the poles (upper and lower) of the equirectangular image. Therefore, if the control cylinder image is left as it is, the accuracy of detecting a person may decrease. Therefore, the person detection unit 52 converts the image data of the equirectangular image into a three-dimensional sphere, and then performs the person detection.

As shown in FIG. 7, the person detection unit 52 performs person detection on the entire range of the spherical image while switching a predetermined area having a size with less distortion. By doing so, it is possible to suppress a decrease in the detection accuracy of the person. The predetermined area is represented as a planar image by a perspective projection transformation that projects three-dimensional coordinates onto a planar display.

An example of the detection result of a person will be described with reference to FIG. FIG. 8 is a diagram schematically showing information obtained from the detection result of a person. FIG. 8 (a) shows the area of the person detected as the rectangle 210, FIG. 8 (b) shows the area of the person detected as the polygon 211, and FIG. 8 (c) shows the detection example of the human apex 212. Is shown. The human apex 212 includes not only the tips of the human body (head, hands, toes) but also movable parts such as joints (neck, shoulders, groin).

Which shape of FIG. 8A or FIG. 8B the person detection unit 52 detects a person is only an example because the developer can adjust it depending on how the teacher data is given at the time of learning. Further, a method in which the person detection unit 52 detects a human vertex 212 instead of a rectangle or polygon as shown in FIG. 8C can also be realized by a method of giving teacher data at the time of learning.

The image management device 5 of the present embodiment can adjust the size of the heat map regardless of the method of detecting the person, the shape of the detected person, the detected apex of the body, and the like.

<Relationship between person detection results and heat map>
As described with reference to FIG. 8, there are various methods for detecting a person, the shape of the detected person, or the detected apex of the body. The adjusting unit 53 adjusts the size of the heat map corresponding to these.

FIG. 9 is a diagram illustrating a method of adjusting the size of the heat map when a person is detected as a rectangle. FIG. 9A shows an example of a person detected by the rectangle 210. In this case, the maximum width may be substantially the same as the width of the rectangle 210. When the size information of the heat map adopted by the adjusting unit 53 according to the viewer Y and the application is the maximum width length, as shown in FIG. 9B, the adjusting unit 53 sets the width of the rectangle 210 to the size of the heat map. Let's say. Specifically, among the coordinates of the four vertices of the rectangle 210, the coordinates of the lower left and lower right vertices are registered in the person width storage table. In FIG. 9B, the ellipse under the foot represents the heat map 213.

On the other hand, where the torso is from is not clear from the rectangle 210, which is the detection result of the person in FIG. 9A. In this case, there are the following methods to make the fuselage the size of the heat map.
(i) The adjusting unit 53 requests the person detecting unit 52 to detect the body.
(ii) The person detection unit 52 detects not only the circumscribing rectangle but also the body part corresponding to the size information of the heat map in advance.
(iii) Multiply the maximum width by a ratio less than 1.

FIG. 9D shows an example of a plurality of rectangles 210 and 215 detected by the person detection unit 52. In FIG. 9D, not only the rectangle 210 meaning the maximum width and length but also the rectangle 215 surrounding the body is required. Since the person detection unit 52 can detect only the head and the body depending on how the teacher data is given, it is easy for the person detection unit 52 to detect the body part as shown in FIG. 9D. Therefore, the adjusting unit 53 can set the width of the rectangle 215 corresponding to the body as the size of the heat map by the methods (i) and (ii). Specifically, among the coordinates of the four vertices of the rectangle 215, the coordinates of the lower left and lower right vertices are registered in the person width storage table.

The method (iii) is a simple adjustment method utilizing the knowledge that the width of the fuselage is smaller than the maximum width. For example, the adjusting unit 53 calculates the size of the heat map from the rectangle 210 with 80% of the maximum width length as the size of the body. The ratio to be multiplied by the maximum width length may be determined according to the aspect ratio of the rectangle 210 shown in FIG. 9A. This is because the appropriate ratio differs between the person with his hands open and the person with his hands closed (the maximum width should be significantly reduced when the person's hands are open). In this case, the adjusting unit 53 has a table in which the ratio is associated with the aspect ratio.

When the body size is obtained by the above methods (i) to (iii), the distribution expression creating unit 55 can create a heat map of the body size as shown in FIG. 9 (c). In FIG. 9C, the ellipse 214 under the feet represents the heat map. In this way, a part of the rectangle can be determined as the width of the person used for the distribution expression.

FIG. 10 is a diagram illustrating a method of adjusting the size of the heat map when a person is detected by the polygon 216. As will be described later, the size of the heat map is the size of the heat map according to the length in the width direction (horizontal direction) of the detected area of the person. That is, it can be carried out by using the value of the coordinates of the area of the person. For example, the heat map can be drawn by obtaining the length in the width direction and the midpoint in the width direction from the coordinate values corresponding to the width direction of the area of the person. FIG. 10A shows an example of a person detected by the polygon 216. In this case, the maximum width may be substantially the same as the width of the polygon 216. As shown in FIG. 10B, when the adopted size information is the maximum width length, the adjusting unit 53 sets the width of the polygon 216 as the size of the heat map. Specifically, the leftmost and rightmost coordinates of the polygon 216 are registered in the person width storage table. In FIG. 10B, the ellipse 217 under the feet represents the heat map. Here, the distribution expression can be expressed within or in contact with the area of the corresponding person. For example, the person's head, abdomen, or feet can be considered. However, it is preferable to express the distribution under the feet, and it is preferable to make an elliptical heat map with the width of the detected person's area as the long side of the elliptical shape. Such a distribution expression is a preferable expression for showing the correspondence with space.

On the other hand, where the torso is from is not clear from the polygon 216 which is the detection result of the person in FIG. 10 (a). In this case, the method of using the body as the size of the heat map is the same as in the case of the rectangles (i) to (iii) in FIG.

FIG. 10D shows an example of a plurality of polygons 216 and 219 detected by the person detection unit 52. In FIG. 10D, not only the polygon 216 containing the entire person but also the polygon 219 containing the body is required. Since the person detection unit 52 can detect only the head and the body depending on how the teacher data is given, it is easy for the person detection unit 52 to detect the body part as shown in FIG. 10 (d). Therefore, the adjusting unit 53 can set the width of the polygon 219 corresponding to the body as the size of the heat map by the methods (i) and (ii).

Regarding the method (iii), since there is no concept of aspect ratio in polygons, the adjusting unit 53 prepares a table in which the ratio is associated with the ratio of the length of the base 216a to the leftmost and rightmost lengths 216b. It is good to have.

Further, when the detection result of the person is the polygon 216, as the (iv) th method, there is a method of regarding the left and right coordinates of the base 216a as the size of the body. With this method, the adjusting unit 53 can estimate the size of the body as it is from the result of the person detection, so that the processing load can be reduced.

When the size of the body is obtained as described above, the distribution expression creating unit 55 can create a heat map of the size of the body as shown in FIG. 10 (c). In FIG. 10 (c), the ellipse 218 under the feet represents the heat map.

FIG. 11 is a diagram illustrating a method of adjusting the size of the heat map when a person is detected at the apex 212 of the body. FIG. 11A shows an example of a person detected at the apex 212. In this case, the maximum width and length may be substantially the same as the spacing between the hands 220 and 221 (leftmost and rightmost vertices). As shown in FIG. 11B, when the adopted size information is the maximum width length, the adjusting unit 53 sets the widths of the hands 220 and 221 as the size of the heat map. Specifically, the coordinates of the hands 220 and 221 of the plurality of vertices 212 (however, the Y coordinates adopt the toes) are registered in the person width storage table. In FIG. 11B, the ellipse 222 under the feet represents the heat map.

On the other hand, it is not directly clear from the apex 212 which is the detection result of the person in FIG. 11A, but the apex that can be regarded as the torso is obtained. For example, the vertices of the shoulders 224 and 225 and the bases of the legs 226 and 227. Therefore, in this case, the adjusting unit 53 regards the widths of the vertices of the shoulders 224 and 225 and the bases of the legs 226 and 227 as the body and registers them in the person width storage table.

As shown in FIG. 11C, when the adopted size information is the torso, the distribution expression creation unit 55 creates a heat map with the width of the bases of the shoulders and legs as the size of the heat map. In FIG. 11 (c), the ellipse 223 under the feet represents the heat map.

<Size of heat map selected by viewer Y>
Next, with reference to FIG. 12, a screen example when the viewer Y selects the size of the heat map will be described. FIG. 12 is an example of the heat map size selection screen 230. FIG. 12A is a selection screen displayed by the viewer Y of the company whose selectable area is “head width, shoulder width, maximum width length” in the application-specific setting item table of Table 6 (a). Therefore, in FIG. 12A, the radio buttons 231 to 233 are displayed in association with the head width, shoulder width, and maximum width length. Viewer Y can select which heat map size he / she wants to create from the head width, shoulder width, and maximum width.

FIG. 12B is a selection screen 240 displayed by the viewer Y of the company whose selectable area is the “free selection length” in the application-specific setting item table of Table 6 (a). Therefore, in FIG. 12B, the slider 241 is displayed. The heat map of this slider 241 increases as it moves from right to left. The positions of the illustrated frame 242 and the slider 241 are linked. Viewer Y can change the size of the heat map by moving the slider 241 while looking at the frame 242. The position of the slider 241 determines what percentage of the maximum width of the person is the size of the heatmap. The percentage is transmitted to the image management device 5 and stored in the distribution expression size information storage unit 5005. The adjusting unit 53 is used to adjust the size of the heat map.

FIG. 12 (c) is a selection screen 250 displayed by the viewer Y of the company whose selectable area is “free selection length” in the application-specific setting item table of Table 6 (a) (separate from FIG. 12 (b)). Example). In FIG. 12C, two sliders 251 and 252 are displayed. The slider 252 on the right side corresponds to the position of the right end of the frame 253, and the slider 251 on the left side corresponds to the position of the left end of the frame 253. Viewer Y can select the size of the heat map by moving the two sliders while looking at the frame 253.

The positions of the sliders 251 and 252 determine which part of the person is the size of the heat map. The positions of the sliders 251 and 252 are transmitted to the image management device 5 and stored in the distribution expression size information storage unit 5005. The adjusting unit 53 is used to adjust the size of the heat map.

Further, as shown in FIG. 13, the viewer Y can also select a heat map creation method and a heat map color. FIG. 13 shows an example of the heat map creation screen 260 displayed by the terminal device 7. The heat map creation screen 260 has radio buttons 261,262 corresponding to an area with a person or an area without a person as a heat map creation method, and radio buttons 263 and a plurality of colors corresponding to a single color or a plurality of colors as the heat map color. It has 264. Viewer Y can select whether he wants to see a heat map that represents an area with or without a person. In addition, it is possible to select whether the color for creating the heat map is a single color or multiple colors. It is preferable that the viewer Y can further select a color in the case of a single color and in the case of a plurality of colors.

<How to create a heat map>
Subsequently, a method of creating a heat map will be described with reference to FIGS. 14, 15 and the like. FIG. 14 is a diagram schematically showing a method of creating a heat map, and FIG. 15 is a flowchart illustrating a method of creating a heat map. Since the method of creating the heat map is the same regardless of the size of the heat map, FIG. 14 describes a case where the size of the heat map is adjusted to the maximum width. The shape of the heat map can be designed by the developer as appropriate, but here it is an ellipse.

(S1)
First, the distribution expression creation unit 55 acquires the size of the heat map (coordinates 1, coordinates 2) from the person width storage table.

(S2)
The distribution expression creation unit 55 sets the center 270 of the coordinates 1 and 2 (FIG. 14 (a)). Coordinates 1 and 2 also include the coordinates of the lower end of the detected person, and the center 270 is set under the feet.

(S3)
Next, the distribution expression creation unit 55 draws an ellipse 271 having coordinates 1 and coordinates 2 as long axes (FIG. 14 (b)). The size of the minor axis is determined by a predetermined ratio with the major axis.

(S4)
The distribution expression creation unit 55 statistically processes the overlap of the ellipses 271 similarly generated for each person to determine the density of each heat map. Details will be described with reference to FIGS. 16 and 17.

FIG. 16 is a diagram illustrating a method for determining the density of the heat map, and FIG. 17 is a flowchart showing a method for determining the density of the heat map. In FIG. 16, four ellipses 280 are drawn. The following regions are formed by the four ellipses 280.
A. Areas that do not overlap B. Region where two ellipses overlap C. Area where three ellipses overlap The distribution expression creation unit 55 can specify how many ellipses overlap at the pixel level. The period for creating the heat map (the number of image data) is set by the viewer, so it is not always constant. For example, from the beginning of the day to the current time, an arbitrary period specified by the viewer Y, a period such as during a sale, and the like may be various. Therefore, although the number of ellipses 280 varies, the distribution expression creation unit 55 processes as follows in order to represent the relative retention of people in the image data to be analyzed.

(S11)
The distribution expression creation unit 55 counts the number of overlapping ellipses 280 for each pixel.

(S12)
The distribution expression creation unit 55 groups the pixels into the number of colors prepared in advance according to the number of overlapping pixels. Since three colors are prepared in FIG. 16, the pixels are divided into three groups in descending order of number. The number of pixels in each group can be the same, but can be different. Therefore, for example, A. Non-overlapping region, B.I. Region where two ellipses overlap, C.I. The area where the three ellipses overlap is color-coded.

(S13)
The distribution expression creation unit 55 assigns a predetermined color to the pixels of the group. By doing so, it is possible to create a heat map represented by a darker color as the number of overlapping ellipses increases.

Although the heat map for emphasizing the area where the person is staying is created in FIG. 16, the distribution expression creating unit 55 can also create the heat map for emphasizing the area where the person does not exist. In this case, in step S13, the distribution expression creation unit 55 creates a heat map represented by a darker color as the number of overlapping ellipses is smaller.

<Other examples of heat map shapes>
Although an elliptical heat map is shown in FIG. 16 and the like, various shapes of the heat map are designed. FIG. 18 is a diagram showing some variations in the shape of the heat map. FIG. 18A is a heat map of the ellipse 291 described so far.

FIG. 18B shows a heat map with a solid line 292. FIG. 18C shows a heat map with a broken line 293. Both the solid line 292 and the broken line 293 have a certain thickness so that the viewer can easily see them. Therefore, the heat map of FIG. 18B can be expressed as a rectangle. In addition, the heat map may have a rhombic shape, a trapezoidal shape, or a cloud-like shape. As described here, the size of the heat map is determined in association with the size of the detected person, so that the distribution expression is easy to see. Further, various shapes can be applied to the shape of the heat map.

<Example of heat map expression>
FIG. 19 is an example of a heat map displayed in a different color for each person. As shown in FIG. 19, the distribution expression creation unit 55 may create a heat map separately for each individual person. In this case, the person detection unit 52 has a function of not only detecting the person but also tracking the same person. Alternatively, the individual may be further identified. As a method of tracking the same person, the person detection unit 52 detects the feature amount (color histogram, shape, arrangement of facial parts, etc.) of the person first detected during the analysis period, and the detection is performed. There is a method of tracking a person by pattern matching with a feature amount.

The distribution expression creation unit 55 creates a heat map for each of the same person by the method described with reference to FIGS. 16 and 17. At this time, a heat map is created for the same person using only the same color system. For example, person A is red, person B is yellow, person C is green, and so on. Since different colors are used in the same system color, the color of the place where the person stays can be darkened.

By changing the system color for each person in this way, it becomes easier for the viewer to pay attention to the trend of a certain visitor and analyze it.

<Heat map showing momentary occupancy>
FIG. 20 shows an example of displaying a heat map at a certain moment. As shown in FIG. 20A, the image pickup apparatus 1 transmits image data to the image management apparatus 5 one after another. In FIG. 16, a heat map is created from image data for a certain period of time, but in FIG. 20 (a), the distribution expression creation unit 55 creates a heat map from only one image data. An example of the heat map in this case is shown in FIG. 20 (b). The distribution expression creation unit 55 can emphasize that the person has been detected by the process of drawing the ellipse 294 if the person is detected and not drawing the ellipse 294 if the person is not detected. That is, the distribution expression creating unit 55 expresses whether or not there is a person by the distribution expression.

Whereas a general heat map expresses the retention of a person for a certain period of time as a concentration as shown in FIG. 20 (c), the heat map of FIG. 20 (b) shows whether or not a person is present at a certain moment. Can be grasped at a glance. For example, it is effective in a store where visitors can easily hide with goods.

<Superposition of heat map image and image data>
Subsequently, the superposition of the heat map image and the image data will be described with reference to FIGS. 21 to 23. FIG. 21 is a diagram schematically showing an overlay of the original image data and the heat map image.

FIG. 21 (a) is the original image data, and FIG. 21 (b) is a heat map image. The contents of the original image data are omitted. 21 (a) and 21 (b) are equirectangular images. Since it is preferable that the original image data can be seen when the terminal device 7 superimposes the heat map image on the original image data, the heat map image is created as a transparent image.

By superimposing FIGS. 21 (a) and 21 (b) as they are, the screen creation unit 54 can create a superimposed image in which the original image data can be seen through the heat map (FIG. 21 (c)).

Next, the conversion to a three-dimensional sphere will be described. Since the spherical image is viewed in the state of a three-dimensional sphere, the screen creation unit 54 converts the equirectangular image into a three-dimensional sphere. As shown in FIG. 21D, the screen creation unit 54 attaches the original image data (inside) and the heat map image (outside) to the sphere in this order.

FIG. 22 is a diagram illustrating the correspondence between the coordinates of the equirectangular image and the three-dimensional coordinates. The equirectangular image shows the coordinates in a spherical coordinate system in the range of θ (longitude) = 0 to 360 degrees and φ (latitude) = 0 to 180 degrees as shown in FIG. 22 (a). It is possible to paste the image data and the heat map image on the 3D sphere as it is in this coordinate system, but considering that the viewer Y rotates the 3D sphere, it is preferable to convert it to a 3D coordinate system suitable for rotation. ..

FIG. 22B is a diagram illustrating the conversion between the spherical coordinate system and the three-dimensional coordinate system. As is clear from the figure, there is the following relationship between arbitrary coordinates (φ1, θ1) in the spherical coordinate system and arbitrary coordinates (x1, y1, z1) in the three-dimensional coordinate system.

画面作成部５４は式（１）〜（３）で画像データとヒートマップ画像を立体球に貼り付けることができる。

The screen creation unit 54 can paste the image data and the heat map image on the three-dimensional sphere by the equations (1) to (3).

FIG. 23 is a diagram showing the actual original image data and the heat map image. FIG. 23A is the original image data (equirectangular image). The actual image data is represented by a diagram. FIG. 23B is a heat map image. FIG. 23 (c) is a duplicate image in which a heat map image of a transparent image is attached to the image data.

<Overall operation procedure>
FIG. 24 is an example of a sequence diagram for explaining the processing performed by the image processing system.
S101, S102: The image data captured by the image pickup device 1 is transmitted to the image management device 5 via the communication terminal 3. The communication unit 51 of the image management device 5 receives the image data and stores it in the image data storage unit 5003.

S103: The viewer Y operates the terminal device 7, and the communication unit 71 of the terminal device 7 requests the analysis of the image data by the heat map. For example, suppose that the image data of a certain day is analyzed. It should be noted that the viewer Y is specified by logging in which company he belongs to.

S104: The communication unit 51 of the image management device 5 receives the analysis request, and the person detection unit 52 acquires one day's image data from the image data storage unit 5003. Then, a person is detected from each image data and registered in the person detection result table of Table 4 (a).

S105: Next, the adjusting unit 53 acquires the size information of the heat map of the viewer Y from the distribution expression size information storage unit 5005, and adjusts the size (width) of the heat map. That is, the X coordinate and the Y coordinate, which are the widths of the heat map, are determined based on the head width, shoulder width, or maximum width length. The adjustment method has been described with reference to FIGS. 9 to 11. The adjusting unit 53 registers the coordinates 1 and the coordinates 2 indicating the width of the heat map in the person range storage table shown in Table 4 (b).

S106: The distribution expression creation unit 55 acquires the width (coordinates 1 and 2) of the heat map of each image data from the person range storage table, draws an ellipse, and draws the heat map image as described in FIG. create.

S107: The screen creation unit 54 converts the heat map image and the image data into a three-dimensional sphere, and creates a Web page (screen information) for the terminal device 7 to display these. The communication unit 51 transmits screen information to the terminal device 7.

S108: The communication unit 71 of the terminal device 7 receives screen information. The display control unit 73 superimposes the heat map image on the image data, performs perspective projection conversion, and displays the solid on a flat display. The terminal device 7 may perform the conversion to a three-dimensional sphere.

<Main effect>
As described above, the image processing system of the present embodiment can adjust the size of the heat map, which has been fixed in the past. For example, since the size of the heat map is adjusted according to the user or the application, the size of the heat map can be changed.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

In the embodiment, the size of the heat map is adjusted according to the viewer Y or the application, but the size of the heat map may be adjusted for each person. For example, the person detection unit 52 may distinguish not only a person but also men and women and ages, and may change the size of the heat map depending on the ages of men and women.

Further, as a method of adjustment, a method of making the person smaller than the circumscribed rectangle of the person has been described, but the heat map may be made larger than the detection range of the original person. For example, the person detection unit 52 detects the body, but the adjustment unit 53 can be adjusted to expand to the maximum width.

Further, instead of adjusting the size, the adjusting unit 53 may change the shape of the heat map. For example, one viewer Y displays an elliptical heat map, and another viewer Y displays a solid heat map. The shape of the heat map may be adjusted for each person detected in the image.

Further, the configuration examples such as FIG. 6 shown in the above examples correspond to the main functions in order to facilitate the understanding of the processes of the image pickup device 1, the communication terminal 3, the image management device 5, and the terminal device 7. It is divided. However, the present invention is not limited by the method and name of division of each processing unit. The processing of the image pickup device 1, the communication terminal 3, the image management device 5, and the terminal device 7 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing. Further, the image processing system 200 may have a plurality of image management devices 5.

Also, the group of devices described in the examples merely represents one of a plurality of computing environments for implementing the embodiments disclosed herein. In certain embodiments, the image management device 5 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including networks, shared memory, and the like, and perform the processes disclosed herein.

Further, the image management device 5 can be configured to share the disclosed processing steps, eg, FIG. 24, in various combinations. For example, a process executed by a predetermined unit can be executed by a plurality of information processing devices included in the image management device 5. Further, the image management device 5 may be integrated into one server device, or may be divided into a plurality of devices.

Further, the database of the storage unit 5000 of the image management device 5 may be directly owned by the image management device 5 or may be on a communication network 9 on which the image management device 5 can read and write.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

1 Imaging device 3 Communication terminal 5 Image management device 7 Terminal device 200 Image processing system

Japanese Unexamined Patent Publication No. 2004-94518

Claims (15)

A person detection unit that detects a person from the image data captured by the image pickup device, and
A distribution expression creation unit that creates a distribution representation of the person using the size of the person detected by the person detection unit, and a distribution expression creation unit.
An information processing device characterized by having. It has an adjustment unit that adjusts the size of the person detected by the person detection unit.
The information processing apparatus according to claim 1, wherein the distribution expression creating unit creates a distribution expression of the person using the size of the person adjusted by the adjusting unit. The information processing device according to claim 2, wherein the adjusting unit makes the size of the person used for distribution expression smaller than the size of the person detected by the person detecting unit. The information processing device according to claim 3, wherein the adjusting unit makes the width of the person used for distribution expression smaller than the width of the person detected by the person detecting unit. The information processing apparatus according to any one of claims 2 to 4, wherein the adjusting unit adjusts the size of the person by using information on the size of the distribution expression. The information processing apparatus according to claim 5, wherein the information regarding the size of the distribution expression is the maximum width, shoulder width, or head width. The person detection unit detects the person from the image data with a rectangle including the entire person.
The information according to claim 5 or 6, wherein the adjusting unit determines a part of the rectangle to be the width of the person used for the distribution expression according to the information regarding the size of the distribution expression. Processing equipment. The information processing device according to any one of claims 5 to 7, wherein the information regarding the size of the distribution expression is registered in the information processing device in association with the user. An image data storage unit that stores image data captured by the image pickup device, and
A person detection unit that detects a person from the image data,
A distribution expression creation unit that creates a distribution representation of the person using the size of the person detected by the person detection unit, and a distribution expression creation unit.
An information processing device having the image data and a first communication unit that transmits the distribution representation to the terminal device.
A program for displaying the distribution representation on the image data on the terminal device,
An image processing system characterized by having. The information processing device has an adjusting unit that adjusts the size of the person detected by the person detecting unit.
The image processing system according to claim 9, wherein the distribution expression creating unit creates a distribution expression of the person using the size of the person adjusted by the adjusting unit. The program further adds to the terminal device.
A display control unit that displays a screen that accepts the selection of information regarding the size of the distribution expression,
A reception unit that accepts selection of information regarding the size of the distribution expression,
It functions as a second communication unit that transmits information regarding the size of the distribution expression received by the reception unit to the information processing device.
The information processing device
Information on the size of the distribution representation transmitted from the terminal device is stored and
The image processing system according to claim 10, wherein the adjusting unit adjusts the size of the person by using information on the size of the distribution expression. The display control unit displays a frame surrounding a person and a slider on which the user changes information regarding the size of the distribution expression on the screen.
The image processing system according to claim 11, wherein the reception unit accepts the movement of the slider, and the display control unit changes the width of the frame according to the movement of the slider. The information processing device
It holds a list of information regarding the size of the distribution expression that is predetermined according to the user.
The list is transmitted to the terminal device, and the list is transmitted to the terminal device.
The display control unit displays information on the size of the distribution expression from the list, and displays the information.
The image processing system according to claim 11, wherein the reception unit accepts selection of information regarding the size of the distribution expression. The display control unit selects whether to express the area occupied by a person in a distribution expression or the part without a person in a distribution expression.
Alternatively, display a screen that accepts settings related to the color of the distribution expression, and display it.
The reception unit accepts operations on the screen,
The image processing system according to any one of claims 11 to 13, wherein the distribution expression creating unit creates a distribution expression of the existence of the person in response to an operation received by the reception unit. The information processing device has a person detection unit that detects a person from the image data captured by the image pickup device, and
A distribution expression creation unit that creates a distribution expression of the person using the size of the person detected by the person detection unit,
A program to function as.

Images