JP6852293B2 - Image processing system, information processing device, information terminal, program - Google Patents

Description

The present invention relates to an image processing system, an information processing device, an information terminal and a program.

In the retail industry, by using POS (Point Of System) sales data, it has been conventionally practiced to discriminate between products that are selling and products that are not selling, and store managers and the like to formulate improvement measures for each product. Specifically, in addition to the conventional improvement measures such as reviewing the price setting, the appearance of the product called VMD (Visual Merchandising) has been improved.

However, with the method of referring to the sales data of POS, it is possible to know "whether or not the product was sold", but not "whether or not the visitor was interested in each product", so it is not always an appropriate improvement measure. Was sometimes unable to be formulated.

Therefore, a monitoring system has been devised that processes an image by combining a camera and an image processing technique and outputs information that can be easily analyzed by a store manager or the like (see, for example, Patent Document 1). Patent Document 1 discloses a technique of determining a visitor's purchasing behavior from an image to a store and acquiring a visitor's behavior toward a product that does not appear in the POS data. If such a technology is utilized, it becomes easy to analyze the behavior of the visitor toward the product, which does not appear in the sales data of POS.

However, even if the conventional technology can determine two types of "whether or not the visitor has accessed the product (or the shelf on which the product is placed)", how much the visitor is interested in the product. There was a problem that it could not be determined. In other words, if a visitor accesses the product, it is determined that he / she is "interested" in the product, and if he / she does not access the product, it is determined that he / she is "not". I couldn't tell if I was interested in the product.

In view of the above problems, an object of the present invention is to provide an image processing system capable of determining the degree of interest of a person in more detail.

The present invention is an image processing system including one or more information processing devices, wherein a person detecting means for detecting a person area including a person from image data and the person area detected by the person detecting means are used. Depending on the motion determining means for determining the motion of a person and the direction of the motion, the region determined by being in the direction among a plurality of predetermined regions, or the direction and the size of the human region. Yes and evaluating means for evaluating in accordance with the prior SL region the operation is determined, and display means for displaying on the display device evaluation by the evaluation unit together with the image data, and a reception unit that receives designation of the area Te Then, the evaluation means imparts the evaluation stage to the region when the region is evaluated in multiple stages, and assigns the same evaluation stage to the region of the plurality of image data in the region. Counting each time, the display means visualizes and displays the number of times of each of the evaluation stages associated with the area received by the reception means .

It is possible to provide an image processing system capable of determining the degree of interest of a person in more detail.

It is a figure explaining the example of the judgment of the degree of interest of a person by an image processing system. This is an example of a schematic configuration diagram of an image processing system. This is an example of the hardware configuration diagram of the image pickup device. This is an example of a hardware configuration diagram of a communication terminal in the case of a cradle having a wireless communication function. This is an example of a hardware configuration diagram of an image management device and an information terminal. This is an example of a functional block diagram of an image pickup device, a communication terminal, an image management device, and an information terminal that form a part of an image processing system. This is an example of a diagram schematically showing a region in which a person is detected. This is an example of a flowchart showing a procedure in which the image analysis unit determines an action. This is an example of a diagram schematically explaining the aggregation for each product zone. This is an example of a flowchart showing a procedure for determining a product zone related to an area. It is a figure which shows typically the definition method of the product zone. This is an example of a flowchart showing a procedure in which the evaluation unit defines a product zone. It is each function block diagram of an image pickup apparatus, a communication terminal, an image management apparatus, and an information terminal (Example 3). This is an example of a diagram showing the degree of interest specified by the viewer. It is an example of the figure which shows the screen example which the information terminal displays for operation 1. It is an example of the figure which shows the screen example which the information terminal displays for operation 2. It is an example of the figure which shows the screen example which the information terminal displays for operation 3. It is an example of the figure which shows the screen example which the information terminal displays for operation 4. It is an example of the figure which shows the screen example which the information terminal displays for operation 5. It is an example of the figure which shows the screen example which the information terminal displays for operation 6. This is an example of a diagram for explaining weighting regarding the creation of a heat map. This is an example of a schematic configuration diagram of an image processing system (Example 4). This is an example of the hardware configuration diagram of the staff terminal. It is each function block diagram of the image management apparatus, the information terminal and the staff terminal which constitute a part of the image processing system of this Example (Example 4). This is an example of a diagram illustrating a usage pattern of an image processing system in a store. It is a figure which shows the store where the event that a visitor collapses occurred. It is a figure which shows an example of the event recording screen displayed on the display of a staff terminal. It is an example of the figure schematically explaining the event flag set by the event record request. This is an example of a sequence diagram showing the overall operation procedure of the image processing system. This is an example of a flowchart showing a procedure in which an image management device manages images. This is an example of a flowchart showing a procedure in which the image management device detects a gesture and sets the event flag to ON. This is an example of a sequence diagram showing an operation performed between the staff terminal and the image management device. This is an example of a sequence diagram in which a communication terminal transmits an event recording request to an image management device. This is an example of a browsing screen displayed on the display of an information terminal. This is an example of a diagram showing a visitor composition screen in which a visitor is synthesized with image data (first embodiment). This is an example of a flowchart showing a procedure in which the information visualization unit synthesizes a visitor with image data. This is an example of a diagram showing image data displayed on a display by an information terminal (second embodiment). This is an example of a flowchart showing a procedure in which the information visualization unit synthesizes a visitor with image data. This is an example of a diagram illustrating an analysis operation performed by a viewer (third embodiment). It is a figure which shows an example of the visitor composition screen. This is an example of a flowchart showing a procedure in which the information visualization unit synthesizes a visitor with image data. This is an example of a diagram illustrating a method of generating image data displayed in the fourth embodiment (fourth embodiment). This is an example of a diagram in which image data showing visitors showing interest in the same product zone are displayed in chronological order together with a visitor composition frame. This is an example of a flowchart showing a procedure in which the information visualization unit synthesizes a visitor with image data. It is an example of the figure explaining the image data displayed on the display (fifth embodiment). This is an example of a flowchart showing a procedure in which the information visualization unit synthesizes a visitor with image data. It is an example of the figure explaining the image data displayed on the display (the sixth embodiment). This is an example of a flowchart showing a procedure in which the information visualization unit synthesizes a visitor with image data.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Example of image analysis of this embodiment>
FIG. 1 is a diagram illustrating an example of determining the degree of interest of a person by an image processing system. In FIG. 1, a case where a visitor visits a store in the apparel industry will be described as an example. Visitors walk around the store to find the products they want, or search for products that interest them. The image processing system captures the visitor with wide-angle cameras installed in various places, analyzes the image data, and determines the action of the visitor.

As shown in FIG. 1A, the image processing system can first determine whether or not the visitor is walking. When walking, the image processing system can further determine whether the product is facing the direction of travel or the direction of the product based on the face or the like. If it is not facing the direction of travel, it can be considered that the visitor has seen some kind of product, so it can be determined that the visitor has seen the direction of the product while walking.

Further, as shown in FIG. 1B, when the visitor is stopped, it can be determined that the visitor is stopping and looking at some product. It can be judged that the degree of interest is higher when watching while stopping than when watching while walking. Further, as shown in FIG. 1C, the image processing system can analyze the image and determine whether or not the visitor has reached out. It can be judged that the degree of interest is higher when looking at it with a hand reached than when looking at it while stopping.

Further, as shown in FIG. 1D, the image processing system can analyze whether or not the visitor puts the clothes, which are the products, on the body. In this case, it can be judged that the degree of interest is higher than when the person is reaching out and looking at it.

As described above, the image processing system of the present embodiment can determine the degree of interest in the product or the product zone in multiple stages by determining various operations from the images obtained by capturing the state of the visitors in the store. it can. Further, the target of the visitor's movement is the target of interest, and if the product zone defined in advance is determined according to the direction of the movement, it is possible to visually display the strength of interest in the product zone.

<Terminology>
The wide-angle image is an image having a wider angle of view than a predetermined value, and is at least an image having a wider angle of view than that of a normal image pickup device (for example, a camera of a 35 mm film). Alternatively, it refers to a curved image such as an image captured by a fisheye lens. An example of a wide-angle image, an omnidirectional image, is an image captured in all directions with a circumference of 360 °. It may be called any way, such as an omnidirectional image, a panoramic image, and an omnidirectional panoramic image. Further, if a part of the image having a circumference of 360 ° is missing due to image processing or the like, it does not interfere with the explanation of the present embodiment, it is referred to as a spherical image. Since the whole celestial sphere image can cover a wide area, the number of image pickup devices installed can be reduced.

A person is a person who exhibits an action such as interest, interest, reaction or response. The action determines the degree of interest, interest, reaction or response. In addition to humans, animals and the like may be detected. The person may be detected indoors or outdoors. Goods and services are provided indoors or outdoors. Goods and services are of interest.

Evaluation means determining or judging the degree of this interest. In this embodiment, the degree of interest is evaluated in multiple stages. The multi-stage is preferably at least 3 stages or more.

<System configuration of image processing system>
FIG. 2 is an example of a schematic configuration diagram of the image processing system 200. The image processing system 200 includes an image management device 5, an image pickup device 1, a communication terminal 3, and an information terminal 7 connected via a communication network 9. The image pickup device 1 is installed in the store by the installer X. The information terminal 7 is operated by the viewer Y.

The communication network 9 is constructed including at least one of the LAN of the company to which the viewer Y belongs in the store, the provider network of the provider that connects the LAN to the Internet, and the line provided by the line operator. There is. When the communication terminal 3 or the information terminal 7 directly connects to the line telephone network or the mobile telephone network without going through the LAN, it can be connected to the provider network without going through the LAN. In addition, the communication network includes WAN and the Internet. The communication network may be constructed either wired or wireless, and may be a combination of wired and wireless.

As described above, the image pickup apparatus 1 is a camera that captures an ambient 360-degree image with one image capture and creates an omnidirectional image. Sometimes referred to as a digital still camera or digital video camera. Further, when the communication terminal 3 is equipped with a camera, the communication terminal 3 can be a digital camera. In the present embodiment, in order to make the explanation easy to understand, the image pickup apparatus 1 will be described as a digital camera for obtaining a spherical image. The image pickup apparatus 1 periodically images the surrounding 360. It does not necessarily have to be periodic, and may be imaged irregularly, may be imaged by the operation of the installer X, or may be imaged from the image management device 5 by the viewer Y requesting the image management device 5. You may take an image by the command of.

The image pickup device 1 may automatically capture several landscapes having different line-of-sights and combine a plurality of image data to create a spherical image.

The communication terminal 3 has a communication function of connecting to the communication network 9 instead of the image pickup device 1. The communication terminal 3 is a cradle for supplying electric power to the image pickup device 1 and fixing it to a store. The cradle is an expansion device that expands the functions of the image pickup apparatus 1. The communication terminal 3 has an interface for connecting to the image pickup device 1, whereby the image pickup device 1 can use the functions of the communication terminal 3. The communication terminal 3 performs data communication with the image pickup apparatus 1 via this interface. Then, data communication is performed with the image management device 5 via the wireless router 9a and the communication network 9.

If the image pickup device 1 has a function of directly communicating data with the wireless router 9a or the communication network 9, the communication terminal 3 may not be provided. Alternatively, the image pickup device 1 and the communication terminal 3 may be integrally configured.

The image management device 5 is, for example, an information processing device that functions as a server, and can perform data communication with the communication terminal 3 and the information terminal 7 via the communication network 9. OpenGL ES (API: Application Interface for 3D graphics) is installed in the image management device 5. By calling OpenGL ES, it is possible to create a spherical image from a Mercator image, or to create a thumbnail image of a part of the spherical image (predetermined area image).

It is preferable that cloud computing is applied to the image management device 5. There is no strict definition for the physical configuration of cloud computing, but the configuration of the information processing device is achieved by dynamically connecting and disconnecting resources such as the CPU, RAM, and storage that make up the information processing device according to the load. It is known that the installation location can be changed flexibly. Further, in cloud computing, the image management device 5 is generally virtualized. One information processing device can provide a function as a plurality of image management devices 5 by virtualization, and a plurality of information processing devices can provide a function as one image management device 5 by virtualization. .. It is also possible that the image management device 5 is provided not as cloud computing but by a single information processing device.

The information terminal 7 is, for example, a notebook PC (Personal Computer), and performs data communication with the image management device 5 via the communication network 9. The information terminal 7 may be a tablet terminal, a PC, a PDA (Personal Digital Assistant), an electronic blackboard, a video conference terminal, a wearable PC, a game machine, a mobile phone, a car navigation system, a smartphone, or the like, in addition to a notebook PC. Moreover, it is not limited to these.

The image pickup device 1, the communication terminal 3, and the wireless router 9a are installed at predetermined positions by the installer X at each sales base such as a store. The information terminal 7 is installed at the head office or the like that controls each sales base, and by displaying an image showing the status of each base sent via the image management device 5, the viewer Y can display the status of each base. You can browse the image showing. However, the information terminal 7 can communicate with the image management device 5 from a place other than the head office. The image management device 5 transmits the image data sent from the communication terminal 3 of each base and the analysis result thereof to the information terminal 7.

<Hardware configuration of the embodiment>
Next, the hardware configurations of the image pickup device 1, the communication terminal 3, the information terminal 7, and the image management device 5 of the present embodiment will be described with reference to FIGS. 3 to 5.

<< Imaging device >>
FIG. 3 is an example of a hardware configuration diagram of the image pickup apparatus 1. In the following, the image pickup device 1 is an omnidirectional image pickup device using two image pickup elements, but the number of image pickup devices may be three or more. Further, the device does not necessarily have to be a device dedicated to omnidirectional imaging, and by attaching a retrofitted omnidirectional imaging unit to a normal digital camera, smartphone, or the like, it may have substantially the same function as the imaging device 1. ..

As shown in FIG. 3, the imaging device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only Memory). ) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, and antenna 117a.

Of these, the image pickup unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two image pickups provided corresponding to each wide-angle lens. It includes elements 103a and 103b. The image sensors 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images from fisheye lenses into image data of electrical signals and output them. It has a timing generation circuit that generates a vertical sync signal, a pixel clock, and the like, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The image sensors 103a and 103b of the image pickup unit 101 are each connected to the image processing unit 104 by a parallel I / F bus. On the other hand, the image pickup elements 103a and 103b of the image pickup unit 101 are connected by a serial I / F bus (I2C bus or the like) separately from the image pickup control unit 105. The image processing unit 104 and the image pickup control unit 105 are connected to the CPU 111 via the bus 110. Further, a ROM 112, a SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

The image processing unit 104 takes in the image data output from the image pickup devices 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data. , Create Mercator image data.

The image pickup control unit 105 generally uses the image pickup control unit 105 as a master device and the image pickup elements 103a and 103b as slave devices, and sets commands and the like in the register group of the image pickup elements 103a and 103b by using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the image pickup control unit 105 also uses the I2C bus to take in the status data and the like of the register group of the image pickup elements 103a and 103b and send them to the CPU 111.

Further, the image pickup control unit 105 instructs the image pickup devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some image pickup devices 1 may have a preview display function on a display and a function corresponding to a moving image display. In this case, the image data is continuously output from the image sensors 103a and 103b at a predetermined frame rate (frames / minute).

Further, as will be described later, the image pickup control unit 105 also functions as a synchronization control means for synchronizing the output timings of the image data of the image pickup elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the image pickup apparatus 1 is not provided with a display unit, but a display unit may be provided.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the image pickup apparatus 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data during processing by the image processing unit 104 and data of the processed Mercator image.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. By operating the operation buttons, the user inputs various imaging modes, imaging conditions, and the like.

Network I / F116 is a general term for interface circuits (USB I / F, etc.) with external media such as SD cards and personal computers. Further, the network I / F116 may be a network interface regardless of whether it is wireless or wired. The Mercator image data stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or if necessary, the communication terminal 3 or the like via the network I / F 116 which becomes the network I / F. It may be sent to an external device.

The communication unit 117 communicates with an external device such as a communication terminal 3 via a remote radio technology such as WiFi (wireless fidelity), NFC, or LTE (Long Term Evolution) via an antenna 117a provided in the image pickup device 1. Do. The communication unit 117 can also transmit the Mercator image data to the external device of the communication terminal 3.

The electronic compass 118 calculates the orientation and tilt (Roll rotation angle) of the image pickup device 1 from the magnetism of the earth, and outputs the orientation / tilt information. This orientation / tilt information is an example of related information (metadata) along with Exif, and is used for image processing such as image correction of captured images. The related information also includes each data of the image capture date and time and the data capacity of the image data.

<< Communication terminal >>
Next, the hardware configuration of the communication terminal 3 will be described with reference to FIG. Note that FIG. 4 is a hardware configuration diagram of the communication terminal 3 in the case of a cradle having a wireless communication function.

As shown in FIG. 4, the communication terminal 3 includes a CPU 301 that controls the operation of the entire communication terminal 3, a ROM 302 that stores basic input / output programs, and a RAM (Random Access Memory) 304 that is used as a work area of the CPU 301. It has a communication unit 305 for data communication via Wi-fi, NFC, LTE, etc., a USB I / F 303 for wired communication with the image pickup device 1, and an RTC (Real Time Clock) 306 for holding a calendar and time information. There is.

Further, a bus line 310 such as an address bus or a data bus for electrically connecting each of the above parts is provided.

The ROM 302 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

The communication unit 305 communicates with the wireless router 9a and the like by using the antenna 305a and using the wireless communication signal.

In addition to the illustration, it is equipped with a GPS receiver that receives GPS signals including the position information (latitude, longitude, and altitude) of the communication terminal 3 by GPS (Global Positioning Systems) satellite or IMES (Indoor MEssaging System) as indoor GPS. You may be.

<< Image management device, information terminal 7 >>
The hardware configuration of the information terminal 7 in the case of the image management device 5 and the notebook PC will be described with reference to FIG. Note that FIG. 5 is a hardware configuration diagram of the image management device 5 and the information terminal 7. Since both the image management device 5 and the information terminal 7 are computers, the configuration of the image management device 5 will be described below. It is assumed that the configuration of the information terminal 7 is the same as that of the image management device 5, and even if there is a difference, there is no problem in the description of the present embodiment.

The image management device 5 includes a CPU 501 that controls the operation of the entire image management device 5, a ROM 502 that stores a program used to drive the CPU 501 such as an IPL, and a RAM 503 that is used as a work area of the CPU 501. Further, it has an HDD (Hard Disk Drive) 505 that controls reading or writing of various data to the HD 504 according to the control of the HD 504 and the CPU 501 that store various data such as a program for the image management device 5. It also has a media drive 507 that controls reading or writing (storage) of data to a recording medium 506 such as a flash memory, and a display 508 that displays various information such as cursors, menus, windows, characters, or images. It is preferable that the display 508 is equipped with a touch panel. In addition, a network I / F 509 for data communication using the communication network 9, a keyboard 511 equipped with a plurality of keys for inputting characters, numerical values, various instructions, etc., selection and execution of various instructions, and processing targets It has a mouse 512 for selecting, moving a cursor, and the like. Further, it has a CD-ROM drive 514 that controls reading or writing of various data to a CD-ROM (Compact Disc Read Only Memory) 513 as an example of a detachable recording medium. Further, as shown in FIG. 5, each of the above components is provided with a bus line 510 such as an address bus or a data bus for electrically connecting the components.

<About the function of the image processing system>
FIG. 6 is a functional block diagram of the image pickup device 1, the communication terminal 3, the image management device 5, and the information terminal 7, which form a part of the image processing system 200 of the present embodiment.

<< Functional configuration of imaging device 1 >>
The image pickup device 1 includes a reception unit 12, an image pickup unit 13, a sound collection unit 14, a connection unit 15, and a storage / reading unit 19. Each of these parts has a function or means realized by operating any of the components shown in FIG. 3 by a command from the CPU 111 according to a program for an imaging device developed on the DRAM 113 from the SRAM 113. Is.

Further, the image pickup apparatus 1 has a storage unit 1000 constructed by one or more of the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. The storage unit 1000 stores a program for the imaging device and a terminal ID.

The reception unit 12 of the image pickup apparatus 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 shown in FIG. 3, and receives an operation input from the user (installer X in FIG. 2). The imaging device 1 automatically and periodically images the surroundings even if there is no operation for imaging by the installer X. The periodic interval may be set by the installer X in the image pickup device 1 or by the viewer Y via the image management device 5.

The image pickup unit 13 is realized mainly by the processing of the image pickup unit 101, the image processing unit 104, the image pickup control unit 105, and the CPU 111 shown in FIG. 3, and images a landscape or the like to create image data.

The sound collecting unit 14 is mainly realized by the processing of the microphone 108, the sound processing unit 109, and the CPU 111 shown in FIG. 3, and collects the sound around the image pickup device 1.

The connection unit 15 is mainly realized by processing the network I / F 116 and the CPU 111, receives power supply from the communication terminal 3, and performs data communication with the communication terminal 3.

The storage / reading unit 19 is mainly realized by the processing of the CPU 111 shown in FIG. 3, and stores various data in the storage unit 1000 and reads various data from the storage unit 1000. In the following, even when the image pickup apparatus 1 reads and writes from the storage unit 1000, the description "via the storage / reading unit 19" may be omitted.

<< Functional configuration of communication terminal 3 >>
The communication terminal 3 has a transmission / reception unit 31, a reception unit 32, a connection unit 33, and a storage / reading unit 39. Each of these parts has a function or means realized by operating any of the components shown in FIG. 4 by an instruction from the CPU 301 according to a program for a communication terminal developed on the RAM 304 from the ROM 302. Is.

Further, the communication terminal 3 has a storage unit 3000 constructed by the ROM 302 and the RAM 304 shown in FIG. A program for a communication terminal is stored in the storage unit 3000.

(Each function configuration of communication terminal 3)
The transmission / reception unit 31 of the communication terminal 3 is mainly realized by the processing of the communication unit 305 and the CPU 301 shown in FIG. 4, and transmits / receives various data to / from the image management device 5 via the wireless router 9a and the communication network 9. I do. In the following, even when the communication terminal 3 communicates with the image management device 5, the description "via the transmission / reception unit 31" may be omitted.

The connection unit 33 is mainly realized by the processing of the USB I / F 303 and the CPU 301 shown in FIG. 4, and supplies power to the image pickup apparatus 1 and performs data communication.

The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 4, and stores various data in the storage unit 3000 and reads various data from the storage unit 3000. In the following, even when the communication terminal 3 reads / writes from the storage unit 3000, the description "via the storage / reading unit 39" may be omitted.

<< Functional configuration of image management device >>
The image management device 5 includes a transmission / reception unit 51, a person detection unit 52, a thumbnail creation unit 53, a screen creation unit 54, an image analysis unit 55, an evaluation unit 56, and a storage / reading unit 59. Each of these parts has a function realized by operating any of the components shown in FIG. 5 by a command from the CPU 501 according to a program for an image management device developed on the RAM 503 from the HD 504. It is a means.

Further, the image management device 5 has a RAM 503 shown in FIG. 4 and a storage unit 5000 constructed by the HD 504. A base management DB 5001, an imaging management DB 5002, an image management DB 5003, a thumbnail management DB 5004, an analysis information management DB 5005, and an interest level management DB 5006 are constructed in the storage unit 5000. Each database will be described below.

表１は、拠点管理ＤＢ５００１に記憶される各情報をテーブル状に示す拠点管理テーブルを示す。拠点管理テーブルでは、地域ＩＤ、地域名、拠点ＩＤ、拠点名、拠点レイアウトマップ、及び、装置ＩＤの各フィールドが関連付けて記憶されている。また、拠点管理テーブルの１つの行をレコードという場合がある。以下の各テーブルでも同様である。このうち、地域ＩＤは、地域を識別するための識別情報である。地域ＩＤの一例としては重複しない番号とアルファベットの組み合わせが挙げられる。

Table 1 shows a base management table showing each information stored in the base management DB 5001 in a table shape. In the base management table, each field of the area ID, the area name, the base ID, the base name, the base layout map, and the device ID is stored in association with each other. In addition, one row of the base management table may be called a record. The same applies to each of the following tables. Of these, the area ID is identification information for identifying the area. An example of a region ID is a combination of a unique number and an alphabet.

The area name indicates an area or range of land such as Kanto, Tokyo, Shibuya Ward, New York State, New York City, etc. It may be called a region name. The identification information refers to a name, a code, a character string, a numerical value, or a combination of two or more of these, which is used to uniquely distinguish a specific object from a plurality of objects. The same applies to the following ID or identification information.

The base ID is an example of identification information for identifying the base. The base ID is assigned to the base name so that it is not duplicated. It may be referred to as site-specific information. An example of a base ID is a combination of a unique number and an alphabet. The base is a place where the image pickup apparatus 1 is installed and serves as a base for photographing the surroundings. An example of a base is a store.

The base name is a store name such as the Shibuya store, a venue name such as the Shibuya venue, etc., and is the name of the base. File names such as the layout of each base and image data showing the map are registered in the base layout map. The location of the image pickup device 1 and the products handled at the base is specified by the base layout map in two-dimensional coordinates.

The terminal ID is identification information for identifying the image pickup apparatus 1. It may be referred to as terminal-specific information. The terminal ID is, for example, a serial number, a serial number, a numerical value that does not overlap with the model number, an IP address, a MAC address, or the like of the image pickup apparatus 1, but is not limited thereto. As shown in Table 1, one or more imaging devices (terminal IDs) are installed at one base, and their positions are registered in the base layout map.

The base management table may be registered by the installer X or the viewer Y, or may be registered by the supplier of the image processing system 200.

表２は、撮像管理ＤＢ５００２に記憶される各情報をテーブル状に示す撮像管理テーブルである。撮像管理テーブルでは、拠点ＩＤごとに、撮像タイトル、撮像開始日時、及び撮像終了日時の各フィールドが関連付けて記憶されている。撮像タイトルは、閲覧者Ｙが入力したイベントのタイトルである。つまり、閲覧者Ｙが消費者の行動を監視したい何らかのイベントが店舗で催される場合に、このイベントの名称が撮像タイトルとなる。当然ながら、イベントの名称は閲覧者Ｙが任意に付与できるためイベントの名称でなくてもよい。例えば、単に撮像年月日とすることもできる。閲覧者Ｙは、画像データの複数のファイルから所望の画像データを抽出する際に、撮像タイトルを参照することができる。なお、１回の撮像イベントで複数の画像データが時系列に（定期的に）撮像される。撮像開始日時は、閲覧者Ｙによって入力された日時であり、撮像装置１が撮像を開始する（又は開始した）日時を示す。撮像終了日時は、閲覧者Ｙによって入力された日時であり、撮像装置１が撮像を終了する（又は終了した）日時を示す。閲覧者Ｙは撮像開始日時と撮像終了日時を事前に登録しておくこともできる（予約撮像）。撮像管理テーブルは、主に画像管理装置５が登録する。

Table 2 is an imaging management table showing each information stored in the imaging management DB 5002 in a table shape. In the image pickup management table, each field of the image pickup title, the image pickup start date and time, and the image pickup end date and time is stored in association with each base ID. The captured title is the title of the event input by the viewer Y. That is, when some event that the viewer Y wants to monitor the consumer's behavior is held in the store, the name of this event becomes the imaging title. As a matter of course, the name of the event does not have to be the name of the event because the viewer Y can arbitrarily assign the name of the event. For example, it may be simply the imaging date. Viewer Y can refer to the captured title when extracting desired image data from a plurality of files of image data. It should be noted that a plurality of image data are imaged in time series (regularly) in one imaging event. The imaging start date and time is the date and time input by the viewer Y, and indicates the date and time when the imaging device 1 starts (or starts) imaging. The imaging end date and time is a date and time input by the viewer Y, and indicates a date and time when the imaging device 1 ends (or ends) imaging. Viewer Y can also register the imaging start date and time and the imaging end date and time in advance (reserved imaging). The image management table is mainly registered by the image management device 5.

表３は、画像管理ＤＢ５００３に記憶される各情報をテーブル状に示す画像管理テーブルである。画像管理テーブルでは、端末ＩＤごとに、画像ＩＤ、画像データのファイル名、及び撮像日時が関連付けて記憶されている。画像ＩＤは、画像データを一意に識別するための識別情報の一例である。画像固有情報と称してもよい。画像データのファイル名は、画像ＩＤで特定される画像データのファイル名である。撮像日時は画像データが端末ＩＤで示される撮像装置１で撮像された日時である。画像データも、記憶部５０００に記憶されている。

Table 3 is an image management table showing each information stored in the image management DB 5003 in a table shape. In the image management table, the image ID, the file name of the image data, and the imaging date and time are stored in association with each terminal ID. The image ID is an example of identification information for uniquely identifying the image data. It may be referred to as image-specific information. The file name of the image data is the file name of the image data specified by the image ID. The imaging date and time is the date and time when the image data was captured by the imaging device 1 indicated by the terminal ID. Image data is also stored in the storage unit 5000.

For example, the information terminal 7 accesses the image management device 5, and selects the base name and the image pickup title from the image pickup management table in Table 2. The image management device 5 can read the terminal ID associated with the base ID from the base management table in Table 1. Since the terminal ID is clarified, the image management device 5 can specify the image data whose imaging date and time is included in the imaging start date and time to the imaging end date and time among the image data associated with the terminal ID of the image management table.

Of course, the viewer Y can also directly specify the terminal ID and the base ID. In the present embodiment, for the sake of simplicity, a mode in which the viewer Y specifies a terminal ID for browsing will be mainly described. The image management table is mainly registered by the image management device 5.

表４は、サムネイル管理ＤＢ５００４に記憶される各情報をテーブル状に示すサムネイル管理テーブルである。サムネイルとは親指程度のという意味であり、サムネイル画像は縮小した、画素数を低減した又は一覧用のイメージデータという意味になる。

Table 4 is a thumbnail management table showing each information stored in the thumbnail management DB 5004 in a table shape. The thumbnail means about the size of a thumb, and the thumbnail image means reduced, reduced number of pixels, or image data for a list.

In the thumbnail management table, the thumbnail ID, the file name of the thumbnail image, and the predetermined area information are stored in association with each image ID. The thumbnail ID is an example of identification information for uniquely identifying a thumbnail image created based on the image data indicated by the image ID. It may be referred to as thumbnail-specific information. The file name of the thumbnail image is the file name of the thumbnail image indicated by the thumbnail ID. The image management device 5 assigns the file name of the thumbnail image. The predetermined area information indicates a predetermined area in which the thumbnail image is created in the image data indicated by the image ID. The thumbnail management table is mainly registered by the image management device 5.

表５は、解析情報管理ＤＢ５００５に記憶される各情報をテーブル状に示す解析情報テーブルである。解析情報テーブルでは、画像ＩＤごとに、領域ＩＤ、領域範囲、アクション及び方向が関連付けて記憶されている。画像ＩＤについては上記のとおりである。領域ＩＤ、範囲、アクション、及び、方向について詳細は後述される。まず、領域は、全天球画像の解析により来客者（人）が検出された来客者の外接矩形である。領域ＩＤは、領域を一意に識別するための識別情報の一例である。領域番号や領域固有情報と称してもよい。例えば、画像ごとに１から始まる連番がarea_ｎの"ｎ"に設定される。

Table 5 is an analysis information table showing each information stored in the analysis information management DB 5005 in a table shape. In the analysis information table, the area ID, the area range, the action, and the direction are stored in association with each image ID. The image ID is as described above. Details of the area ID, range, action, and direction will be described later. First, the region is a circumscribed rectangle of a visitor in which a visitor (person) is detected by analysis of a spherical image. The area ID is an example of identification information for uniquely identifying the area. It may be referred to as an area number or area-specific information. For example, a serial number starting from 1 is set to "n" in _{area n for each image.}

The area range is information for specifying the area. Assuming that the area where the visitor is detected is rectangular, for example, the coordinates (x, y), width (width), and height (height) of the upper left vertex are the area ranges. The area may be specified by the coordinates of two diagonal points. The area range is determined based on the coordinate system of the plane image in the state of the plane before the image is pasted on the whole celestial sphere. Supplementally, the image pickup apparatus 1 initially outputs a plane image, but at the time of viewing, the plane image is attached to the whole celestial sphere to create the omnidirectional image. The action is the action content of the visitor detected in the area, and is replaced with the number shown in Table 8. The direction indicates the direction of the action (direction of action).

表６は、興味度管理ＤＢ５００６に記憶される各情報をテーブル状に示す興味度管理テーブルである。興味度管理テーブルでは、来客者のアクションごとにアクションがどの程度の興味の強さを意味するかが登録されている。この興味の強さが興味度である。したがって、画像管理装置５はアクションから興味度を決定できる。なお、来客者のアクションとは商品に対し来客者が示した行動、動作又は反応等を示す。アクションには単に通り過ぎたことが含まれる。

Table 6 is an interest level management table showing each information stored in the interest level management DB 5006 in a table shape. In the interest level management table, the degree of interest that the action means for each action of the visitor is registered. The strength of this interest is the degree of interest. Therefore, the image management device 5 can determine the degree of interest from the action. The visitor's action refers to the action, action, reaction, or the like shown by the visitor to the product. Actions simply include passing by.

(Each function configuration of the image management device)
The transmission / reception unit 51 of the image management device 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 5, and the communication terminal 3 or the information terminal 7 and various data are connected to the communication terminal 3 or the information terminal 7 via the communication network 9. Send and receive. In the following, even when the image management device 5 communicates with the information terminal 7, the description "via the transmission / reception unit 51" may be omitted.

The person detection unit 52 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and detects a person from the image data. This person is detected by setting an appropriate feature amount and setting a learning identification device. For example, as feature quantities, Haar-like features, LBP (Local Binary Patterns) features, HOG (Histogram of Oriented Gradients) features, and the like are known. Further, as a learning identification device, SVM (Support Vector Machines), cascade classification using AdaBoost, and the like are known. However, it is not limited to these, and it is sufficient if a person can be detected.

The thumbnail creation unit 53 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and creates a thumbnail image of a predetermined area image.

When transmitting the image data to the information terminal 7, the screen creation unit 54 creates screen information for the information terminal 7 to display the image data using HTML data, JavaScript (registered trademark), CSS, or the like.

The image analysis unit 55 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and performs image processing for determining the action of the visitor with respect to the area where the visitor is detected. Since it is difficult to judge the movement itself from the image, the action is judged by matching with the posture shape. That is, the operator of the image processing system or the like registers a representative posture, and the image analysis unit 55 matches the shape of the detected person. Typical actions (above, walking in the direction of travel, walking without facing the direction of travel, not walking, reaching for the product, hitting the product on the body, etc.) are used as SVM, for example. Let them learn. Since there is a correlation between the visitor's movement and posture, the posture is learned individually. Preferably, the SVM learns the posture separately in the moving state and the stationary state. As a result, the number of postures learned by one SVM is reduced, so that the posture recognition accuracy of the SVM is improved. If the spherical image is a moving image, it may be determined whether the celestial sphere image is in a moving state or a stationary state by whether or not the visitor moves, or it may be determined by changing the image, or the leg portion. The size of the opening may be determined by pattern matching or the like. When judging the posture, the direction of the action is also judged. Please refer to Non-Patent Document 1 for the determination of posture.

The evaluation unit 56 mainly converts the determined action realized by the processing of the CPU 501 shown in FIG. 5 into the interest level based on the interest level management DB 5006. Further, the evaluation unit 56 specifies a product zone to be described later from the direction of the action for a plurality of time-series image data, and counts the number of times the same action (interest level) is given for each product zone.

The storage / reading unit 59 is mainly realized by the processing of the HDD 505 and the CPU 501 shown in FIG. 5, and stores various data in the storage unit 5000 and reads various data from the storage unit 5000. In the following, even when the image management device 5 reads / writes from the storage unit 5000, the description “via the storage / reading unit 59” may be omitted.

<Functional configuration of information terminal 7>
The information terminal 7 has a transmission / reception unit 71, a reception unit 72, a display control unit 73, and a storage / reading unit 79. Each of these parts has a function realized by operating any of the components shown in FIG. 5 by an instruction from the CPU 501 according to a program for the information terminal 7 developed on the RAM 503 from the HD 504. It is a means.

Further, the information terminal 7 has a RAM 503 shown in FIG. 5 and a storage unit 7000 constructed by the HD 504. The storage unit 7000 stores a program for the information terminal 7. The program for the information terminal 7 is, for example, browser software, but application software having a communication function such as browser software may also be used.

(Each function configuration of information terminal 7)
The transmission / reception unit 71 of the information terminal 7 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 5, and transmits / receives various data to / from the image management device 5 via the communication network 9. In the following, even when the information terminal 7 communicates with the image management device 5, the description "via the transmission / reception unit 71" may be omitted.

The reception unit 72 is realized mainly by the processing of the keyboard 511 and the mouse 512 shown in FIG. 5 and the CPU 501, and receives an operation input from the user (viewer Y in FIG. 2).

The display control unit 73 is mainly realized by the processing of the CPU 501 shown in FIG. 5, and interprets the screen information transmitted from the image management device 5 to display various screens on the display 508 of the information terminal 7. Is controlled.

The storage / reading unit 79 is mainly realized by the processing of the HD 504 and the CPU 501 shown in FIG. 5, and stores various data in the storage unit 7000 and reads various data from the storage unit 7000. In the following, even when the information terminal 7 reads / writes from the storage unit 7000, the description "via the storage / reading unit 79" may be omitted.

<Example of human detection and action judgment>
In order for the image management device 5 to determine the degree of interest of the visitor in the product, the image management device 5 needs to detect a person. The person detection unit 52 detects a person by image processing, registers the detected area as coordinates in the image in the analysis information management DB 5005, and the image analysis unit 55 determines an action from the area.

FIG. 7 is an example of a diagram schematically showing a region 602 in which a person is detected. In FIG. 7A, five regions 602 are detected. The planar image of FIG. 7A is, for example, an image created by the Mercator projection before being attached to the whole celestial sphere. The ceiling may be distorted, but there are few inconveniences because there are no people on the ceiling. In addition, although there is a possibility that the portion of the spherical image near the feet is also distorted, it is not necessary to detect a person in a predetermined range near the floor (below the threshold latitude). Alternatively, a person may be detected or an action may be determined in a state of being converted into a spherical image.

The image analysis unit 55 determines the action to be taken by the visitor with respect to the area 602 in which the person is detected. As a result, as shown in FIG. 7B, each region 602 has the determined action 603 and the direction of the action.

For example, when walking, it can be determined by pattern matching whether the face is facing forward or sideways. In case of action 0, there is no direction. When the face orientation is the orientation of the product, it is determined to be action 1, but in this case, the face orientation is the action orientation. Further, when it is determined that the user is standing still and it is determined that the hand is reached by pattern matching, it is determined that the action 3 is performed. In this case, the direction is the direction of the hand extended by the visitor.

When it is determined by pattern matching that the product is applied to the body, it is determined to be action 4. In this case, the direction of the shelf on which the visitor picked up the product is determined from the past image data. When it is determined by pattern matching that the product is applied to the body, it is determined to be action 5. In this case, the direction of the shelf on which the visitor picked up the product is determined from the past image data.

The content of the action is not limited to 6 from 0 to 5, and more types of actions may be determined. Further, the direction takes a value of 360 degrees clockwise or counterclockwise with a predetermined direction (for example, x direction, y direction) of the plane image as 0 degrees. The image analysis unit 55 registers the rectangular data and the action determination result in the analysis information management DB 5005.

FIG. 7C shows an example of an image captured by the imaging device 1. Two visitors have been detected as humans and their respective areas 602 are illustrated. Such an image can be acquired by the information terminal 7 from the image management device 5.

<Action judgment procedure>
FIG. 8 is an example of a flowchart showing a procedure in which the image analysis unit 55 determines an action. The process of FIG. 8 starts when the process of detecting a person in the image data is performed. Alternatively, the viewer Y requests the image management device 5 to perform processing.

The image analysis unit 55 refers to the image management DB 5003 and acquires, for example, a series of image data (S10). The series of image data is, for example, image data taken by the same image pickup apparatus 1 for a certain period of time.

Next, the image analysis unit 55 reads out the image data one by one in order (S20).

Then, the image analysis unit 55 determines whether or not there is a region 602 in which a person is detected in the image data (S30).

If the determination in step S30 is No, the process proceeds to step S170. If the determination in step S30 is Yes, the image analysis unit 55 estimates the posture (S40). The face orientation is also estimated by estimating the posture.

First, it is determined whether or not the person is walking based on the posture (S50). If the determination in step S40 is Yes, the image analysis unit 55 determines whether or not the face is facing the traveling direction (S60).

If the determination in step S60 is Yes, the image analysis unit 55 determines that the visitor has taken the action of passing by without looking at the product (S70).

If the determination in step S60 is No, the image analysis unit 55 determines that the visitor has performed the action of looking at the product while walking (S80).

Next, if the determination in step S50 is No, the visitor is stopped, and the image analysis unit 55 determines whether or not the user is reaching out (S90).

If the determination in step S90 is Yes, the image analysis unit 55 determines that the visitor is reaching for the product (S100).

If the determination in step S90 is No, the image analysis unit 55 determines whether or not the visitor has the product (S110).

If the determination in step S110 is No, the image analysis unit 55 determines that the visitor has taken an action of stopping and looking at the product (S120).

If the determination in step S110 is Yes, the image analysis unit 55 determines whether the visitor has put (or brought) the product in the basket, put it on the body, or returned it to the shelf (S130).

As a result, the image analysis unit 55 determines whether the visitor has taken the action of putting the product in the basket (S140) or the visitor has taken the action of hitting the product on the body (S150). Alternatively, it is determined that the visitor has returned the product to the shelf (S160).

Next, the process returns to step S30, and if No is determined in step S30, the image analysis unit 55 determines whether or not there is the next image data (S170). If there is image data, the process returns to step S20, and if there is no next image data, the process of FIG. 8 ends.

In this way, various actions can be judged from the posture. If various actions can be judged, the degree of interest can be judged in multiple stages.

<Aggregation by product zone>
Since the action and direction of the area 602 in which the person is detected are known, it is possible to create an "action distribution for each product zone" by using these. A product zone is a place in a store where products of the same type and related products are gathered. Specifically, it is a place where one or more shelves are set or products are closely arranged.

FIG. 9 is an example of a diagram schematically explaining the aggregation for each product zone 610. The shape of the product zone 610 is irregular, but it may be polygonal. Viewer Y defines in advance the product zone 610 that he / she wants to aggregate. In FIG. 9, two product zones 610A and B are illustrated. The area 602 in which a person is detected is adjacent to or overlaps the product zone 610. It is considered that visitors often take some action on the product zone 610. Therefore, the action of the area 602 facing the product zone 610 is considered to have been performed for the product zone 610, and the evaluation unit 56 totals the actions.

Regions 602A to 602C in FIG. 9 are adjacent to the product zone 610A, respectively. Adjacent means that the center of the product zone 610 and the center of the area 602 are within a predetermined distance. This distance may be the number of pixels on the image. Further, the actions performed in the areas 602A to 602C face the product zone 610A. Facing the product zone 610 means that the product zone 610 is on the extension line when the direction of the action is extended. For example, the outer edge of the product zone is represented as a set of several points, and when the extension line in the direction of the action passes within a predetermined distance from this point, it can be determined that the product zone 610 is on the extension line. Alternatively, for simplification, when the circumscribed rectangle of the product zone and the extension line intersect, it may be determined that the product zone 610 is on the extension line.

Therefore, the actions in the areas 602A to 602C are counted by the evaluation unit 56 as being performed on the product zone 610A. On the other hand, the region 602D is adjacent to (or has an overlapping region) not only the product zone 610A but also the product zone 610B. However, since the direction of the action faces the product zone 610B (when extended in the direction of the action, the product zone 610B is reached), the actions are totaled as the action to the product zone 610B.

Further, although the area 602E overlaps with the product zone 610A, it is not counted as an action to the product zone 610A because the direction of the action does not face the product zone 610A.

The evaluation unit 56 performs these processes on all the areas 602, and associates the area 602 with the product zone 610. The product zones 610A and 610B are associated with the area 602 and its actions as follows.

Product Zone 610A: Action 4x1, Action 2x2
Product Zone 610B: Action 3x1

表７は、解析情報管理ＤＢ５００５に記憶される各情報をテーブル状に示す解析情報テーブルである。なお、表７の説明では、表５との相違を主に説明する。表７の解析情報テーブルには、対象ゾーンの座標のフィールドが追加されている。対象ゾーンの座標のフィールドは、商品ゾーン６１０を特定する情報が登録されている。したがって、各領域６０２のアクションが向いている方向の商品ゾーン６１０と領域６０２が対応付けられる。商品ゾーン６１０を特定する情報は、対象ゾーンの外縁の点を示す座標の集合、又は、外接矩形を生成した場合の左上頂点の座標（x,y）と幅（width）と高さ（height）などである。

Table 7 is an analysis information table showing each information stored in the analysis information management DB 5005 in a table shape. In the description of Table 7, the differences from Table 5 will be mainly described. A field of coordinates of the target zone is added to the analysis information table of Table 7. Information that identifies the product zone 610 is registered in the coordinate field of the target zone. Therefore, the product zone 610 and the area 602 in the direction in which the action of each area 602 is facing are associated with each other. The information that identifies the product zone 610 is a set of coordinates that indicate points on the outer edge of the target zone, or the coordinates (x, y), width (width), and height (height) of the upper left vertex when an circumscribing rectangle is generated. And so on.

更に、評価部５６は、商品ゾーン６１０ごとにアクションの数を集計することにより、表８に示すように、各商品ゾーン６１０に対する各アクションの分布テーブルが得られる。表８では、ある集計期間の間、各商品ゾーン６１０において、アクション０〜５が判断された回数が示されている。したがって、商品ゾーン６１０ごとに各アクションの頻度が分かり、店舗内で見ればアクションの分布が得られることになる。

Further, the evaluation unit 56 aggregates the number of actions for each product zone 610 to obtain a distribution table of each action for each product zone 610, as shown in Table 8. Table 8 shows the number of times actions 0 to 5 were determined in each product zone 610 during a certain aggregation period. Therefore, the frequency of each action can be known for each product zone 610, and the distribution of actions can be obtained when viewed in the store.

<Judgment of product zone>
FIG. 10 is an example of a flowchart showing a procedure for determining the product zone 610 with respect to the area 602. The process of FIG. 10 starts when the action determination is completed. Alternatively, the viewer Y requests the image management device 5 to perform processing.

The evaluation unit 56 refers to the image management DB 5003 and acquires, for example, a series of image data (S10). The series of image data is, for example, image data taken by the same image pickup apparatus 1 for a certain period of time.

Next, the evaluation unit 56 reads out the image data one by one in order (S20).

Then, the evaluation unit 56 determines whether or not there is a region 602 in which a person is detected in the image data (S30).

If the determination in step S30 is Yes, the evaluation unit 56 determines whether or not there is an adjacent or overlapping product zone 610 in the region 602 where the person is detected (S40). This determination may be made by determining whether or not the center of the product zone 610 and the center of the area 602 are within a predetermined distance.

If the determination in step S40 is No, the process returns to step S30. If the determination in step S40 is Yes, the evaluation unit 56 determines whether or not the action determined in the area 602 has a direction (S50). This can be done by determining the action number. Other than action 0, it has a direction.

If the determination in step S50 is No, the evaluation unit 56 determines that the action is for all adjacent product zones 610, and increases the number of actions by one (S60). As a result, the area 602 can be totaled even when the direction is unknown.

If the determination in step S50 is Yes, the evaluation unit 56 determines whether or not there is an adjacent product zone 610 on the extension line in the direction of the action (S70).

If the determination in step S70 is No, the process returns to step S30. If the determination in step S70 is Yes, the evaluation unit 56 increases the number of actions in the product zone 610 on the extension of the action direction by one (S80).

If the determination in step S30 is No, the evaluation unit 56 determines whether or not there is the next image (S90).

If the determination in step S90 is Yes, the process returns to step S20, and if the determination in step S90 is No, the process in FIG. 10 ends.

As described above, in the image processing system 200 of this embodiment, the degree of interest of the visitor in the product can be evaluated in multiple stages. In addition, since the product zone 610 can be determined based on the direction of the action and the number of interests (actions) performed on the product zone 610 can be counted, it is possible to quantify what kind of interest is present in each product zone 610. it can. Therefore, the viewer Y or the like can analyze the content of the action (interest) for the product zone 610.

In this embodiment, the image processing system 200 that determines the product zone 610 when the product zone 610 is not determined in advance will be described. It should be noted that the functions of the image processing system 200 are the same as those in the first embodiment, or even if there are differences, there is no problem in the explanation of the present embodiment.

In the first embodiment, the product zone 610 is defined in advance, but the image management device 5 can define the product zone 610 based on the area 602 in which a person is detected even if the product zone 610 is not defined in advance. This is because it may be estimated that the direction of the action is the product zone 610. Therefore, the image management device 5 can estimate or specify the product zone 610 from the direction of the action of the area 602 and the area 602.

FIG. 11 is a diagram schematically showing a method of defining the product zone 610. First, the areas 602A to 602F are detected by the person detection unit 52 detecting a person. The image analysis unit 55 determines the action of the areas 602A to 602F, and knows the direction. Since it can be considered that there is often a product zone 610 in the direction of the action, the evaluation unit 56 defines a product zone 610 having a size corresponding to the size of the area 602 in the direction of the action.

In FIG. 11, a square having a side having the same length as the short side (width) of each area 602 is illustrated as a product zone 610. The evaluation unit 56 determines the intersection of the straight line indicating the direction of the action and the vertical line or the horizontal line separated from the area 602 by half of the short side (width) at the center of the square. That is, when a straight line indicating the direction of action intersects the upper side of the area 602, an intersection with a horizontal line parallel to the upper side separated by half the short side (width) from the upper side of the area 602 is required. When a straight line indicating the direction of action intersects the lower side of the area 602, an intersection of a horizontal line parallel to the lower side separated by half the short side (width) from the lower side of the area 602 is required. When a straight line indicating the direction of action intersects the right side of the area 602, an intersection with a vertical line parallel to the right side separated by half the short side (width) from the right side of the area 602 is required. When a straight line indicating the direction of action intersects the left side of the area 602, an intersection with a vertical line parallel to the left side separated by half the short side (width) from the left side of the area 602 is required.

Half of the short side (width) from the center of this square to the top, bottom, left, and right is the square required by the rectangle surrounded by a straight line. Since the product zone 610 is a square having a side having the same length as the short side (width) of the region 602, the size of the region 602 can be reflected in the product zone 610. A square having the same length as the long side (height) of the area 602 or a rectangle having the same width and height as the area 602 may be defined as the product zone 610. Further, the product zone 610 may be a rectangle such as a square or a rectangle, or may be a circle or an ellipse. Also in this case, it is preferable that the size of the region 602 is reflected in the diameter of the circle and the long axis or the short axis of the ellipse.

As an exception, the action may have no direction, as in area 602F. In this case, since it is unknown in which direction the visitor in the area 602F is acting, the evaluation unit 56 does not define the product zone 610. In addition, there is a possibility that an action is detected in the image data before and after. However, the product zone 610 may be set in any direction or all directions around the area 602F, assuming that the action can be performed in all directions around the area.

In FIG. 11, the product zone 610A is defined for the area 602A, the product zone 610B is defined for the area 602B, the product zone 610C is defined for the area 602C, the product zone 610D is defined for the area 602D, and the product zone 610E is defined for the area 602E. .. No merchandise zone is defined for area 602F.

FIG. 12 is an example of a flowchart showing a procedure in which the evaluation unit 56 defines the product zone 610. The process of FIG. 12 starts when the action determination is completed. Alternatively, the viewer Y requests the image management device 5 to perform processing.

The evaluation unit 56 refers to the image management DB 5003 and acquires, for example, a series of image data (S10). The series of image data is, for example, image data taken by the same image pickup apparatus 1 for a certain period of time.

Next, the evaluation unit 56 reads out the image data one by one in order (S20).

Then, the evaluation unit 56 determines whether or not there is a region 602 in which a person is detected in the image data (S30).

If the determination in step S30 is Yes, the evaluation unit 56 determines whether or not the area 602 has the direction of action (S40).

If the determination in step S40 is No, the process returns to step S30. If the determination in step S40 is Yes, the evaluation unit 56 defines the product zone 610, which is the direction of action and is adjacent to the area 602 (S50).

After this, the process returns to step S30, and steps S30 to S50 are executed while there is an area 602.

If the determination in step S30 is No, the evaluation unit 56 determines whether or not there is the next image data (S60), and when the next image data is exhausted, the process of FIG. 12 ends.

By the above processing, the product zone 610 calculated from the area 602 and the direction of the action is defined, and the analysis information management DB 5005 in which the product zone 610 is associated with the area 602 is obtained.

表９は、本実施例において解析情報管理ＤＢ５００５に記憶される解析情報テーブルを模式的に示す。表９の説明では、表５との相違を主に説明する。表９の解析情報テーブルには、商品ゾーン６１０の座標のフィールドが追加されている。商品ゾーン６１０の座標は、領域６０２とアクションの方向に基づいて定義された商品ゾーン６１０の左上頂点の座標である。商品ゾーン６１０である正方形の中心の座標でもよい。幅と高さは互いに同じであり、領域６０２の幅と同じである。

Table 9 schematically shows the analysis information table stored in the analysis information management DB 5005 in this embodiment. In the explanation of Table 9, the difference from Table 5 will be mainly described. A field of coordinates of the product zone 610 is added to the analysis information table of Table 9. The coordinates of the product zone 610 are the coordinates of the upper left vertex of the product zone 610 defined based on the area 602 and the direction of the action. It may be the coordinates of the center of the square which is the product zone 610. The width and height are the same as each other and are the same as the width of the area 602.

As described above, the image processing system of the present embodiment can define the product zone 610 based on the area 602 even if the product zone 610 is not defined. If the product zone 610 is defined, the viewer Y or the like can analyze the content of the action (interest) for the product zone 610.

In this embodiment, the information terminal 7 that displays the contents of the analysis information table obtained in Examples 1 and 2 will be described. This makes it possible to display the degree of interest of visitors in a distribution expression such as a heat map. Viewer Y can visually grasp the degree of interest in the product.

FIG. 13 is a functional block diagram of the image pickup device 1, the communication terminal 3, the image management device 5, and the information terminal 7, which form a part of the image processing system 200 of this embodiment. Further, in the present embodiment, since the components having the same reference numerals in FIG. 6 perform the same function, only the main components of the present embodiment may be mainly described. The information terminal 7 of FIG. 13 has an information visualization unit 74. The information visualization unit 74 is realized by the processing of the CPU 501 shown in FIG. 5, and performs various evaluations for visualizing the analysis information table and the distribution table.

Since the visualization is slightly different between Example 1 and Example 2, the case of Example 1 and the case of Example 2 will be described separately.

<Example 1 When the product zone is defined in advance>
In Example 1, the distribution table shown in Table 8 is obtained. The information visualization unit 74 processes the number of actions of the distribution table according to the operation of the viewer Y, and the display control unit 73 displays it on the display 508. Hereinafter, a display example for the three operations of the viewer will be described.

1. 1. Operation 1
The viewer Y specifies a specific action (interest level), and the information terminal 7 visually displays the number of times the interest level is counted. FIG. 14 is an example of a diagram showing the degree of interest designated by the viewer Y, and FIG. 15 is an example of a diagram showing an example of a screen displayed by the information terminal 7 for the operation 1.

1.1 First, the viewer Y specifies the degree of interest that he / she wants to pay attention to. As shown in FIG. 14A, the information terminal 7 acquires image data, and the viewer Y presses a predetermined menu with a pointing device such as a mouse or a touch panel. The reception unit 72 receives this operation, and the display control unit 73 displays the pull-down menu shown in FIG. 14 (b). Viewer Y selects the degree of interest on the pointing device. The reception unit 72 receives this operation, and the display control unit 73 displays the degree of interest (degree of interest 2 in the figure) selected by the viewer Y on the menu as shown in FIG. 14 (c).

1.2 The information visualization unit 74 acquires the number of times of interest received by the reception unit 72 from one or more product zones 610 associated with the image data displayed in the distribution table. For example, assuming that the degree of interest is 2, taking Table 8 as an example, the number of times of 13, 17 ... N2 m is acquired. The information visualization unit 74 converts the magnitude of the number of times into a color. There are two main methods for converting to color. One is a method of assigning a predetermined color to the number of times included in the two thresholds by dividing the color by several thresholds. The other is a method of calculating a percentage by dividing the total number of times of all product zones 610 by the number of times of each product zone 610, and similarly dividing by a threshold value and assigning a color to the number of times. In the former method, the magnitude of the absolute number of times can be displayed in color, and in the latter method, it is possible to make a relative comparison of each product zone 610 with respect to a certain degree of interest in a certain image data.

In FIG. 15, two product zones 610 are shown superimposed on the image data, and each is displayed in a different color. Although it is black and white in the figure, it is actually displayed in color. Further, a color guide 620 is displayed at the right end of the image data. The color guide 620 indicates how many times a color represents. Therefore, the viewer Y can compare the color of the product zone 610 with the color guide 620 and visually grasp how many times the action of the degree of interest specified in each product zone 610 is counted.

Further, as shown in FIG. 15, the number of times may be superimposed and displayed on the product zone 610. Viewer Y can grasp the number of times itself.

Since the origin, width, and height of the product zone 610 are registered in the distribution table, the display control unit 73 can draw a rectangle indicating the product zone 610 in the image data. The area 602 may be made semi-transparent and drawn. As a result, the viewer Y can grasp what kind of products are in the colored product zone 610.

2. Operation 2
The viewer Y designates the product zone 610, and the information terminal 7 visually displays the number of times each degree of interest is counted in the product zone 610. FIG. 16 is an example of a diagram showing an example of a screen displayed by the information terminal 7 for the operation 2.

2.1 The viewer Y displays the image data obtained by capturing the image of the store on the display 508. In addition, the viewer Y specifies a place (product zone) in which he / she wants to know the degree of interest of the visitor with the pointing device 640. The reception unit 72 receives the coordinates of the position specified by the viewer Y.

2.2 The information visualization unit 74 searches the distribution table for the product zone 610 including or closest to the received coordinates.

2.3 The information visualization unit 74 acquires a set of the degree of interest and the number of times corresponding to the specified product zone 610 from the distribution table. In the example of Table 8, when the viewer Y acquires the product zone 610 specified by x1 and y1, the degree of interest is 0:52, the degree of interest is 1:23, the degree of interest is 2:13, the degree of interest is 3:11, and the degree of interest is A set of interest level and number of times of 4: 7 and interest level 5: 4 is acquired.

2.4 The information visualization unit 74 visually displays the correspondence between the degree of interest and the number of times (for example, pop-up display). In FIG. 16B, the number of times of each degree of interest is displayed in a bar graph. Colors are associated with interest levels 0 to 5 in advance. The difference in the number of times is reflected in the height in the bar graph of each degree of interest. For example, the height of interest, which is the most frequent, becomes a certain height, and the height of graphs of other interests is determined by proportional calculation.

Further, for example, the percentage may be calculated by summing the number of times of each degree of interest and dividing each number of times, and the relative difference may be displayed in a graph. In FIG. 16B, the number of times of each degree of interest is displayed in a pie chart. This makes it easier for the viewer Y to grasp the relative difference.

Therefore, according to the operation 2, the viewer Y can visually grasp the degree of interest in the designated product zone and the number of visitors showing the degree of interest.

3. 3. Operation 3
The difference in the degree of interest of all the product zones 610 of the image data is displayed in a list. The information terminal 7 calculates the representative values of the degree of interest and the number of times for each product zone 610 and visually displays them. FIG. 17 is an example of a diagram showing an example of a screen displayed by the information terminal 7 for the operation 3.

3.1 Viewer Y performs an operation of displaying a list of differences in the degree of interest of all product zones 610 in the image data. The reception unit 72 accepts this operation, and the information visualization unit 74 calculates a representative value that reflects the number of people with multiple levels of interest indicated by the visitor for each product zone 610 by a predetermined calculation formula.

As an example of the calculation formula, there is a formula in which weights are given for each degree of interest, multiplied by the number of times, and added. For example, if the degree of interest is 0 to 5, the weight of the degree of interest 0 is 0, the weight of the degree of interest 1 is 1, the weight of the degree of interest 2 is 2, the weight of the degree of interest 3 is 3, and the weight of the degree of interest 4 is 4. Is 4, and the weight of interest level 5 is 5. The higher the interest level, the larger the value of interest levels 0 to 5. In the example of Table 8, for the product zone 610 specified by x1 and y1, the degree of interest is 0 × 52, the degree of interest 1 × 23, the degree of interest 2 × 13, the degree of interest 3 × 11, the degree of interest 4 × 7, and the degree of interest. 5 × 4, is calculated. The sum of the products of these interests and the number of times is a representative value.

In addition, the product of the most frequent number of times of interest and the number of times (mode), the product of the degree of interest with the median number of times and the number of times (intermediate value), and the average value obtained by dividing the total number of times by 6. , The product of the highest number of interests and the number of times (maximum value) may be used. Alternatively, the degree of interest may be extracted based on the number of times without obtaining the product with the number of times.

Further, the information terminal 7 may display the method of obtaining these representative values on the display 508 and allow the viewer Y to select them.

3.2 After the representative value is calculated, the information visualization unit 74 makes the product zone 610 correspond to the color according to the representative value. The method of associating the colors may be the same as in step 1.

3.3 As shown in FIG. 17, the five product zones 610 of the image data are displayed in colors corresponding to the representative values.

Therefore, according to the operation 3, the viewer Y can visually grasp the difference in the degree of interest for all the product zones.

<Example 2 When the product zone is not defined in advance>
In the second embodiment, the analysis information table shown in Table 9 is obtained. The information visualization unit 74 uses the coordinates of the product zone 610 of the analysis information table according to the operation of the viewer Y, and the display control unit 73 displays the product zone 610 on the display 508. Hereinafter, a display example for the three operations of the viewer will be described.

4. Operation 4
The viewer Y specifies a specific action (interest level), and the information terminal 7 visually displays the product zone 610 corresponding to the interest level. FIG. 18 is an example of a diagram showing an example of a screen displayed by the information terminal 7 for the operation 4.

4.1 Viewer Y selects the degree of interest. The selection method may be the same as the method described in operation 1.

4.2 The information visualization unit 74 reads out the coordinates of the product zone 610 associated with the selected degree of interest from the analysis information table. The information visualization unit 74 associates the product zone 610 with a color determined based on the degree of interest. As shown in FIG. 18, the product zone 610 having the selected degree of interest is displayed in duplicate in the image data. When a plurality of product zones 610 overlap, the range including the overlapping product zones 610 is regarded as one continuous product zone 610.

Therefore, according to the operation 4, the viewer Y can visually grasp the product zone in which the designated degree of interest has occurred.

5. Operation 5
The viewer Y designates the product zone 610, and the information terminal 7 visually displays the degree of interest in the product zone 610. FIG. 19 is an example of a diagram showing an example of a screen displayed by the information terminal 7 for the operation 5.

5.1 As shown in FIG. 19A, the user specifies a part of the image data in which he / she wants to know the degree of interest of the visitor with a pointing device or the like. The reception unit 72 acquires the coordinates of the position input by the pointing device 640.

5.2 The viewer searches the analysis information table for the product zone 610 closest to the acquired coordinates. When there are two or more corresponding product zones 610, the information visualization unit 74 presents a plurality of product zones 610 to the viewer Y as shown in FIG. 19 (b), and any one of the product zones 610 is presented. Let me select. As shown in FIG. 19C, the reception unit 72 accepts the selection of one product zone 610. The display control unit 73 enlarges and displays (zooms in) the image data as needed.

5.3 The information visualization unit 74 acquires the degree of interest corresponding to the product zone 610 obtained in this way from the analysis information table, and assigns a color determined by the degree of interest.

5.4 As shown in FIG. 19D, the display control unit 73 displays the product zone 610 in the image data in the assigned color.

Therefore, according to the operation 5, the viewer Y can visually grasp the degree of interest in the designated product zone.

6. Operation 6
The difference in the degree of interest of all the product zones 610 of the image data is displayed in a list. The information terminal 7 indicates the degree of interest in each product zone 610 in color. The area 602 on which the product zones 610 overlap is converted into a color after adding the degree of interest. FIG. 20 is an example of a diagram showing an example of a screen displayed by the information terminal 7 for the operation 6.

6.1 Viewer Y performs an operation of displaying a list of differences in the degree of interest of all product zones 610 in the image data. The reception unit 72 accepts this operation.

6.2 The information visualization unit 74 acquires the degree of interest corresponding to each product zone 610 from the analysis information table. In the range where the product zones 610 overlap, the degree of interest is calculated according to the overlap. This calculation method will be described with reference to FIG.

6.3 The information visualization unit 74 applies the degree of interest to colors to generate a heat map. The method of creating the heat map will be described later in FIG.

6.4 As shown in FIG. 20, the display control unit 73 superimposes and displays a heat map on the image data. In FIG. 20, a product zone 610 in which the density of concentric colors gradually changes (gradation) is displayed. The darker the color, the more the product zones 610 overlap.

Therefore, according to the operation 6, the viewer Y can visually determine how much the visitor is interested in each product zone 610 based on the color depth.

<Creating a heat map>
A method of creating a heat map will be described with reference to FIG. FIG. 21 is an example of a diagram illustrating weighting with respect to the creation of a heat map.

The information visualization unit 74 sets a weighting value for rendering the gradation in the product zone 610. As shown in FIG. 21, the center point (cx1, cy1) of the rectangle which is the product zone 610 is calculated, and grid points are set at regular intervals in the vicinity of this point. In order to keep the number of grid points constant, the interval is set to a value according to the size of the product zone 610. As a result, it is possible to make a difference in the size of the gradation to be drawn according to the size of the product zone 610. Then, a weighting value that is a value according to the degree of interest and has the maximum value at the center is set for each grid point.

Further, when there is another product zone 610 that overlaps with one product zone 610, the average center point (cxn, cyn) of each center point is calculated. cxn = cx1 + cx2, cyn = cy1 + cy2. Similarly, grid points are set at regular intervals in the vicinity of the center. Then, the weighting value is calculated so that the center point of the average becomes a relatively high value. The weighting value at this time is larger than that in the case of one product zone 610. For example, the number of all product zones 610 that overlap with the product zone 610 of interest is counted, and the maximum value corresponding to the product of this number and the average degree of interest is set as the center point of the average. It should be noted that each product zone 610 is counted as a number that overlaps only once.

Therefore, the higher the degree of interest and the more overlapping product zones, the higher the weighting value of the center point. The grid points around the center point are weighted as they move away from the center point. As a result, the product zone 610 can be displayed in gradation, and the portion where the product zone 610 gathers can be drawn darkly.

Instead of setting grid points around the center point, a gentle color change may be given around the center point by a Gaussian function or the like.

Further, when there is another product zone 610 that overlaps with one product zone 610, the weighted values of the respective grid points may be summed as they are. Since it is rare that the grid points completely match, the weighted values of the grid points of the product zone 610 of interest and the nearest grid points are summed. In this case, a color having a high density is set according to the weighting value. As the number of product zones 610 overlaps, the weighting value of the grid points increases, so that the place where there are many product zones 610 can be displayed darkly.

It is assumed that the image processing system 200 described in Examples 1 to 3 is arranged in a retail store, for example. Although the image pickup device 1 periodically images the surroundings, the staff may notice some remarkable event in the store. For example, there are troubles between visitors, customers who behave peculiarly, outbreak of sick people, scattering of products, damage to facilities, and so on. These events are imaged by the image pickup apparatus 1, but such events should be grasped by the viewer Y as well. Therefore, when the staff encounters such an event, the viewer Y often wants to confirm "what happened now" with an image later.

In order for the viewer Y to check the image later, it is necessary to memorize or make a note of the time and place where the event was noticed. However, such work is troublesome for busy staff. Therefore, it is conceivable that the staff or the like records that an event has occurred in the image data of the image pickup apparatus 1 that continues to take an image. For example, if the fact that an event has occurred is recorded in the image data in a certain time range, it is possible to easily find the image when the event occurred later.

However, only the person who has the authority to operate the imaging device 1 (generally, the store manager, the manager of the head office, etc., in this embodiment, the viewer Y) has the authority to record the fact that the event has occurred. The general staff, who have the most chances of actually encountering an event in a store, do not have the authority to operate the image pickup device 1 in the first place. If the staff is given such authority, the staff or the viewer Y can record that an event has occurred in the image data when viewing the image, but record that the event has occurred in real time. Was difficult.

Therefore, in the present embodiment, when a remarkable event occurs during the imaging of the image pickup device 1, the staff notifies the image management device 5 so that the image management device 5 can add the event to the specific image data in which the event is captured. Record that the event has occurred. With such a configuration, the staff can record the fact that an event has occurred in the image data in real time. Therefore, the viewer Y can quickly access the image data in which the event is captured, such as by searching the record later.

<About system configuration>
FIG. 22 is an example of a schematic configuration diagram of the image processing system 200. In the description of FIG. 22, since the components having the same reference numerals in FIG. 2 perform the same function, only the main components of the present embodiment may be mainly described.

FIG. 22 shows a staff terminal 4 operated by a staff member. Staff Z includes store staff, employees, members, employees, part-time workers, part-time workers, and the like. The staff Z may include the viewer Y and the installer X. The staff terminal 4 is a highly portable information processing device, and is, for example, a smartphone, a tablet terminal, a notebook PC, a wearable PC, a game machine, a mobile phone, a PDA, and the like, but is not limited thereto. Further, the staff Z may use a stationary information processing device, which does not need to be highly portable.

On the staff terminal 4, the program for the staff terminal 4 is running. The program for the staff terminal 4 is, for example, browser software or application software dedicated to the image processing system 200. The dedicated application software holds the IP address and the like of the image management device 5 and communicates with the image management device 5. Further, in the case of browser software, the staff terminal 4 communicates with the image management device 5 by the operation of the staff Z.

The staff terminal 4 accepts the user's operation and transmits the event recording request to the image management device 5. The event record request is a request for recording that an event has occurred. Details will be described later.

FIG. 23 shows an example of the hardware configuration diagram of the staff terminal 4. The staff terminal 4 has a CPU 401 that controls the overall operation of the staff terminal 4, a ROM 402 that stores programs such as an IPL (Initial Program Loader), and a RAM 403 that is used as a work area of the CPU 401. Further, it has a flash memory 404 that stores various data such as a program 430 for the staff terminal 4 and image data, and an SSD (Solid State Drive) 405 that controls reading or writing of various data to the flash memory 404 according to the control of the CPU 401. .. Further, a media drive 407 that controls reading or writing (storage) of data to a recording medium 406 such as a flash memory, an operation button 408 that accepts various operations on the staff terminal 4, and a power ON / OFF switch for the staff terminal 4. It has a power switch 409 and a network I / F (Interface) 417 for transmitting data wirelessly or by wire using a communication network.

Further, the staff terminal 4 has a built-in camera 412 that captures a subject and acquires image data under the control of the CPU 401, an image sensor I / F 413 that controls the drive of the camera 412, and a built-in microphone that inputs audio. It has a built-in speaker 415 that outputs 414 and audio. Further, various types of audio input / output I / F 416 for processing audio signal input / output between the microphone 414 and the speaker 415 under the control of the CPU 401, and a display I / F 417 for transmitting image data to the display 450 under the control of the CPU 401. It has an external device connection I / F 418 for connecting the external device of the above. In addition, a GPS receiver 419 that receives radio waves from GPS satellites to detect the position, an acceleration sensor 420 that detects the acceleration generated in the staff terminal 4, and an LTE communication unit that performs voice communication and data communication via a mobile phone network. It has a 421 and a bus line 422 such as an address bus and a data bus for electrically connecting each of the above components as shown in FIG.

The display 450 is composed of a liquid crystal or an organic EL, and is a display area for displaying a menu operated by a user, an captured image, a message, and the like. The display I / F417 corresponds to the touch panel function of the display 450.

The camera 412 includes a lens and a solid-state image sensor that converts light into electric charges to digitize an image (video) of the subject. As the solid-state image sensor, CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) Etc. are used.

Various external devices can be attached to the external device connection I / F418 by a USB (Universal Serial Bus) cable or the like. For example, a short-range wireless communication device such as Bluetooth (registered trademark) can be connected.

Further, the flash memory 404 stores a program 430 for the staff terminal 4. The program 430 for the staff terminal 4 is downloaded from the server for program distribution by the network I / F411.

The recording medium 406 has a structure that can be attached to and detached from the staff terminal 4. Further, the above program 430 may be distributed in a state of being recorded on the recording medium 406.

<About functions>
FIG. 24 is a functional block diagram of each of the image management device 5, the information terminal 7, and the staff terminal 4, which constitutes a part of the image processing system 200 of this embodiment. The image pickup device 1 and the communication terminal 3 are omitted. In the present embodiment, the components having the same reference numerals in FIG. 13 perform the same functions, and therefore, only the main components of the present embodiment may be mainly described.

The image management device 5 of FIG. 24 has an event recording unit 57. The event recording unit 57 is realized by the processing of the CPU 501 shown in FIG. 5, and when the event recording request from the staff terminal 4 is received, the event recording unit 57 sets an event flag in the image management table.

表１０は、本実施例の画像管理ＤＢ５００３に記憶される画像管理テーブルを示す。表１０の画像管理テーブルでは、新たに事象フラグのフィールドを有している。事象フラグにはＯＮ又はＯＦＦ（ＯＦＦの場合は何も記録されなくてよい）が設定される。ＯＮの場合には、更にスタッフＩＤとカテゴリＩＤが設定される。スタッフＩＤはスタッフＺを特定する機能を有する情報であり、スタッフ特定情報と称してもよい。また、カテゴリＩＤは後述する事象のカテゴリを特定する機能を有する情報であり、事象分類情報と称してもよい。

Table 10 shows an image management table stored in the image management DB 5003 of this embodiment. The image management table in Table 10 has a new event flag field. The event flag is set to ON or OFF (when it is OFF, nothing needs to be recorded). When it is ON, a staff ID and a category ID are further set. The staff ID is information having a function of identifying the staff Z, and may be referred to as staff identification information. Further, the category ID is information having a function of specifying the category of the event described later, and may be referred to as event classification information.

For example, since the event flag with the image ID P0001 is set to "ON: S001, C001", it is shown that the staff Z with the staff ID S001 has set the event flag with the category ID C001. .. It should be noted that a plurality of staff Zs can set event flags for one image data.

<< Functions of staff terminal 4 >>
As shown in the figure, the staff terminal 4 has a transmission / reception unit 41, a reception unit 42, a display control unit 43, an event record request unit 44, and a storage / reading unit 49. Each of these parts is realized by operating any of the components shown in FIG. 23 by an instruction from the CPU 401 according to a program for the staff terminal 4 expanded on the RAM 403 from the flash memory 404. A function or means.

Further, the staff terminal 4 has a storage unit 4000 constructed by one or more of the ROM 402, the RAM 403, and the flash memory 404 shown in FIG. 23. The program 430 for the staff terminal 4 is stored in the storage unit 4000. Further, the staff ID and the password for authentication may be stored.

The transmission / reception unit 41 of the staff terminal 4 is mainly realized by the processing of the network I / F 411 and the CPU 401 shown in FIG. 23, and transmits / receives various data to / from the image management device 5 via the communication network 9. In the following, even when the staff terminal 4 communicates with the image management device 5, the description "via the transmission / reception unit 41" may be omitted. Further, the staff terminal 4 may communicate using the LTE communication unit 421.

The reception unit 42 of the staff terminal 4 is realized mainly by the processing of the operation buttons 408 and the CPU 401 shown in FIG. 23, and receives the operation input from the staff Z.

The display control unit 43 is mainly realized by the processing of the display I / F 417 and the CPU 401 shown in FIG. 23, and creates a screen called a CUI (Character User Interface) or a GUI (Graphical User Interface) on the display 450. Display it.

The event recording request unit 44 is mainly realized by the processing of the CPU 401 shown in FIG. 23, and when the user performs a predetermined operation, the event recording request unit 44 transmits the event recording request to the server. The event recording request preferably includes the terminal ID of the imaging device 1, the event start information, the event end information, the staff ID, and the category ID, in addition to the request for recording that the event has occurred. The event start information is information about the time when the event flag setting is started, and the event end information is information about the time when the event flag setting is finished.

The storage / reading unit 49 is mainly realized by the processing of the CPU 401 shown in FIG. 23, stores various data in the storage unit 4000, and reads various data from the storage unit 4000.

<About event recording>
FIG. 25 is an example of a diagram illustrating a usage pattern of the image processing system 200 in a store. An imaging device 1 is installed in the store to regularly image the inside of the store (fixed point observation). In addition, staff Z carrying the staff terminal 4 is working in the store. As described above, the captured image data is registered in the image management table. In FIG. 25, items excluding the file names of the image data in Table 10 are shown. That is, FIG. 25 schematically shows how the three image data are captured, and the visitor G is captured in each of them. A terminal ID, an image ID, an imaging date and time, and an event flag are shown in one of the three image data 601. Viewer Y can view the image data during or after imaging, in the backyard of the store or outside the store.

Next, as shown in FIG. 26, it is assumed that a notable event has occurred in the store. FIG. 26 is a diagram showing a store in which the event that the visitor G collapses occurs. Staff Z encounters this event and determines that it is a noteworthy event.

Since the appearance of the visitor G falling down is imaged, the event itself is recorded in the image data. In FIG. 26, four image data are imaged, and the fallen visitor G is imaged in the fourth image data. When the viewer Y browses this event, it is preferable that the image data in which the event is captured can be easily found. Therefore, the staff Z gives a mark (event flag) indicating that the event is captured in the image data in which the event is captured in the store.

The operation of the staff terminal 4 for event recording by the staff will be described with reference to FIG. 27. By this operation, the event recording request is transmitted to the image management device 5. FIG. 27 shows an example of the event recording screen 680 displayed on the display 450 of the staff terminal 4. The event recording screen 680 is one of the screens generated by the program 430 for the staff terminal 4. The event recording screen 680 has a camera number 611, a category selection field 612, a start time field 613, an end time field 614, a start button 615, and an end button 616.

The camera number 611 is a number for identifying the image pickup apparatus 1. Staff Z designates camera number 611 of the imaging device 1 that is imaging the location where the event occurred. The camera number 611 may be the same as or different from the terminal ID. If they are different, the image management device 5 converts the camera number 611 into the terminal ID using the table in which the camera number 611 and the terminal ID are associated with each other.

The category selection field 612 is a field for the staff Z to select the event category. Examples of categories are troubles between visitors, visitors with unusual behavior, outbreaks of sick people, clutter of goods, damage to facilities, and so on.

The start time column 613 is a column for the staff Z to specify a time for tracing back the setting of the event flag based on the time when the staff Z transmits (or receives) the event recording request to the image management device 5. Since the time when Staff Z discovers the event is already after the event has occurred, the event has started before the event recording request. Therefore, the staff Z estimates how long ago the event occurred and sets it in the start time column 613. The time in the start time column corresponds to the event start information.

In the end time column 614, the staff Z sets the time for extending the setting of the event flag based on the time when the staff Z presses the end button 616 and transmits (or receives) the event recording end to the image management device 5. This is a field for specifying. It would be good if the staff Z could press the end button 616 when confirming the end of the event, but the staff Z may forget to press the end button 616 because of the work. Therefore, the staff Z estimates how long it takes to record the event after pressing the start button 615 and sets it in the end time column 614. As a result, the end button 616 can be pressed early, so that it is possible to suppress forgetting to press the end button 616. The time in the end time column corresponds to the event end information.

The start button 615 is a button for the staff Z to send an event recording request to the image management device 5. The end button 616 is a button for the staff Z to send the event recording end to the image management device 5. Even if the staff forgets to press the end button 616, the event recording unit 57 of the image management device 5 automatically considers that the end button 616 has been pressed after a predetermined time has elapsed from the event recording request.

As a modification, the staff terminal 4 may transmit both the start time column 613 and the end time column 614 to the image management device 5 by pressing the start button 615. As a result, the staff Z can transmit the event start information and the event end information with one operation. In this case, the end button 616 may not be provided, and the end button 616 may be provided as a button for immediately ending recording. Further, the staff Z is logged in to the image management device 5 when the program 430 for the staff terminal 4 is started. Therefore, the staff ID is included in the event record request and the event record end.

FIG. 28 is an example of a diagram schematically illustrating an event flag set by an event record request. The event recording unit 57 of the image management device 5 sets the event flag of the image management table to ON based on the event recording request, and ends the ON setting based on the event recording end. The event flag is OFF, which is the initial state (default value). It is assumed that the time "13:52:24" is specified by the event start information, and 6 seconds after that is specified by the event end information.

In this case, as shown in FIG. 28, the event recording unit 57 sets the event flag to ON for the image data of “13:52:24” to “13:52:29”. That is, the ON event flag is set for the image data highlighted by the thick frame in FIG. 28. Further, although not shown in FIG. 28, a staff ID and a category ID are set.

Therefore, the viewer Y can quickly browse the image data in which the event is captured by searching the image data in which the event flag is ON. In addition, the image data can be browsed retroactively from the time when the staff Z notices the event.

<About the operation of the image management device>
FIG. 29 is an example of a sequence diagram showing the overall operation procedure of the image processing system 200.
S1: The image pickup device 1 periodically images the surroundings according to an instruction from the image management device 5.
S2: The connection unit 15 of the image pickup device 1 transmits image data to the communication terminal 3.
S3: The transmission / reception unit 31 of the communication terminal 3 transmits the image data to the image management device 5.
S4: The storage / reading unit of the image management device 5 registers image data in the image management table. Steps S1 to S4 are repeated.
S5: Upon receiving the operation of the staff Z, the staff terminal 4 transmits the event recording request to the image management device 5.
S6: The event recording unit 57 of the image management device 5 sets the event flag of the image management table to ON. Details will be described with reference to FIG.
S7: The staff terminal 4 transmits the event recording end to the image management device 5. This ends the setting of ON for the event flag. Details will be described with reference to FIG.

FIG. 30 is an example of a flowchart showing a procedure in which the image management device 5 manages images. The process of FIG. 30 is repeatedly executed, for example, every time image data is captured.

The image management device 5 shall periodically cause the image pickup device 1 to take an image. Therefore, the image management device 5 determines whether or not the imaging time has come (S10). If the determination in step S10 is No, the process of FIG. 30 ends.

If the determination in step S10 is Yes, the image management device 5 requests the image pickup device 1 to capture an image (S20). The image data captured by the image capturing device 1 is transmitted to the image management device 5.

Next, the event recording unit 57 determines whether or not the event recording request has been received (S30). If the event recording request is not received, the process of FIG. 30 ends.

If the determination in step S30 is Yes, the event recording unit 57 determines whether or not the event recording end has been received (S40). The end time column 614 may or may not be zero at the end of event recording.

When the event recording end of zero in the end time column 614 is received (Yes in S40), the process of FIG. 30 ends.

When the event recording end of zero in the end time column 614 is not received, or when the event recording end is not received (No in S40), it is determined whether or not the scheduled end time has elapsed (S50). This scheduled end time is a time specified in the end time column 614 other than zero, or a time automatically set from the start of recording when the staff Z forgets to press the end button 616. If the determination in step S50 is Yes, the process of FIG. 30 ends.

If the determination in step S50 is No, the event recording unit 57 sets the event flag of the image data to ON (S60). The event recording unit 57 sets the event flag of the image data immediately after the image is captured to ON, and also sets the event flag of the past image data specified in the recording start information to ON.

In this way, the staff Z can send the event recording request to the image management device 5 using the staff terminal 4, and the viewer Y searches the image data in which the event flag is ON, so that the image data in which the event is captured is captured. Can be viewed quickly.

<Request for recording that a gesture event has occurred>
In the above embodiment, since the staff Z needs to operate the staff terminal 4, it tends to take time to request a record that an event has occurred. Therefore, it is convenient if the staff Z can set the event flag to ON by gesture. The staff Z who encounters the event makes a predetermined gesture toward the image pickup apparatus 1.

In this case, the gesture may be an operation that is not performed in daily work. For example, the operation of raising both hands in the vertical direction and opening 90 ° to the left and right is repeated, and the two hands raised above the head form a circle. Alternatively, a specific tool (flag, etc.) may be moved in a fixed direction, or a light may be turned on and pointed at the image pickup apparatus 1. Since the imaging device 1 captures the spherical image, the staff Z can perform the gesture at a position that does not interfere with the imaging of the event. Further, when the staff Z makes a gesture at a position away from the place where the event appears, the staff Z performs an operation of pointing to the place where the event appears. By making this action a gesture, the staff Z can send a request for recording that the event has occurred, and the staff Z can inform the location of the event.

The image data showing the gesture of the staff Z is transmitted to the image management device 5. That is, the staff Z is transmitted to the image management device 5 in a state where the staff Z is reflected in the image data of the spherical surface in which the event appears. The image analysis unit 55 analyzes the image and detects the gesture of the staff Z. When a predetermined gesture is displayed, the event recording unit 57 sets the event flag of the image management table to ON. It is assumed that how much the image data goes back to the past and how long the ON setting is completed is registered in advance in the event recording unit 57.

FIG. 31 is an example of a flowchart showing a procedure in which the image management device 5 detects a gesture and sets the event flag to ON. The process of FIG. 31 is repeatedly executed during the acquisition of image data.

First, the transmission / reception unit 51 of the image management device 5 receives the image data (S10).

Next, the image analysis unit 55 of the image management device 5 analyzes the image data and determines whether or not the gesture is included (S20). If no gesture is included, the process proceeds to step S40.

When the gesture is detected, the event recording unit 57 sets the event flag of the image management table to ON until a predetermined time before a predetermined time (S30). It should be noted that which image pickup device 1 has taken an image is determined by the image pickup device 1 that has transmitted the image data including the gesture. Further, the category ID and the staff ID do not have to be registered. Alternatively, the image analysis unit 55 may identify the staff Z by performing face recognition or character recognition of a name tag or the like. Further, the event recording unit 57 can specify the category ID by changing the gesture according to the event category by the staff Z.

When the gesture is not detected (No in S20), the event recording unit 57 determines whether or not a predetermined predetermined time has elapsed (S40). Until the predetermined time elapses, the event flag of the image management table is continuously set to ON (S50).

When the predetermined time has elapsed (No in S40), the process of FIG. 31 ends without setting the event flag to ON.

In this way, it is possible to request a record that an event has occurred due to a gesture without the staff Z operating the staff terminal 4.

<Send event record request by gesture>
Further, the staff Z may transmit the event recording request to the image management device 5 by a gesture using the staff terminal 4. That is, both the staff terminal 4 and the gesture are used.

FIG. 32 is an example of a sequence diagram showing an operation performed between the staff terminal 4 and the image management device 5.

S1: First, when the staff Z encounters an event, he makes a gesture using the staff terminal 4. For example, the staff terminal 4 is shaken left and right, shaken up and down, turned in a large circle, and the like. The acceleration sensor 420 detects changes in time-series acceleration caused by these gestures. When the event record request unit 44 of the staff terminal 4 detects a predetermined acceleration pattern, it determines that a gesture for transmitting the event record request has been detected.

S2: The event recording request unit 44 of the staff terminal 4 transmits the event recording request to the image management device 5. For this event record request. The staff ID is included, but the terminal ID, event start information, event end information, and category ID are not included.

S3: The transmission / reception unit 51 of the image management device 5 receives the event recording request, and the event recording unit 57 determines the time before and after the reception (before / after time). For example, 5 minutes before and after, 3 minutes before reception, 5 minutes after reception, and so on. This eliminates the need for event start information and event end information.

S4: Next, the image analysis unit 55 detects the gesture of step S1 from the image data of the time before and after determined in step S3. That is, when there are a plurality of image pickup devices 1, it is possible to identify the image pickup device 1 in which the gesture is detected.

S5: The event recording unit 57 of the image management device 5 sets the event flag to ON in association with the image data captured by the identified image pickup device 1. The image data for which the event flag is set to ON can be specified from the time before and after, and the staff ID is also transmitted.

Therefore, the staff Z can transmit the event recording request without operating the event recording screen 680. Even if the event start information and event end information cannot be transmitted, the approximate time before and after can be determined, and the staff ID can also be registered. It is also possible to register a category ID depending on the type of gesture.

<Sending event record request due to change in ambient volume>
In addition to the staff Z proactively transmitting the event recording request, the imaging device 1 or the like may detect a change in the environment and transmit the event recording request to the image management device 5. For example, when some event occurs, the visitor G may make a noise or make a loud noise. Therefore, when the ambient volume exceeds the threshold value, the communication terminal 3 transmits an event recording request to the image management device 5.

FIG. 33 is an example of a sequence diagram in which the communication terminal 3 transmits an event recording request to the image management device 5.
S1: Since the image pickup apparatus 1 includes the microphone 108, the microphone 108 always collects ambient sounds.
S2: Further, the image pickup apparatus 1 periodically images the surroundings.
S3: The image pickup device 1 transmits the image data to the communication terminal 3 together with the sound data collected before and after the image pickup.
S4: The connection unit 33 of the communication terminal 3 receives the sound data and the image data, and for example, the reception unit 32 determines whether or not the volume of the sound data is equal to or higher than the threshold value.
S5: When the threshold value is equal to or higher than the threshold value, the transmission / reception unit 31 transmits an event recording request together with the image data to the image management device 5. The event record request includes a terminal ID. Although the event start information and the event end information are not included, the image management device 5 can determine the approximate time before and after. Further, although the category ID is not transmitted, a category ID indicating that the surroundings are noisy is set.
S6: When the threshold value is equal to or higher than the threshold value, the transmission / reception unit 31 transmits the image data to the image management device 5.

Since the event recording request can be transmitted to the image management device 5 triggered by the volume in this way, the event flag can be set to ON even if the staff Z does not transmit the event recording request to the image management device 5. In addition to the volume, vibration (earthquake, etc.), temperature, humidity, etc. generated in the building may be a trigger.

<About browsing>
FIG. 34 is an example of the browsing screen 690 displayed on the display 508 of the information terminal 7. The browsing screen 690 has a menu field 627, an image unit 625, and a thumbnail unit 626. The menu column 627 has a category designation column 621 and a staff selection column 622. The category designation field 621 is a field for the viewer Y to select a category, and the staff selection field 622 is a field for the viewer Y to select the staff Z. Therefore, the viewer Y can search the image data in which the event flag is ON by designating the category and the staff Z.

Further, the thumbnail image 623 is displayed on the thumbnail unit 626. The icon 624 is displayed on the thumbnail image 623 of the image data in which the event flag is ON. Therefore, the viewer Y can grasp at a glance which image data the event flag is ON without searching for the image data whose event flag is ON.

As described above, in the image processing system 200 of the present embodiment, it is possible to record in the image data that an event has occurred at the timing when the staff encounters the event.

In this embodiment, the information terminal 7 that displays the contents of the analysis information table obtained in Examples 1 and 2 will be described. According to Examples 1 and 2, the degree of interest of the visitor G in each product zone 610 was evaluated in multiple stages. Further, in the third embodiment, the degree of interest of the visitor could be displayed on a heat map or the like. However, in the display of the third embodiment, the viewer Y could not browse the visitor G. That is, in order for the viewer Y to visually grasp how the actual visitor G is interested in each product zone 610 in the store, it is necessary to compare the heat map with the image data.

Therefore, in this embodiment, an image processing system 200 capable of arranging and displaying a visitor G having a predetermined degree of interest in image data will be described. As a result, the viewer Y can analyze the purchasing behavior of the visitor G in more detail.

In this embodiment, the functional block diagram of the image management device 5 and the information terminal 7 may be the same as that of FIG. 13 of the third embodiment. The information visualization unit 74 of the information terminal 7 displays the visitor G who is interested in the product in association with the product zone 610 of the image data. Therefore, it becomes easy for the viewer to understand which product zone 610 on the image data is interested in and how. Specifically, the visitor G (image) showing interest in the product zone 610 is displayed superimposed on the product zone 610 of interest, or one by one in chronological order. Displayed in a slide show. Hereinafter, the description will be divided into several embodiments.

<First Embodiment>
In the first embodiment, the viewer Y can specify an arbitrary degree of interest. The information terminal 7 causes the visitor G showing the designated degree of interest to be displayed in one image data.

FIG. 35 is an example of a diagram showing a visitor composition screen 660 in which a visitor is synthesized with image data. A plurality of visitors G are shown on the visitor composition screen 660. These visitors G are visitors who have shown the same degree of interest. The reason why a plurality of visitor Gs are displayed in one image data is that the information visualization unit 74 synthesizes the visitor Gs. The procedure for displaying such image data will be described below.

FIG. 36 is an example of a flowchart showing a procedure in which the information visualization unit 74 synthesizes the visitor G with the image data. The process of FIG. 36 starts when, for example, the viewer Y performs an operation for analyzing the image.

The reception unit 72 of the information terminal 7 accepts the setting of the population (S10). The population refers to the image data to be analyzed. For example, the image pickup device 1 is specified by the terminal ID, and the viewer Y can set a certain time range, a certain day, a certain day of the week, and the like.

Next, the reception unit 72 accepts the setting of the degree of interest (S20). That is, the viewer Y specifies the degree of interest from the user interface shown in FIG. The viewer Y can select a plurality of interest levels.

Next, the information visualization unit 74 identifies the image data associated with the designated degree of interest from the analysis information table (S30). The image ID is associated with the action (interest level) as shown in the analysis information table of Table 7, and the image ID is associated with the action (interest level) as shown in the analysis information table of Table 9.

Next, the information visualization unit 74 reads out an unmanned image in which no person is shown (S40). The unmanned image is, for example, image data captured before the store opens. It is not necessary to use an unmanned image, and any image data specified in step S30 may be used.

Next, the information visualization unit 74 acquires the region range of the degree of interest specified in the analysis information table and trims the human portion from the image data (S50). The human part is the same as the area range. Since the area range in which the area where a person is detected can be specified is registered in the analysis information table, the rectangular person part in which the person is shown can be trimmed.

Next, the information visualization unit 74 synthesizes the trimmed human portion into the unmanned image (S60). This composition may be a process of simply superimposing (or arranging) a human part on a place specified by a region range. Since the image pickup apparatus 1 captures image data in the same range, the background of each visitor G is the same even if they are combined, so that a sense of discomfort is unlikely to occur.

The information visualization unit 74 displays image data in which one or more visitors G are shown in one unmanned image (S70). As a result, the image data of FIG. 35 is displayed. Since the image data actually shows the visitor G of a certain degree of interest, the viewer can confirm what kind of visitor G (male / female, age group, clothes, etc.) showed this degree of interest. In addition, it is possible to confirm in which place the number of visitors G is large or small. When the viewer Y selects a plurality of interest levels in FIG. 14, it is preferable that each visitor G is color-coded according to the interest level as in the fifth or sixth embodiment.

<Second Embodiment>
The second embodiment is the same in that the viewer Y can specify an arbitrary degree of interest, but the information terminal 7 displays the image data of the visitor G indicating the specified degree of interest one by one.

FIG. 37 is a diagram showing image data 601 displayed on the display 508 by the information terminal 7. In FIG. 37, image data showing the visitor G showing the same degree of interest is displayed in chronological order. When a plurality of visitors G are shown in the image data, the plurality of visitors G are displayed in one image data. In FIG. 37, a play button 641, a fast forward button 643, and a rewind button 642 are displayed. Viewer Y presses the play button 641 to display the next image data, the fast forward button 643 automatically displays the image data one after another, and the rewind button 642 displays the previous image data one after another in time. Let me. The procedure for displaying such image data will be described below.

FIG. 38 is an example of a flowchart showing a procedure in which the information visualization unit 74 synthesizes the visitor G with the image data. The process of FIG. 38 starts when, for example, the viewer Y performs an operation for analyzing the image. The processing of steps S10 to S30 is the same as in FIG. 36.

In step S40, the information visualization unit 74 sorts the image data in chronological order (S40).

After that, the display control unit 73 switches the image data and displays it on the display 508 according to the operation received by the reception unit 72 (S50).

Therefore, since only the visitor G captured at one time is shown in the image data, the viewer Y can easily analyze each visitor G. In addition, it is possible to analyze how the visitor G has moved over time and what kind of visitor G is common.

<Third Embodiment>
In the third embodiment, the visitor G who has shown interest is displayed in the product zone 610 in which the viewer Y is interested.

First, the operation of the viewer and the image data displayed on the display 508 will be described with reference to FIG. 39. FIG. 39 is an example of a diagram illustrating an analysis operation performed by the viewer.

1. 1. As shown in FIG. 39 (a), the viewer Y designates (clicks) the product zone 610 with the pointing device 640 or the like. This product zone 610 is a product zone 610 in which the viewer Y wants to know how many visitors G have shown how much interest.

2. As shown in FIG. 39B, the information terminal 7 displays the product zone 610 including the clicked position on the display 508.

3. 3. As shown in FIG. 39 (c), when there are a plurality of product zones 610, the viewer Y selects one product zone 610 with a mouse or the like.

4. As shown in FIG. 40, the information terminal 7 displays a visitor composition screen 660 in which the visitor G who is interested in the product zone 610 is combined into one image data. According to the visitor composition screen 660, the viewer Y can grasp how many visitors G are interested in each product zone 610. The visitor composite frame 670 of FIG. 40 is a circumscribed rectangle of a plurality of visitor Gs, but the larger the visitor composite frame 670 is, the wider the location of the visitor G who is interested in one product zone 610 is. become. Therefore, the viewer Y can change the arrangement of the products by judging whether or not the target customer is attracted. The shape of the visitor composite frame 670 may be circular, polygonal, or the like, and the shape is not limited.

The procedure for displaying such image data will be described below.
FIG. 41 is an example of a flowchart showing a procedure in which the information visualization unit 74 synthesizes the visitor G with the image data. The process of FIG. 38 starts when, for example, the viewer Y performs an operation for analyzing the image. The process of step S10 is the same as in FIG. 38.

Next, the reception unit 72 accepts the designation of the location of arbitrary image data (S20). Any image data may be an unmanned image, but any image data may be used because the imaging range of the image data is the same.

Next, the information visualization unit 74 identifies the product zone 610 including the location of step S20 (S30). Since the coordinates of the target zone are registered in the analysis information table, the product zone 610 including the location of step S20 can be specified.

The information visualization unit 74 determines whether or not a plurality of product zones 610 have been identified (S40). When there are a plurality of products, the display control unit 73 emphasizes and displays the plurality of product zones 610 on the image data, and the reception unit 72 accepts the selection of one product zone 610 (S50).

Next, the information visualization unit 74 identifies the area range and image data associated with the selected product zone 610 (S60). As described above, the area range associated with the product zone 610 can be specified from the analysis information table, and the image data having this area range can be specified. Subsequent processes in steps S70 to S100 may be the same as in steps S40 to S70 of FIG.

In this way, since the visitors who are interested in the product zone that the viewer Y wants to know can be displayed, the viewer Y can analyze what kind of visitors are interested and how much.

<Fourth Embodiment>
Similarly to the third embodiment, the fourth embodiment also displays the visitor G who has shown interest in the product zone 610 in which the viewer Y is interested. However, in the fourth embodiment, the information terminal 7 causes the designated product zone 610 to display the image data of the visitor G who has shown the degree of interest one by one.

FIG. 42 is an example of a diagram illustrating a method of generating image data displayed in the fourth embodiment.
1. 1. The method of generating the image data in FIG. 42 (a) is the same as that in FIG. 40. In FIG. 42 (a), the product zone 610 designated by the viewer Y is displayed, but this may not be necessary.
2. Next, as shown in FIG. 42B, the information terminal 7 calculates the circumscribed rectangle of the area range occupied by the visitor G who is interested in the product zone 610. This is the visitor composition frame 670 of FIG. 40.
3. 3. As shown in FIG. 42 (c), the information terminal 7 highlights and displays the visitor composition frame 670.
4. FIG. 43 is an example of a diagram in which image data showing a visitor G who has shown an interest in the same product zone 610 is displayed in chronological order together with a visitor composition frame 670.

When a plurality of visitors G are shown in the image data, the plurality of visitors G are displayed. Of the visitors G who are interested in a certain product zone 610, only the visitors G imaged at one time are displayed, so that the viewer Y can easily analyze each visitor G. In addition, it is possible to analyze how the visitor G has moved over time and what kind of visitor G is common. In FIG. 43, the visitor G may be shown outside the visitor composite frame 670, but since there is a visitor composite frame 670, the viewer can see the visitor G outside the visitor composite frame 670. It can be ignored. Further, only the visitor G in the visitor composite frame 670 may be trimmed and arranged in the unmanned image.

The procedure for displaying such image data will be described below.
FIG. 44 is an example of a flowchart showing a procedure in which the information visualization unit 74 synthesizes the visitor G with the image data. The process of FIG. 44 starts when, for example, the viewer Y performs an operation for analyzing the image. The processing of steps S10 to S60 is the same as in FIG. 41.

In step S70, the information visualization unit 74 sorts the image data in chronological order (S70).

After that, the display control unit 73 switches the image data and displays it on the display 508 according to the operation received by the reception unit 72 (S80). At this time, the display control unit 73 displays the visitor composition frame 670.

Therefore, the viewer Y can easily analyze the visitor G who is interested in a certain product zone 610 individually. In addition, it is possible to analyze how the visitor G moves with time in the visitor synthesis frame 670 and what kind of visitor G is large.

<Fifth Embodiment>
In the fifth embodiment, an image processing system for displaying the visitor G in a color-coded manner according to the degree of interest in the third embodiment will be described.

FIG. 45 is an example of a diagram illustrating image data displayed on the display 508. In FIG. 45, each visitor G is displayed in a different color depending on the degree of interest. The process up to color coding may be the same as in the third embodiment.

FIG. 46 is an example of a flowchart showing a procedure in which the information visualization unit 74 synthesizes the visitor G with the image data. The process of FIG. 46 starts when, for example, the viewer Y performs an operation for analyzing the image. First, FIG. 46 is the same as FIG. 41 except for step S85.

In step S85, the information visualization unit 74 colors the human part according to the degree of interest (S85). Actions (interest) are associated with the area range as shown in the analysis information table of Table 7, and actions (interest) are associated with the area range as shown in the analysis information table of Table 9. There is. As a result, as shown in FIG. 45, the visitor G can be displayed in different colors depending on the degree of interest. Since the human portion is a rectangular area, it may be difficult to display it in a human shape as shown in FIG. 45. In this case, it is preferable that the display control unit 73 applies a semi-transparent process to each person's portion.

Therefore, even if a plurality of visitors G are displayed for a certain product zone 610, the viewer Y can easily determine the degree of interest of each visitor G.

<Sixth Embodiment>
In the sixth embodiment, an image processing system for displaying the visitor G in a color-coded manner according to the degree of interest in the fourth embodiment will be described.

FIG. 47 is an example of a diagram illustrating image data displayed on the display 508. In FIG. 47, each visitor G is displayed in a different color depending on the degree of interest. The process up to color coding may be the same as in the fourth embodiment.

FIG. 48 is an example of a flowchart showing a procedure in which the information visualization unit 74 synthesizes the visitor G with the image data. The process of FIG. 48 starts when, for example, the viewer Y performs an operation for analyzing the image. First, FIG. 48 is the same as FIG. 44 except for step S80.

In step S80, the information visualization unit 74 accepts the operation of the viewer, colors the human part according to the degree of interest, and switches and displays it (S80). Actions (interest) are associated with the area range as shown in the analysis information table of Table 7, and actions (interest) are associated with the area range as shown in the analysis information table of Table 9. There is. As a result, as shown in FIG. 47, the visitor G can be displayed in different colors depending on the degree of interest.

Therefore, the viewer Y can analyze the degree of interest of the visitor G who is interested in a certain product zone 610 one by one (one image data at a time). In addition, it is possible to analyze how it changes over time.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, in the above embodiment, the image management device 5 detects a person or determines an action, but the information terminal 7 may detect a person or determine an action. Further, in the above embodiment, the information terminal 7 determines the color of the product zone 610, but the image management device 5 may determine the color of the product zone 610. The image management device 5 may perform the processing of the fifth embodiment. Further, the information processing device in which the information terminal 7 and the image management device 5 are integrated may execute the processing described in the present embodiment by one unit. Further, the information terminal 7 may have all or one or more of the functions of the image management device 5, and the image management device 5 may have all or one or more of the functions of the information terminal 7.

Further, the configuration examples of FIGS. 6, 13, and 24 shown in the above embodiments are mainly used in order to facilitate understanding of the processes of the image pickup device 1, the communication terminal 3, the image management device 5, and the information terminal 7. It is divided according to the function. However, the present invention is not limited by the method and name of division of each processing unit. The processing of the image pickup device 1, the communication terminal 3, the image management device 5, and the information terminal 7 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing. Further, the image processing system 200 may have a plurality of image management devices 5.

Further, the database of the storage unit 5000 of the image management device 5 may be directly owned by the image management device 5 or may be on a communication network 9 on which the image management device 5 can read and write.

The person detection unit 52 is an example of a person detection means, the image analysis unit 55 is an example of an operation determination means, the evaluation unit 56 is an example of an evaluation means, and the display control unit 73 is an example of a display means. , Display 508 is an example of a display device. The product zone 610 is an example of a predetermined area, the area 602 is an example of a person area, the reception unit 72 is an example of the reception means, and the event recording unit 57 is an example of the flag setting means.

1: Imaging device 3: Communication terminal 5: Image management device 7: Information terminal 44: Event recording request unit 51: Transmission / reception unit 52: Person detection unit 55: Image analysis unit 56: Evaluation unit 57: Event recording unit 73: Information vision Chemical unit 74
200: Image processing system

Japanese Patent No. 4972491

Yoshito Nakajima, "Study on Detection and Recognition of Moving People by Image Processing for Surveillance Systems", Graduate University for Advanced Studies, Bachelor's Degree Paper, September 29, 2006

Claims (19)

An image processing system having one or more information processing devices.
A person detection means that detects a person area including a person from image data,
An motion determining means for determining the motion of the person and the direction of the motion from the human area detected by the human detecting means, and
An evaluation means for evaluating the region determined by being in the direction among a plurality of predetermined regions, or the region determined according to the direction and the size of the human region according to the operation. When,
A display means for displaying the evaluation by the evaluation means on the display device together with the image data,
It has a reception means for accepting the designation of the area.
The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. Count and
The display means is an image processing system that visualizes and displays the number of times of each of the evaluation stages associated with the area received by the reception means. The evaluation means evaluates the region in multiple stages according to the operation.
The image processing system according to claim 1, wherein the display means displays a multi-step evaluation by the evaluation means on a display device together with the image data. The image processing system according to claim 1 or 2, wherein the evaluation means determines the region 1 from a plurality of predetermined regions based on the direction and evaluates the region according to the operation. The image processing system according to claim 3, wherein the evaluation means determines the region on an extension line in the direction from a plurality of predetermined regions and evaluates the region according to the operation. Said evaluating means, among the plurality of regions are determined in advance, it is an extension of the direction, and, in response to the operation to determine the area adjacent to the person region in which the direction is determined The image processing system according to claim 4 to be evaluated. The image processing system according to claim 3, wherein when the direction is not determined, all the regions adjacent to the human region are evaluated according to the operation. The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. Count and
It has a reception means that accepts the designation of the area.
The image processing system according to any one of claims 1 to 6, wherein the display means visualizes and displays the number of times of each of the evaluation stages associated with the area received by the reception means. .. The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. Count and
It has a reception means that accepts predetermined operations,
When the receiving means receives the predetermined operation, the display means calculates a representative value representing the area from the evaluation stage and the number of times, and visualizes and displays the representative value. The image processing system according to any one of 6. Said evaluating means, the image processing system according to claim 1 to the person region determines the range adjacent to the direction of the operation to the region, it is evaluated according to the region in the operation. The image processing system according to claim 9 , wherein the evaluation means determines the size of the range adjacent to the human region according to the width and height of the human region. The evaluation means imparts the evaluation stage to the region when the region is evaluated in multiple stages.
It has a reception means that accepts the designation of the evaluation stage.
The image processing system according to claim 9 or 10 , wherein the display means visualizes and displays the area to which the evaluation stage received by the reception means is given. The evaluation means imparts the evaluation stage to the region when the region is evaluated in multiple stages.
It has a reception means that accepts the designation of the position of the image data.
The image processing system according to claim 9 or 10 , wherein the display means visualizes and displays the evaluation stage given to the area closest to the position received by the reception means. The evaluation means imparts the evaluation stage to the region when the region is evaluated in multiple stages.
It has a reception means that accepts predetermined operations,
If the reception unit receives the predetermined operation, the display means, overlaps a plurality of the regions as the concentration of the center is increased larger, the region in the color gradually concentration decreases with increasing distance from the center The image processing system according to claim 9 or 10. When a notification that the image pickup device that captures image data is capturing a predetermined event is obtained from an arbitrary information processing device,
The image processing system according to any one of claims 1 to 13 , further comprising a flag setting means for setting an event flag indicating that an event is captured in the image data at least when the notification is acquired. Person the region of the specified evaluation phase and direction of operation, or the display means to display the image data including the person who indicates the direction of the operation to the region on the display device,
The image processing system according to any one of claims 1 to 13. An information processing device capable of communicating with an information terminal that displays the evaluation by the information processing device together with the image data captured by the image pickup device on the display device.
A person detection means that detects a person area including a person from image data,
An motion determining means for determining the motion of the person and the direction of the motion from the human area detected by the human detecting means, and
An evaluation means for evaluating the region determined by being in the direction among a plurality of predetermined regions, or the region determined according to the direction and the size of the human region according to the operation. And have
The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. Count and
An information processing device that transmits the number of times the same evaluation stage is given to the area to the information terminal for each area. A person detection means that detects a person area including a person from the image data captured by the image pickup device, and
An motion determining means for determining the motion of the person and the direction of the motion from the human area detected by the human detecting means, and
An evaluation means for evaluating the region determined by being in the direction among a plurality of predetermined regions, or the region determined according to the direction and the size of the human region according to the operation. And have
The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. An information terminal that can communicate with the information processing device that counts
Display means for displaying on the display device evaluation by the evaluation unit together with the image data,
It has a reception means for accepting the designation of the area.
The display means is an information terminal that visualizes and displays the number of times of each of the evaluation stages associated with the area received by the reception means. An information processing device that can communicate with an information terminal that displays the evaluation by the information processing device on the display device together with the image data captured by the image pickup device.
A person detection means that detects a person area including a person from image data,
An motion determining means for determining the motion of the person and the direction of the motion from the human area detected by the human detecting means, and
An evaluation means for evaluating the region determined by being in the direction among a plurality of predetermined regions, or the region determined according to the direction and the size of the human region according to the operation. To function as,
The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. Count and
A program that transmits the number of times the same evaluation stage is given to the area to the information terminal for each area. A person detection means that detects a person area including a person from the image data captured by the image pickup device, and
An motion determining means for determining the motion of the person and the direction of the motion from the human area detected by the human detecting means, and
An evaluation means for evaluating the region determined by being in the direction among a plurality of predetermined regions, or the region determined according to the direction and the size of the human region according to the operation. And have
The evaluation means assigns the evaluation stage when the region is evaluated in multiple stages to the region, and assigns the same evaluation stage to the region of the plurality of image data for each region. An information terminal that can communicate with the information processing device that counts
Display means for displaying on the display device evaluation by the evaluation unit together with the image data,
To function as a reception means for accepting the designation of the area,
The display means is a program that visualizes and displays the number of times of each of the evaluation stages associated with the area received by the reception means.

Images