TW202143167A - System and method for visitor interest extent analysis - Google Patents

Abstract

A system for a visitor’s interest extent analysis and a method thereof is disclosed. The system and method is applied to analyze image data to calculate out the targets that a visitor may be looking at. With the positions of objects in the environment, it is inferred that what the visitor is interested in and the degree of interest. The system and method recognizes, through the deep learning according to an algorithm under the convolutional neural network (CNN) architecture, the head direction of the visitor in the image data. Further, by means of a virtual light source placed behind the head of the visitor, the gazing area where the visitor may be looking at is accordingly obtained. When the position of a target is located in a sub-area representing a certain probability within the gazing area, it is determined that the visitor has a degree of interest on the target, and it also means that the visitor may be looking at the target.

Description

System and method for analyzing the degree of visitor's interest

The present invention discloses a method for judging (Estimation) what kind of object the people/customers are looking at in the image data. In particular, it is a detection method based on image data under the architecture of Convolutional Neural Network (CNN). This method does not need to use the image data recorded by two camera lenses to track the face and The direction of the eyeball, only through the image data recorded by a single image lens, can achieve a method of identifying what kind of object people are looking at in the image data.

Please refer to FIG. 1A, which illustrates a Gaze Tracking determination method implemented in the prior art. FIG. 1A shows that there is a person watching (Gaze) an object (Object) 50 in front of his eyes in the environment. In addition, cameras are set up at two different locations (Location) in this environment, which are called Camera 1 (Cam 1, the dotted circle) 60 and Camera 2 (Cam 2, the dotted circle) 70. These two cameras are paired with each other. People entering the environment and objects 50 in the environment are photographed; among them, the face images captured by the two cameras Cam 1 60 and Cam 2 70 at a certain point in time are shown in FIG. 1B. The two cameras Cam 1 60 and Cam 2 70 calculate the center line of the face (Facial Central Line, not shown) and the direction of the eyeballs in the eyes (not shown) based on the information on the captured face images. The direction of the person's gaze (Gazing Line) 40 is obtained, and then it is estimated whether the target of the person's gaze is the object 50.

In the prior art in FIG. 1A, the following conditions must be met in order to obtain the person's gaze line 40 and the object 50 that the person is looking at in the image data:

(a) Under normal circumstances, two cameras must be used to implement this prior art. If only one camera is installed in the environment, it cannot be achieved. In another situation, if a 3D Camera is used, because one 3D Camera will be equipped with two or more lenses (Lens), the above effect of setting two general cameras can also be achieved. However, whether it is to use two general cameras or one 3D camera, a higher camera cost is required to achieve this prior art.

(b) Because this prior art needs to recognize the eyes that are not a large part of the human face and the direction of the eyeballs in the eyes, the "resolution" of the image data needs to be higher than a certain threshold to achieve the recognition of the "face" The purpose of "center line" and "eyeball deviation angle". In addition, if the face of the person in the image data is hidden or because the image resolution is not enough, when one of the person’s "face center line" and "eyeball deviation angle" in the image data is not known , It is impossible to calculate what the character is looking at.

(c) A larger storage space for image data is required to store image data with higher resolution. Because this prior art requires higher-resolution image data, the size of each image data will increase, and the same amount of data requires more storage space, thus increasing the cost of storage space.

In addition, due to the complicated calculations when using this prior art to identify image data, it takes more time or saves time with higher hardware costs. In other words, using lower-cost hardware will take a long time and poor performance; if you need to shorten the time and increase performance, you must invest in higher hardware costs.

From the above description, the shortcomings of the prior art represented by FIGS. 1A and 1B can be known. Therefore, the present invention proposes a technical method to solve the above-mentioned shortcomings of the prior art, which not only reduces the cost of the hardware used, but also increases the overall performance of the system.

An object of the present invention is to provide a visitor interest level analysis system for analyzing a visitor’s interest level in at least one item. The system includes: at least one image capturing device set in a place for capturing the An image data of a location, in which a first head image of the visitor is recorded; and an image analysis server, connected to the at least one image capture device, for calculating and analyzing from the at least one image capture Taking the image data of the device, the image analysis server further includes: a data processing center for executing an image analysis application program, which can be based on a first feature mapping (feature mapping) obtained from the first head image ) To determine a first head direction corresponding to the first head image, and calculate a first projection area according to the first head direction; and a memory unit for temporarily storing the image data, the The first head image, the first feature mapping (feature mapping), and other related data required or produced during the operation of the data processing center; wherein the calculation of the first projection area uses a virtual light source , The virtual light source is placed behind the head position of the visitor and casts light in a direction consistent with the first head direction to form the simulated first projection area, when the first projection area covers the position of the at least one object Time, it means that the visitor has a degree of interest in the at least one item.

Another object of the present invention is to provide a visitor interest level analysis system, which is connected to at least one image capture module and receives an image data from the at least one image capture module, which is used to analyze a visitor to at least one target The system includes: a data processing center that executes an image analysis application program, and can determine the first feature mapping based on a first feature mapping obtained from a first head image in the image data A first head direction corresponding to the head image, and a first projection area is calculated according to the first head direction; and a memory unit for temporarily storing the image data, the first head image, The first feature mapping (feature mapping) and other related data required or produced during the operation of the data processing center; wherein, the calculation of the first projection area uses a virtual light source which is placed on The first projection area is formed by projecting light rays behind the visitor's head position in a direction consistent with the first head direction. When the first projection area covers the at least one target position, it means the visitor There is a degree of interest in the at least one subject.

A further object of the present invention is to provide a method for analyzing the degree of interest of visitors. The method is executed by an image analysis server to determine the degree of interest of a visitor in at least one subject. The method includes: providing an image analysis application The program is in the image analysis server; the image analysis server obtains an image data; the image analysis server detects a first head image with a first head feature in the image data; the image analysis server Analyze the first head image, and determine a first head direction corresponding to the first head image through a first feature mapping obtained by the analysis; the image analysis server calculates the first head direction The location of a head image in a three-dimensional space, and a simulated first projection area is calculated based on the location of the first head image, the first head direction, and a virtual light source; and the image analysis The server determines whether the visitor corresponding to the first head image has a degree of interest in the at least one object according to the coverage of the first projection area and the position of the at least one object.

Please refer to FIG. 2, according to an embodiment of the present invention, FIG. 2 is an image data capture and analysis system (Image Data Capture and Analysis System, IDCAS) 100, the system 100 includes a plurality of image capture devices (Image Capture Device ) 400A to 400N, a Cloud Storage Unit 150 and an Image Analysis Server 300. The plurality of image capturing devices 400A to 400N and the cloud data storage unit 150 transmit signals and data to each other through a network 180 or transmission line, and the cloud data storage unit 150 and the image The analysis servers 300 transmit signals and data to each other through a network 190 or a transmission line.

According to an embodiment of the present invention, the plurality of image capturing devices 400A to 400N in FIG. 2 can be any form of general cameras or IP cameras, for example: can remotely control up and down elevation angle, left and right rotation, and the distance of the lens. Tripod Head Camera, Doom Camera, Infrared Camera, Fisheye Camera, 3D Camera, etc. The plurality of image capturing devices 400A to 400N are set in different locations (Sites), adopting continuous uninterrupted photography mode, time-lapse photography mode, or motion detection mode to capture the image of each location. The image data in the environment is transmitted to the cloud data storage unit 150 for storage through the network 180 or transmission line. The network 180 may be a local area network (LAN), a wide area network (Wide Area Network, WAN), an Internet (Internet), or a wireless network (Wireless Network). The format of the image data captured by the image capture devices 400A to 400N can be the audio-visual format established by the "Moving Picture Experts Group" (MPEG), such as MPEG-4, MPEG-2, etc., Or other audio and video formats, such as: Audio Video Interleave (AVI), Real Media Variable Bitrate (RMVB)... etc. The specifications of the image data can be, for example, resolution (Resolution) 1980X1080 pixels (pixel)/30 frames (Frame Per Second, FPS) image data, but it is not limited to the image data of this specification, and it can also be any Image data with pixel/frame number specifications. The above-mentioned continuous uninterrupted photography mode refers to the continuous photography of the camera regardless of day and night; the above-specified photography mode refers to the setting of the camera to only shoot at a certain or certain period of time. For example: 09:00 – 20:00 (business hours) for photography and no photography at other times; or 11:00-14:00 and 17:00-21:00 for photography, and other times No photography. The aforementioned motion detection shooting mode means that when the camera detects the movement of objects in its environment, it will trigger the camera's shooting function. The shooting time may be a preset period of time or last until No more movement of objects is detected until now. According to another embodiment of the present invention, the plurality of image capturing devices 400A to 400N may process part of the image data through an image processing device (not shown) before transmitting the image data to the cloud data storage unit 150 for storage. Then it is sent to the cloud data storage unit 150, and then the image analysis server 300 downloads the image data from the cloud data storage unit 150 for further processing. Since the processing of image data will take up considerable resources of the graphics processing unit (GPU) 370 of the image analysis server 300, this approach can reduce the amount of image data sent to the image analysis server 300, and can also reduce the cost of the image analysis server 300. The workload increases the overall performance of the system 100. According to an embodiment of the present invention, the aforementioned image processing device (not shown) may be other data servers (Data Server), personal computers (PC), notebook computers (Notebook PC), and tablet PCs. Or another image analysis server 300.

According to an embodiment of the present invention, the cloud data storage unit 150 in FIG. 2 refers to a cloud data storage environment constructed by a cloud provider. The cloud provider may be, for example, Google, Amazon, or Alibaba. Baba, DropBox… and other vendors that provide cloud data environments. The cloud data storage unit 150 provides sufficient data storage space for the image capture devices 400A to 400N in FIG. 2 for long-term storage of image data, while the internal storage space of the image capture devices 400A to 400N is only used for temporary storage of data . According to another embodiment of the present invention, the cloud data storage unit 150 can be omitted, and the image capture devices 400A to 400N in FIG. 2 can also directly send the image data to the image analysis server 300 for processing.

Please refer to FIG. 2, according to an embodiment of the present invention, the image analysis server 300 in FIG. 2 downloads the image data sent to the cloud data storage unit 150 from the image capture devices 400A to 400N through a network 190, and then performs a Series of image analysis and processing. The network 190 may be a local area network (LAN), a wide area network (WAN), an Internet (Internet), or a wireless network (Wireless Network). Before the image analysis server 300 performs the above-mentioned image analysis and processing, it must first perform deep learning to train itself to have the ability to recognize specific features in the image; among them, the deep learning is basically The above is the application of Convolutional Neural Network (Convolutional Neural Network, CNN) algorithm. The image analysis server 300 reads a data set (Data Set) of a defined specific category from a connected database (Data Base) 450, and learns through the algorithm of Convolutional Neural Network (CNN) Represents the feature of the data set; for example: "learning" the feature of the "face" in the image data from the data set (Data Set) that defines the "face". After the training/learning phase is completed, the image analysis server 300 is capable of identifying whether other image data has the specific feature (Feature).

Please refer to FIG. 3A, which is a block diagram of the basic architecture of the hardware 480 of the image analysis server 300 in FIG. 2 according to an embodiment of the present invention. According to an embodiment of the present invention, the basic hardware 480 architecture of the image analysis server 300 includes: a central processing unit (CPU) 310, a read-only memory (ROM) 340, and a dynamic random access memory (DRAM) ) 320, a Storage Interface Controller 350, a Physical Storage Device Array (PSD Array) 380, a Non-Volatile Memory (NVRAM) 330, and a Network Interface Controller (Network Interface Controller) 380 Interface Controller (NIC) 360 and a graphics processing unit (GPU) 370. The above-mentioned units transmit information and data to each other through one or more buses (Bus) 390. Wherein, the central processing unit (CPU) 310 and the graphics processing unit (GPU) 370 may be two independent units, or integrated into a chip or a software/hardware module to form a data processing center; the entity The storage device array (PSD Array) 380 also includes a plurality of physical storage devices (PSD) 385, such as hard disk (Hard Disk Drive, HDD), solid state disk (Solid State Disk, SSD), or other A physical storage device 385 that achieves the same storage function.

In FIG. 3A, the central processing unit 310 (or the data processing center after the central processing unit (CPU) 310 and the graphics processing unit (GPU) 370 are integrated) is a core of the image analysis server 300 The unit is used to execute data processing procedures between hardware, operating systems, and applications. The central processing unit 310 (or the data processing center) can be a Power PC, an x86 or any architecture CPU. The read-only memory 340 is used to store the basic input/output system (BIOS) and/or other programs when the image analysis server 300 is turned on. The dynamic random access memory (DRAM) 320 is used as a temporary storage place for CPU commands or various image data, which can be used to store the image data from the cloud data storage unit 150 for waiting for the central processing unit 310 And/or processed by the graphics processing unit 370; or temporarily store the processed data of the central processing unit 310 and/or the graphics processing unit 370, and then store it in the physical storage device array 380 at an appropriate time, or The above-mentioned processed data is sent out through the network interface controller 360. The graphics processing unit 370 is another core unit of the image analysis server 300, and is basically used to process image data related to graphics. The hardware system of the graphics processing unit 370 is specially designed to meet the requirements of processing graphics or images. Therefore, it is generally much faster than the central processing unit 310 in processing image data, and is suitable for processing a large amount of image data. The non-volatile memory (NVRAM) 330 can be implemented by flash memory (Flash memory), which is used to store data related to the execution status of the I/O request, in preparation for the I/O request (I/O request). O request) When the related operation of accessing the physical storage device array 380 is not completed, it is used for inspection when abnormal power off occurs. The storage interface controller 350 is a storage interface for storing data processed by the central processing unit 310 and/or the graphics processing unit 370 in the physical storage device array 380 or from the physical storage device array 380 Relevant data is read internally and temporarily stored in the dynamic random access memory 320 for processing by the central processing unit 310 and/or the graphics processing unit 370. The communication protocol used by the storage interface controller 350 can be Fibre Channel (FC), serial attached small computer system interface (Serial Attached SCSI, SAS), serial advanced technology access (Serial ATA, SATA) , Or any applicable transmission protocol. The physical storage device array 380 is composed of a plurality of physical storage devices 385 to provide space for the image analysis server 300 to store data. According to another embodiment of the present invention, when the image analysis server 300 does not provide storage space for data, the physical storage device array 380 can be omitted. In this case, the data is modified and stored in the non-volatile memory ( NVRAM) 330 or stored in an external storage device, such as JBOD. The network interface controller 360 is externally connected to a network, which transmits data or messages processed by the central processing unit 310 and/or the graphics processing unit 370 to other devices on the network through the network, or transmits Data from other devices on the network are sent to the dynamic random access memory 320 for temporary storage. According to another embodiment of the present invention, when the data processing capability of the central processing unit 310 is sufficient for processing external commands and processing image data at the same time, the aforementioned data processing center only includes the central processing unit 310, and the graphics processing unit 370 can be omitted.

Please refer to FIG. 3B. FIG. 3B is a schematic diagram illustrating the relationship between the software and hardware architecture of the image analysis server 300 in FIG. 2 according to an embodiment of the present invention. The soft architecture of the image analysis server 300 in FIG. 3B is based on the hardware 480, and the organization structure of the hardware 480 is as shown in FIG. 3A.

According to the embodiment shown in FIG. 3B, there is a hypervisor (Hypervisor) 470 between the hardware 480 of the image analysis server 300 and the operating system (OS) 475, which can also be referred to as a virtual machine monitor (Virtual Machine Monitor). Machine Monitor, VMM), the Hypervisor 470 (or VMM) can be implemented in software, firmware or hardware. The super controller 470 provides a virtual operating platform for one or more operating systems (OS) to share the resources of the hardware 480. Therefore, the super controller 470 can also be regarded as the "meta-operation of the operating system 475" in FIG. 3B. The "Pre Operating System" is mainly used to coordinate and allocate hardware 480 resources for use by multiple operating systems running on the image analysis server 300. Without interrupting the operation of each operating system, the super controller 470 can automatically adjust (increase or decrease) the hardware resources that any operating system can use, such as: allocated CPU time, memory space, network interface , Hard disk storage space and other hardware resources, making the workload between operating systems close to balance. Although only one operating system 475 is depicted in FIG. 3B, in fact, there may be multiple operating systems running on the super controller 470. According to another embodiment of the present invention, a second operating system can be executed on the super controller 470. The program in the second operating system mainly organizes the multiple physical storage devices 385 in FIG. 3A into a plurality of data blocks (Data Block), and uses a RAID (Redundant Array of Independent Disks) mechanism (for example: RAID 5). Or RAID 6) protect the data stored in the plurality of data blocks to avoid data loss due to damage to one or two of the physical storage devices 385. According to another embodiment of the present invention, if the image analysis server 300 only needs a single operating system, the super controller 470 can be omitted.

In FIG. 3B, the operating system 475 may be a common operating system, such as Windows, Linux, Solaris, etc. The operating system 475 provides a multi-task (Multi-Task) and time-sharing (Time Sharing) operating environment, allowing multiple applications and programs to be executed at the same time. In FIG. 3B, there are two functional blocks on the operating system 475, representing an image analysis application 460 and a Cloud Gateway Service application 465, respectively. The image analysis application 460 is a software, or a module with both software and hardware, which is composed of the central processing unit (CPU) 310, or the graphics processing unit (GPU) 370 in FIG. 3A, or the integrated Executed by the data processing center (corresponding to hardware 480). The image analysis application 460 is used to detect and analyze the image data obtained by the image capturing devices 400A to 400N in FIG. 2. After analyzing the image data through the image analysis application 460, the relevant information of visitors and products captured in the image data can be obtained, and with the relevant data such as date and time, after proper calculation, many users can finally be obtained. Statistics information and output. The cloud channel service application 465 is an intermediary service program that performs data transmission between the external and the cloud data storage unit 150 as a related application executed on the operating system 475, and accepts instructions from the related application to the cloud data The storage unit 150 performs file access. According to another embodiment of the present invention, when the system 100 does not need to store/analyze data through the cloud data storage unit 150, the cloud channel service application 465 can be omitted.

According to another embodiment of the present invention, the image analysis server 300 in FIG. 2 can also be located on the cloud data storage unit 150; in other words, the image analysis server 300 can be provided by the supplier of the cloud data storage unit 150 One of the virtual machines (Virtual Machine, VM). The operating system running on the virtual machine is also provided by the supplier of the cloud data storage unit 150. The image analysis application 460 in FIG. 3B can be executed in the operating system on the virtual machine; that is, after the user uploads the image analysis application 460 to the cloud data storage unit 150, the image analysis is executed through the virtual machine Application 460. In this case, the Data Base 450 used by the image analysis server 300 in the cloud may be one or more object files in the cloud data storage unit 150. In this way, the image data sent by the image capturing devices 400A to 400N to the cloud data storage unit 150 can be immediately analyzed and processed by the virtual machine also located in the cloud, and the relevant data generated after the analysis and processing can also be stored in the cloud data storage unit 150 , And then send it to other countries as needed.

According to another embodiment of the present invention, the image analysis server 300 in FIG. 2 can also be integrated into the image capture device 400 (ie, 400A to 400N). In this case, the image analysis application 460 in FIG. 3B is moved to the image capturing device 400 for execution. The image analysis application 460 further includes a plurality of functional modules, all or part of the functional modules as shown in FIG. 4. According to an embodiment of the present invention, the image capturing device 400 has the functions of recording image data and analyzing/processing the image data at the same time. After the image capture device 400 analyzes/processes the image data, the complete or partial related information obtained will be stored in an internal memory unit, such as an SD card (Secure Digital Memory Card) or a built-in flash memory ( Flash Memory). According to another embodiment of the present invention, after the image capture device 400 analyzes/processes the image data and obtains complete or partial related information, it will transmit the analysis result to the cloud data storage unit 150 for storage or transmission via the network 180 Go to the image analysis server 300 for further analysis and processing. The above-mentioned complete or partial related information can be, for example: image data after face detection, image data after product detection... etc.

Please refer to FIG. 4, which shows the functional modules further included in the image analysis application 460 in FIG. 3B. According to an embodiment of the present invention, the image analysis application 460 includes all the following functional modules; according to another embodiment of the present invention, the image analysis application 460 only includes some of the functional modules in FIG. 4. Although each functional module in FIG. 4 is shown in an independent manner, according to an embodiment of the present invention, in actual implementation, two or more functional modules may be integrated together. The description of each functional module in Figure 4 is as follows:

(A) People Flow analysis module 405: Detect the part of the image data belonging to the characteristics of people, and analyze the trajectory of the people flowing in the image data, to calculate the number of people in the environment, evaluate the degree of crowding, or distinguish visitors in the environment The direction of movement in. One of the applications can be to use the image data recorded in one or more image capture devices 400 installed in a shopping mall to count the number of shoppers and/or the attributes of visitors (for example: age, gender, Occupation...), which also includes the movement directions of visitors with different attributes when shopping.

(B) Head Detection (Head Detection) module 410: Detect the part of the image data that has the characteristics of the human head, and analyze the position and attributes of each head image. The attribute of the so-called head image can be, for example, by analyzing the features of the head or face in the image data to determine the head direction of the person's head; the above determining the head direction ( The head direction is based on the "feature mapping" or "feature vector" obtained by the image analysis server 300 after the training phase of the head direction. In addition, the head detection module 410 can also further determine other attributes of the head or face, such as open or closed eyes, emotions expressed, hair color, visual geometry of the face, or other head related attributes. Attributes. The attribute data of the head image and the metadata tags (for example: happiness, glasses, age range, member information...) can help users quickly sort or search or identify people. The head detection module 410 may require the image data to have a certain pixel size or a certain resolution or higher, so that the characteristics of the customer's face/head can be identified.

(C) Visitor's ID identification module 415: compare the captured facial features of the visitor with the existing visitor data (face features) in the database to identify the identity of the visitor (for example: name) Or code, etc.), as well as further information about the identity (for example: frequently purchased items, payment habits, etc.). In order to capture the facial features of visitors, you can use the room in function of the video camera, or the function of changing the resolution of the image data during recording, to help capture the facial features of the traveling visitor, and The facial feature data in the database is compared to see if there are the same or close facial data, so that it can be understood whether it is the same person who has entered the environment at different time points. According to an embodiment of the present invention, for a visitor's ID, at least 10 (but not limited to this number) of facial feature image data of various angles are recorded in the database, and these facial features are used The feature image data compares/recognizes the facial features of the visitor in the image data captured by the image capturing device 400. According to another embodiment of the present invention, if the image capturing device 400 is unable to identify the features of the visitor’s face, the visitor identification module 415 will identify the visitor’s identity based on the unique features displayed by each part of the visitor’s body while walking in the image. Identity, such as: the swing of the visitor's hands and feet when walking, the walking posture, the length ratio between the head/back/hands and feet... and other unique appearance characteristics of the visitor to identify the visitor's identity.

(D) Product Detection (Product Detection) module 420: Detects the part of the image data with product characteristics, and can display various or specific products in a frame according to the setting. Generally speaking, the pixel/resolution of the captured image data needs to be above a certain size before the product detection module 420 can identify the characteristics of various products, and then according to the settings, the image of the product can be highlighted (Boxed) action.

(E) Gazing Area (Gazing Area) analysis module 425: Analyze the area of the gazing area of the image identified as "person" in the image data and its relative coordinate position. The gaze area analysis module 425 determines the direction in which the eyes are gazing by judging the head direction of others. Simulate a light source, and assume that the light source is located directly behind the head area of the person in the image data. Set the light source to emit light in the same direction as the head, and project the light on a virtual plane in front of the head. It is inferred that the area of the light projection is the area that the visitor may look at. Through such a simulation method, the projection (or gaze) area can be calculated, and the distance between the gaze area and the image capturing device 400 can also be converted. An image data may include one or more visitors, these visitors may (a) are moving and continue to look at a particular item, or (b) stay in a certain place in the environment and look at a particular item. The gaze area analysis module 425 can process the gaze behaviors of the various visitors presented in the image data, and can calculate (speculate) the area of each visitor's gaze. With the location of each item in the environment, it is possible to know the items that are located within the gaze area of the visitor, and these (this) items are most likely to be of interest to the visitor.

(F) Gazing Object analysis module 430: Analyze which object(s) the visitor is looking at in the image data. The gaze analysis module 430 is a combination of the output results of at least two of the head detection module 410, the visitor identification module 415, the product detection module 420, and the gaze area analysis module 425, resulting in further application. The gaze analysis module 430 not only analyzes what items the visitor is looking at in the image data, but also various data such as the number of gazes of a specific visitor, the time of each gaze, the gender of the visitor, the age of the visitor... write it down. In addition, the gaze analysis module 430 also combines and records other basic information such as date, time, location, temperature, humidity, etc.

(G) Intensity of Interest analysis module 435: Through various data provided by other modules, the interest analysis module 435 uses big data analysis methods to generate statistical information required by users. These statistical information can be used by users to adjust the placement of products, change sales strategies, or change advertising/promotional content to improve overall revenue and operating performance. For example, analyzing the record data of a certain period of time and multiple locations (shopping malls) can summarize "Which products are most likely to attract customers' attention", "Which products are less interesting to customers", and "Specific products cause Customer’s attention to age group, gender”, “the degree of linkage between various products and weather, day of the week, and specific customer levels”...etc.

(H) Moving Path analysis module 440: Tracking the moving path (trajectory) of visitors (specific groups) under specific conditions through a single or multiple image capturing devices 400. By analyzing its moving path and stagnation time, it can provide visitor behavior information for users to refer to. The user can adjust the location of the item/commodity or change the display method of the item/commodity based on the visitor’s behavior information to attract the attention of the visitor/customer. The visitors with the above-mentioned specific conditions can be set by the user, for example: women between 40-50 years old, male students in elementary school aged 10-12, men over 70 years old who use crutches...etc.

(I) Visitor attribute analysis module 445: The image analysis server 300 is allowed to perform deep learning in the training phase through the predefined data sets of various attributes in the database 450. After the deep learning is completed, the visitor attribute analysis module 445 is specifically responsible for judging various attributes of the visitor in the image data, such as gender, age, occupation, etc. As mentioned above, the present invention is based on the CNN architecture, through in-depth learning of data sets defined by various characters, plus analysis of visitors’ clothing characteristics, movement characteristics, categories of purchased goods and other information. Further cross-analysis and comparison, so that the visitor attribute analysis module 445 can improve the accuracy of the visitor attribute inference.

(J) Returning Customer analysis module 455: Through the historical visitor image data recorded in the database 450, the visitor identification module 415 can identify whether the visitor in the current image data has visited before. If it is determined to be a returning customer, the database 450 will record historical statistical information related to the returning customer, for example, historical shopping records. The aforementioned historical statistical information of return visitors can be combined with the information provided by the gaze item analysis module 430 on the number of times the visitor looks at different items, the gaze time, etc., and the return customer analysis module 455 can further infer the probability that the visitor will purchase an item again. Frequency is for users to refer to to make early preparations. For example, shopping malls can know most of the products that customers are interested in to prepare sufficient quantities in advance.

Please refer to FIG. 5, according to an embodiment of the present invention, FIG. 5 is a flowchart of the overall application concept of identifying people and objects in image data according to the present invention. The process is divided into two stages. The first is the training stage to help the image analysis application 460 in the image analysis server 300 do deep learning; then it enters the application stage, and the image analysis application 460 in the image analysis server 300 can perform various related tasks. Practical application of image detection and recognition.

The training phase in FIG. 5 includes step 510 and step 520.

In step 510, the data set (Data Set) of various human/object characteristics defined in the database 450 is used to help the image analysis server 300 to perform deep learning. According to an embodiment of the present invention, a data set may be multiple image data with various human/object characteristics and presenting various angles, for example, a series of related pictures with the same feature. The so-called data set in the present invention refers to the general term of multiple image data that conform to the same subject and have the subject feature; the subject can be, for example, All kinds of boys, girls, old people, children, clothes, pants, hats, plants, animals, cars, tools, food... etc. The image data of the same subject (subject) has features that conform to the subject, and may be presented under different "degrees of freedom of rotation in space". The so-called "degree of freedom of rotation in space", for example, includes three axial dimensions of roll, pitch (pitch), and yaw (Yaw), or it is called "degree of freedom of rotation in three-dimensional space" . The three dimensions of Roll, Pitch, and Yaw representing the "degree of freedom of rotation in space" will be recorded in the "Label" data of the image data. In addition, the tag data will also record the subject that defines the image data, such as boys, girls, old people, children, clothes, pants, hats, plants, animals, cars, tools, food... etc. Therefore, the "Label" of the image data records different values, which also represents the various states of the people/objects in the image data under different rotational degrees of freedom in the three-dimensional space. For example: an image data in a data set (Data Set) shows the state of "human head turns to the right (roll) 45, elevation (pitch) 0, oblique (yaw) 0", where " The head of a person, turning right 45, elevation angle 0, oblique angle 0" is the "Label" recorded in the image data. The image analysis server 300 uses data sets of various subjects to perform deep learning through a convolutional neural network (CNN) architecture. After deep learning training, the image analysis server 300 itself can obtain the "feature mapping" of the learning theme; in this way, the image analysis server 300 has the ability to recognize the image captured by the image capture device 400 What type of theme (person or thing) characteristics are present in the data. According to an embodiment of the present invention, the aforementioned data set (Data Set) defining various human/object characteristics is stored in a database (Data Base) 450 connected to the image analysis server 300 for image analysis during the training phase Application 460 is used. According to an embodiment of the present invention, the graphic files of the data set may conform to Joint Photographic Experts Group (JPEG), Bitmap (BMP), PC Paintbrush Exchange (PCX) , Graphics Interchange Format (GIF)...and other graphics files of any imaging standard.

In step 520, the deep learning result of step 510 is tested. After the deep learning process in step 510, the image analysis application 460 can learn the ability to recognize various "people/objects" in the image data through the obtained "feature mapping", but this recognition ability is The pros and cons still have to be tested. According to an embodiment of the present invention, the testing method may be to input test image data to verify whether the ability of the image analysis application 460 to recognize "people/objects" meets the requirements. Before passing the set evaluation criteria, the image analysis application 460 will repeat the deep learning process of step 510 to improve its ability to recognize various "people/objects" until the set evaluation criteria can be passed.

The above steps 510 and 520 represent the pre-training (training phase) before the image analysis application 460 is officially launched. The result of this pre-training is closely related to the recognition ability of the image analysis application 460 when it is actually online. If the training phase is done well, it means that the image analysis application 460 has better image recognition capabilities and can output good quality and correct results. Furthermore, after step 510 and step 520 are executed, the image analysis application 460 can learn to identify the "feature mapping" or "feature vector" of various "person/object" features in the image data. ". According to the learned "feature mapping", the image analysis application 460 does not need to compare the data set in the database (Data Base) 450 when it is actually online. The "feature mapping" is used to detect or identify the characteristics of people or objects in the image data.

The application phase in FIG. 5 includes step 530 to step 570.

Step 530 is that the image analysis application 460 recognizes the image data from the image capture device 400. Step 530 is one of the application phases. According to an embodiment of the present invention, if the image capturing device 400 is installed in a supermarket/sales environment, the image data captured by the image capturing device 400 may include parts of the face/head and body of the shopping customer, and at the same time It may also contain various items on the shelf. In step 530, the head detection module 410 and the product detection module 420 in the image analysis application 460 can detect which part of the image in the image data is a shopping customer (including the clothes worn by the shopping customer) , Which part of the image is the product on the shelf, etc. Basically, whether it is a shopping customer (face/head feature) or the image of the goods on the shelf, the pixels must be greater than a threshold value, for example: 40X40 pixels or more, but not limited to this specification, then Shopping customers (face/head features) or goods on the shelf can be detected and identified by the image analysis application 460. The above-mentioned "40X40 pixels" is only an example when explaining the threshold value of image quality. This threshold value may be any other value, such as: 50X50 pixels, 80X80 pixels, 100X100 pixels... etc., end view Required for practical applications. It can be seen that, according to the technology of the present invention, the shopping customers and the products on the shelf presented on one piece/frame of image data, no matter the number, as long as the respective image quality exceeds the threshold value, can be simultaneously used by the image analysis application 460 identified it.

In step 540, the identified data of various human/object characteristics is provided to other functional modules of the image analysis application 460 for further application. According to an embodiment of the present invention, the so-called person/object feature refers to a specific shape, color, size, or combination of the above that can be recognized as a person or object in the image data; in addition, the recognized person or object is also calculated in step 540 The position and distance of the object relative to the image capturing device 400. The above-mentioned further application refers to the use of various functional modules of the image analysis application 460, such as those listed in FIG. 4: the flow analysis module 405, the visitor identification module 415, the gaze area analysis module 425, and Item analysis module 430, interest degree analysis module 435, movement path analysis module 440, visitor attribute analysis module 445, return visitor analysis module 455... and other functional modules, which can analyze the data of various people/object characteristics after identification Do further processing and analysis to generate the data and/or information the user needs.

In step 550, the historical output results of each functional module are recorded in a database (Data Base) 450. The output result recorded in the database (Data Base) 450 is not limited to image data, but can also be, for example, related data including text, tables, database, etc., or a combination of image data and any related data. According to an embodiment of the present invention, when the technology of the present invention is applied to a supermarket or a store, the customers (Customer, which can be regarded as each ID), the products (Product Item) that are consumed or watched (interested) are recorded in the supermarket/store. , Customer's moving path, and shopping behavior (Behavior) are important basic elements provided for further analysis applications. In addition to the aforementioned records of customers, products, movement paths, and consumption behaviors, the image analysis application 460 can also automatically combine other reference information such as date/time, location, weather, temperature, humidity, festivals, advertising and promotion, etc., to form another A kind of reusable information.

In step 560, the historical output data of each functional module stored in the database (Data Base) 450 is then further statistically analyzed. Although only single-day records, single-person records, or single-item records can also provide some reference information; but if you combine all the data to do big data analysis, you can get the trends and relationships. Step 560 is to perform statistical analysis on the historical records output by each functional module to provide users with various useful reference information, such as: products that easily arouse customer interest, products that all customers pay attention to at first sight, and those that are not easy to arouse customer interest. The output of statistical information such as the influence of regions and promotional products on customers, the customer’s course of action, and the best display of products.

In step 570, the result of statistical analysis of the historical records output by each functional module in step 560 is performed, and suggestions are even provided based on the results, and the information is output for the user's reference. According to an embodiment of the present invention, the output information may be in the form of report information, graph information, narrative information, or a combination of any two or more of the above. For example, the output information may be, "a certain item attracts more attention from customers, among which the gender ratio of the customer, the age of the customer", "the degree of relevance between items that are easily attracted to customers and promotional activities", "customers The ratio of the favorite product at first glance to the actual purchase of this product", "Areas that is not easily noticed by customers", "The degree of connection between customers and promotional products", "Customer's moving route", "The best display of a specific product"... Wait. In addition, the output information can also provide users with suggestions for improvement, such as: "The frequency and quantity of increasing inventory for products that are noticed by customers", "Putting which products customers pay most attention to on shelves in which locations can increase sales "Quantity", "Based on the customer's moving route, it is recommended to arrange the position of the goods on the shelf", "Which are less popular products should be placed in other positions"... etc.

Please refer to FIG. 6, which is a flowchart of the training phase (step 510, step 520) in FIG. 5. The process of this training phase starts at step 610.

In step 610, the image analysis application 460 in the image analysis server 300 is based on the CNN architecture and uses the input data sets of various "people/object" characteristics to perform deep learning. According to an embodiment of the present invention, the image data of the data set (Data Set) can be defined as "female heads showing various angles" as defined in Figure 7A, or as defined in Figure 7B and Figure 7C as "various objects and "Food" image data. According to an embodiment of the present invention, the image data in FIG. 7A at least includes the female image data numbered from 1 to 91 in each "Head Direction". Each image data numbered 1-91 in Figure 7A has its label representing the main content and the three dimensions of the head direction (Roll), pitch (Pitch), and yaw (Yaw). )Information. The subject in the "tag" of the image data No. 1-91 is a woman, and the different values of Roll, Pitch, Yaw and other information in the tag represent that the woman’s face/head is in a three-dimensional space and is in different The different states presented under "Rotational Freedom in Space". The "different attitudes presented by the woman in different degrees of freedom of rotation in different spaces" in FIG. 7A can be referred to as the "head direction" of the woman. The different values of Roll, Pitch, and Yaw in the image data of each number above are part of the information of the "tag" of the image data of that number. In addition, the subject of women is also another part of the "label" of information. For example, the image data No. 46 (in the thick line box) is characterized by the face/head facing the front and the face/head in the image data has no elevation angle or offset (that is, no roll, pitch, yaw value). Therefore, for the image data No. 46, the values of Roll, Pitch, and Yaw are respectively defined as (0, 0, 0). Compared with the image data of No. 46, the elevation angle and offset of the face/head have different degrees of changes in the image data of other numbers other than No. 46 in Figure 7A, so the values of Roll, Pitch, and Yaw may be (-20, 10, 0), (5, -10, 30), (10, 20, -5)… and other different situations, and this value in the image data represents a part of its label, which is used to distinguish different faces/ The image data of the head direction formed by the roll, pitch, and yaw of the head. The above-mentioned (0, 0, 0), (-20, 10, 0), (5, -10, 30), (10, 20, -5) and other values do not necessarily represent actual angles, but may represent The degree of difference from the benchmark image data. After the image analysis application 460 in the image analysis server 300 undergoes the deep learning process in step 610, it can obtain the "feature mapping", or "feature vector", of the learning topic. To identify "people" or "things" with specific thematic characteristics.

The image data of the "Object and Food" data set (Data Set) in FIGS. 7B and 7C may be, for example, various commodities in the store, such as fruits, clothes, pants, beverages, food, etc. Similarly, in the Data Set of various products, each image data of each product will also have a label, indicating different values of Roll, Pitch, and Yaw, so as to present "the same product but different Image data under "Stylish".

In step 620, the test image data is used to verify the learning result of the image analysis application 460 in step 610. Although after the deep learning process of step 610, the image analysis application 460 should have a certain level of ability to recognize the characteristics of the learned subject, whether the success rate meets the requirements still needs to be verified by the test image data. The number of test image data may be "one-quarter" or "one-fifth" of the number of image data in the data set, but it is not limited to this. According to an embodiment of the present invention, the correctness of the test process may still require the intervention of the "rater (person)" to make the final judgment. To a certain extent. This verification process can be recorded for reference in future learning and actual identification of image data.

In step 630, it is determined whether the learning result of the image analysis application 460 meets a predetermined standard. For example, the criterion for this determination can be: the error rate of the test using test image data must be less than 10%; the above threshold of 10% is just an example and not limited to this. In practice, users can set any ideas. The required threshold. Taking 10% as an example, the meaning of an error rate of less than 10% is: assuming that there are 10 "persons" in the test image data, the image analysis application 460 must correctly identify at least 9 of them. Face/head area, and predict features such as Roll, Pitch, Yaw, etc. presented by the face/head. If it can reach the correct recognition rate of more than 90% of the test image data, it is considered qualified, otherwise it is not up to the standard. Similarly, when the test object is a "thing", the same inspection standard is also used. If the learning result of the image analysis application 460 reaches the predetermined standard, the training phase is ended. Otherwise, proceed to step 640.

In step 640, for some subjects that have not reached the standard after the test, input more relevant data sets (Data Set) to perform deep learning again. After step 640 is executed, it will return to step 620 to verify whether the learning result of the image analysis application 460 meets the predetermined standard by using the test image data. Repeated learning and verification in this way, until the results of the learning reach the predetermined standard.

According to another embodiment of the present invention, when the judgment result of step 630 is no, then no more relevant data sets are input in step 640 and deep learning is performed again, but another algorithm is selected. It is used as a tool to identify the characteristics of various subjects in various aspects. According to another embodiment of the present invention, when the judgment result of step 630 is negative, the learning parameters therein are changed/adjusted in step 640 so that the learning result can be closer to the expected result. The above-mentioned variables that reach the predetermined standard through changes can be achieved by: (a) inputting more image data related to the data set (Data Set); (b) choosing another algorithm; (c) adjusting the learning parameters. , And other methods to achieve the result of changing learning. The way to change the variable cause can be changed one variable at a time, or change multiple variables at the same time to change the result of learning.

Please refer to FIGS. 8A to 8C. FIGS. 8A to 8C illustrate step 620 in FIG. 6 to test the image to verify the learning result of the image analysis application 460. A frame of test image is a picture composed of many pixels, which contains a specific image that the user wants the image analysis application 460 to recognize. After deep learning training, the image analysis application 460 should be able to recognize the learned characteristics of the person or object in the picture of the test image. Generally speaking, in the following two situations, the image analysis application 460 should be able to successfully complete the identification: (a) the characteristic image of the person or object to be identified in the test image exceeds a certain pixel size, (b) mutual appearance When overlapping characteristic images of people or objects, the overlapping parts do not exceed a certain ratio, which affects the degree of recognition. As shown in Figures 8A to 8C, after being identified by the image analysis application 460, the position of the "person's head" in the picture of the test image is marked with a box of a certain color; The box marks the position of the specific object in the picture of the test image data. Although in the pictures of the test images in Figures 8A and 8B, the angle of the "human head" is different, as long as the feature image to be recognized exceeds a certain pixel size, the feature should be recognized come out. The overall recognition rate of the test image must reach a predetermined standard before the training phase can end.

Please refer to FIG. 9. FIG. 9 illustrates the detailed flow of step 540 in FIG. 5. According to an embodiment of the present invention, the process of FIG. 5 is described for one of the many applications of the present invention, "analyzing the products that are most likely to arouse the interest of shopping customers in the store". The flow of FIG. 5 starts at step 700.

In step 700, the image analysis server 300 obtains image data to be analyzed. According to an embodiment of the present invention, the method of obtaining image data may be that the image capture device (Image Capture Device) 400 first uploads the captured image data through the network 180 to the cloud data storage unit 150 for storage, and then The image analysis server 300 then downloads the image data from the cloud data storage unit 150 via the network 190 for further analysis. According to another embodiment of the present invention, the image data can also be directly transmitted from the image capture device (Image Capture Device) 400 to the image analysis server 300 via a network or a connection line. In the present invention, the image analysis server 300 for analyzing the image data is not limited to one, and there may be more than one. According to an embodiment of the present invention, a single image analysis server 300 is responsible for the image data analysis work. In this case, the process of FIG. 9 is all performed in the same image analysis server 300. In another case, if a plurality of image analysis servers 300 work together to complete the image analysis work, the process of FIG. 9 will be divided into several parts, and different image analysis servers 300 will each perform one part. For example, the image data is first sent to a first image analysis server 300 for partial image processing and/or analysis, and then the partially processed and/or analyzed image data is uploaded to the cloud data storage unit 150 for storage; In addition, a second image analysis server 300 can download image data from the cloud data storage unit 150 for further processing and analysis at any time. The above-mentioned first image analysis server 300 and second image analysis server 300 may both be image analysis servers 300 with complete functions, or two image analysis servers 300 or one of them may only have partial image processing and analysis functions . According to an embodiment of the present invention, after the image analysis server 300 receives the image data, it must first determine whether the file format of the image data is an acceptable file format, if not, the image data must first undergo a file format conversion step Follow-up processing will be carried out later.

In step 705, the part of the image data with a subject is identified; the so-called "subject" is a subject with specific characteristics that can be identified by the image analysis server 300 after deep learning. Each picture of the image data may contain one or more objects with specific characteristics. These objects may be objects with the characteristics of "face or head" or objects with the characteristics of "articles or commodities". . According to an embodiment of the present invention, since each picture in the image data is composed of a large number of pixels, step 705 must determine which "person" feature objects and which "thing" feature objects are included in the image. In addition, It is also necessary to know where each object is located in the picture; in other words, determine each object with the characteristics of "people" and each object with the characteristics of "thing" in the picture, as well as the individual occupancy in the picture The range (pixel size) and the position in the picture are the main purposes of performing step 705. Generally speaking, although the dynamic specification of image data is composed of pictures from a few to dozens of pictures per second per second, according to an embodiment of the present invention, it is not necessary to detect every picture in practice. It includes how many objects with the characteristics of "people" and objects with the characteristics of "things". If you take the most common 30 frame/second (second) specification as an example, you can choose, but not limited to this, select a frame (picture) at every 1/6 second interval to detect Measure how many objects with the characteristics of "people" and objects with the characteristics of "things" are included in the picture, as well as their respective positions and extents in the picture. In other words, under this assumption, step 705 must determine the subject of the "person" feature and the subject of the "thing" feature of the 6 pictures every 1 second. The foregoing detection of a frame of pictures every 1/6 second is only an embodiment of the present invention, in fact, the time interval can be any other numerical time interval. It must be noted that the detection of "people" and/or "objects" in the image in step 705 is not achieved through the comparison of the Data Base, but by the "features" learned in the deep learning of the training process. "Feature mapping" to perform the task of identifying the "person" and/or "thing" in the picture.

After the image data is identified in step 705, the identified objects of "person" and "thing" can be divided into two parallel processes of "customer" and "commodity" for processing. Therefore, after step 705, the object identified as "person (customer)" proceeds to step 710 and the subsequent steps are executed; the object identified as "thing (commodity)" proceeds to step 715 and the subsequent steps are executed.

Continuing from the above description, step 710 is the step where the subject identified as "person (customer)" in the picture enters. Since the range of the face/head shown in the picture is large or small, if there is a small area (pixel) of the face/head in the picture, it may not be effectively detected. In an embodiment of the present invention, the pixel size that can be detected is more than 40×40 pixels (inclusive). The threshold value of 40X40 pixels is only an implementation reference provided by the present invention, and the present invention is not limited to this. The lowest recognizable pixel threshold value may be any other value of pixel size, depending on the hardware and software. Depends on the ability. After the subject identified as "person" is determined in step 710, according to an embodiment of the present invention, if the customer's identity (member) data does not need to be maintained, proceed to step 730 of "detection of the direction of the customer's head" and step 735 The "Estimation of Customer's Head Position" and other two steps, these two steps will be explained later. According to another embodiment of the present invention, if the customer's identity (member) information needs to be maintained, step 720 is entered.

In step 720, identity recognition (Identity Recognition) is performed on the customer in the picture. According to an embodiment of the present invention, the identities of all customers in the picture are compared one by one. People in the store may walk around at random to look around at the merchandise, or they may pause and focus on certain merchandise. According to an embodiment of the present invention, the so-called customers refer to those who have stopped and the length of time they watch an item (commodity) exceeds a time threshold; the set time threshold may be, for example: 2 seconds, 5 seconds, 10 seconds, but not limited to this, can be any length of time set. Under this setting, the person in the picture who does not meet the above conditions will not be recognized as a customer. According to another embodiment of the present invention, for people who are moving in the picture, but their gaze continues on the same item (commodity) for more than a set time (for example, more than 2 seconds, but not limited to this), Will be identified as customers and further identify their identity.

In addition, step 720 needs to compare the data of the database (Data Base) 450 in FIG. 2 to achieve the purpose of identifying the customer. Among them, the data of the database (Data Base) 450 includes the ID of the customer who has visited and The image data of the "facial features" of each visiting customer. The image data of the facial features includes image data of different head directions. Based on the image data of the "facial characteristics" of each customer recorded in the database 450, the image analysis application 460 can determine whether the recorded "facial characteristics" match the customer's "facial characteristics" in the current picture. To identify the identity of the customer. The aforementioned customer ID may be the customer's member ID or a temporary ID assigned through the image analysis application 460. Temporary ID may be, for example: 20200101-F0000, a combination of numbers composed of data such as date and shopping area. When the customer identity comparison is successful, it means that the customer has visited and is recorded in the database (Data Base) 450. At this time, there is no need to add the customer's information in the database (Data Base) 450. Conversely, when the customer identity comparison is unsuccessful, it means that the customer has not been recorded in the Data Base 450, or the quality of the customer’s head image data is not good enough for the comparison to proceed smoothly. At this time, a temporary customer ID can be added to the database (Data Base) 450 to represent the identity of the customer.

In step 725, if the data of the customer's facial features cannot be compared in the database (Data Base) 450 in step 720, add the customer ID and record the image data of the facial features in the database 450, and add The related record of the product that it looks at; if step 720 can compare the data of the customer’s facial features in the Data Base 450, then compare the ID of the customer to add the related record of the product that they look at. In addition, it can also be based on The setting determines whether to add the image data of the customer's facial features to the database (Data Base) 450. In one embodiment of the present invention, in step 725, in addition to recording the customer ID and the image data of the customer's facial features, the related information of the watched product is also recorded, including: date/time, the name/code of the watched product, and the watched product. The duration of the product, the number of times the product was looked at, etc.

According to another embodiment of the present invention, steps 720 and 725 in FIG. 9 are omitted to improve the overall performance of the image data acquisition and analysis system 100. In this case, the step of "compare with the database (Data Base) 450 to identify the customer ID" in step 720 is not performed, and each customer in the picture is treated as a new customer; in this case, also The step of "recording the image data of the customer's face/head and the product they watched in the database (Data Base) 450" in step 725 will not be performed. The relevant customer data will only be temporarily stored in the dynamic random access memory (DRAM) 320, non-volatile memory (NVRAM) 330, or other suitable storage space in the image analysis server 300, and will not be stored permanently .

According to another embodiment of the present invention, in addition to the "identification of the customer in the picture" in step 720, the attribute data for identifying the customer is further included, such as gender, age, occupation, etc., but not Is limited. Please refer to the part shown by the dotted line in FIG. 9. If the identification of the customer's gender, age, and occupation is taken as an example, after step 720, three steps such as step 722A, step 722B, and step 722C can be performed. These 3 steps are all shown in dashed lines, which means that these 3 steps can be selectively executed or omitted. In practical applications, performing more identification of customer attributes is likely to reduce the overall performance of the system 100, so users can decide whether to increase the attributes of identifying customers according to their needs.

In step 722A, gender identification and recording are performed on the customer in the picture. As mentioned above, the present invention is based on the CNN architecture, after using gender-related data sets (Data Set) for deep learning, the image analysis application 460 learns to obtain the "feature mapping" that identifies the subject as gender. To identify whether the customer in the picture is a young woman, an older woman, a young man, an older man, a girl or a boy, or a child, etc. When the gender of the customer is identified, the customer's gender data will be added to the gender field of the database 450. As mentioned above, step 722A is drawn with a dashed line, which means that step 722A can be omitted.

In step 722B, the department estimates and records the age of the customer in the picture. As mentioned above, the present invention is based on the CNN architecture. After deep learning is performed using the age-related data set (Data Set), the image analysis application 460 learns to obtain the "feature mapping" whose subject is the age. Estimate the age of the customer in the picture. In the training phase, the more data sets (Data Sets) of male/female/children of various ages provided by the database 450 for deep learning, the more accurate the customer age estimated by the image analysis application 460 in step 722B will be . When the customer's age is estimated, the customer's age field in the database 450 will be added with its age data. As described above, step 722B is drawn with a dashed line, which means that step 722B can be omitted.

In step 722C, the department estimates and records the career of the customer in the picture. As mentioned above, the present invention is based on the CNN architecture. After deep learning is done with the occupation-related data set (Data Set), the image analysis application 460 learns to obtain the "feature mapping" that recognizes the subject as the occupation, which can be based on Estimate the occupation of the customer in the picture. In the training phase, the database 450 provides more data sets of clothing worn by various occupations or tools used in work for deep learning, which is estimated by the image analysis application 460 in step 722C The career of the exported customer will be closer to the truth. When the customer's occupation is estimated, the customer's occupation data will also be added to the customer's occupation field in the database 450. As mentioned above, step 722C is drawn with a dashed line, which means that step 722C can be omitted.

In step 730, the head direction of the customer in the picture is detected. Following the description of step 710 above, after step 710 is executed, the "detection of the customer's head orientation" in step 730 and the "estimation of the customer's head position" in step 735 will be executed in parallel processing. According to an embodiment of the present invention, before step 730 is performed, the data set (DataSet) shown in FIG. 7A must be used in the training phase of FIG. The "feature mapping" of the direction, and then the "feature mapping" of the head direction to determine the most likely pattern of the customer's head direction in the picture. Take Figure 7A as an example. The pictures numbered 1-91 show images of the same woman with different head orientations. Each image has a corresponding label, and the information recorded in each label At least include "subject" (e.g. female head), and "subject's roll, pitch, and yaw three values". From the above, it can be seen that the pictures No. 1-91 show a female face/head in different "head directions". In actual application, the direction of the customer’s head in the picture (that is, the degree of freedom of turning in the space formed by the head roll (Roll), pitch (Pitch), yaw (Yaw), etc.) can roughly correspond to the learning The number 1-91 is one of the conditions, so the image analysis application 460 can use the "feature mapping" obtained by deep learning in the past training phase to determine the roll corresponding to the direction of the customer’s head in the picture. Pitch, Yaw and other values. If the customer’s head orientation in the picture does not completely match one of the above numbers 1-91, in step 730, the customer’s head orientation will also be optimized according to the learned "feature mapping" Close judgment, estimate its most likely Roll, Pitch, Yaw and other values. In addition, according to an embodiment of the present invention, step 730 can further calculate the position and range of the customer's head in the picture, and distinguish (mark) the part of the customer's face/head in the picture. In the case of identifiable, if a picture includes 10 customers, the head orientation of 10 customers will be detected, and further the faces/heads of the 10 customers will also be marked in the picture. The face/head of the customer is marked with a bold colored frame in Fig. 8A or Fig. 8B.

In step 735, the position of the customer's face/head in the picture in the environment (store) is estimated. In the present invention, before executing the application process shown in FIG. 9, the image capturing device 400 must be calibrated to obtain the relevant parameters of the image capturing device 400 (for example, Intrinsic/Extrinsic matrix). After learning and training, step 735 can calculate the distance and relative position between the customer in the picture and the image capturing device 400 from the parameters of the image capturing device 400 and the image data of the customer in the picture. According to an embodiment of the present invention, the relative distance between the specific object and the image capturing device 400 can be regarded as the distance between two points in space. Generally, the image capturing device 400 can be set as the origin, so the coordinates of the image capturing device 400 in the 3D space are expressed as (0, 0, 0). If the coordinates of the specific object in the 3D space can be calculated, for example: (100, 552, 211), then the distance of the specific object relative to the origin (the image capturing device 400) is a calculable length. According to another embodiment of the present invention, the origin in the 3D space can also be set as the triangle intersection point between two adjacent walls of the store in the picture and the ground. In summary, the present invention first performs calibration on the image capturing device 400 to know the relative distance of a specific point in the 2D picture to the image capturing device 400 or a reference point (the set origin) in 3D space . Accordingly, in step 735, the calibrated image capturing device 400 can be used to estimate the position of the customer's head in the 3D space.

In step 740, the Gazing Area of the customer is estimated and the probability that each part of the Gazing Area may be the customer's line of sight is estimated. According to an embodiment of the present invention, after the detection of the customer's head orientation in step 730 and the estimation of the customer's head position in step 735, the "head orientation" (roll, pitch, yaw three values) and the position of the customer’s head in the three-dimensional space. Through these two kinds of information, the following simulations are carried out: a virtual light source is set directly behind the customer's head, and the virtual light source projects a light in a direction directly in front of the head, and sets the projected light to extend a set distance, Or it extends until there is an object (commodity), and then at the set distance or position of the object (commodity), suppose there is a simulated screen and the projected light forms a simulated projection on it. The projection area formed by the simulated projection is the possible gaze area of the customer, such as the elliptical projection areas 815A and 815B as shown in FIGS. 11A and 11B. According to an embodiment of the present invention, the projection area is further divided into a plurality of sub-projection areas, where the sub-projection area with the highest probability is the most likely focus area of the customer’s line of sight. The range of interest is the sub-projection area 820 in FIG. 11B. Taking FIG. 11B as an example, the percentage value displayed in the sub-projection area 820 is 95%, which represents a 95% probability that the customer may look at the sub-projection area 820. In addition, the other sub-projection areas 830 and 840 radiating outward from the center of the sub-projection area 820 have different probability values, respectively 65% and 15%, which also means that customers may look at the sub-projection area. Probability. Generally speaking, people should mostly look straight ahead and have more opportunities to stare at objects directly in front of them. Based on this assumption, according to an embodiment of the present invention, it is set to divide a plurality of concentric sub-projection areas with the center point of the elliptical projection area as the reference point. The probability is smaller, as shown in Figure 11B. According to another embodiment of the present invention, considering that people’s eyeballs will rotate, their line of sight may not be looking straight ahead. Therefore, the reference point for dividing each sub-projection area is set to deviate from the center of the projection area, or it may be left, right, or up. , Or lower and so on. In step 740, each customer in the picture has its own projection area according to its different head orientation, and each customer's respective projection area represents the gaze area of each customer.

跟一第三向量

。最後，依據前述之第二向量

跟第三向量

在屏幕810上之交點，據以形成一模擬投影區域（即為投影區域820）。Please refer to FIG. 11C, according to another embodiment of the present invention, suppose a customer turns to the left at an angle θ to look in a direction (3), using the image capturing device 400 in step 735 to set the origin O to calculate the customer's head The distance between the center point H of the part and the origin is the left-turn angle θ in the direction of the customer’s head. Next, a Gazing Point G is calculated according to a rotation matrix formula, and then the center point H of the customer's head is calculated in the direction of a first vector formed by the gazing point G and the center point H of the customer's head The proper position P directly behind and from the center point H is the Fictitious Projection Point P of a virtual light source; in an embodiment of the present invention, the proper position P may be about 3 behind the head. The head is twice the diameter of the sphere. At this time, use a formula (xH _x ) ² +(yH _y ) ² =r ^{2 to} simulate the outer shape of the customer’s head, where (H _x , H _y ) represents the center point of the head simulation sphere, and r represents the head simulation sphere Then, calculate the virtual projection starting point P through the equation of all points and the two tangent points L and R between the outer shape of the customer’s head simulated by the _{equation (xH x} ) ² +(yH _y ) ² =r ^2, Thus a two vector is formed, that is, a second vector is formed by the tangent point L and the virtual projection starting point P, and the tangent point R and the virtual projection starting point P

Followed by a third vector

. Finally, according to the aforementioned second vector

With the third vector

At the intersection on the screen 810, a simulated projection area (that is, the projection area 820) is formed accordingly.

According to another embodiment of the present invention, the required number of tangent points can be calculated according to requirements to form a corresponding number of vectors, and then the intersection points of the aforementioned vectors on the screen 810 can be obtained, and an area (that is, the projection area 820) can be formed accordingly .

跟一第六向量

。最後，依據前述之第二向量

跟第三向量

在一第一虛擬平面（屏幕810）上之交點，據以形成一模擬投影區域（即為投影區域820）。Please refer to FIG. 11D, according to another embodiment of the present invention, suppose that a customer's head rolls to the right at an angle α, an upward pitch angle β, and a right yaw angle γ to look in one direction (4), using the image in step 735 The capturing device 400 is set as the origin O, and the distance between the center point H of the customer's head and the origin is calculated and the head rotation angle in the direction (direction (4)) directly in front of the customer's head is calculated. Then, a Gazing Point G is calculated according to a rotation matrix formula, and then the center point H of the customer's head is calculated in the direction of a fourth vector formed by the gazing point G and the center point H of the customer's head Right behind and at a position r from the center point H, is a virtual projection starting point (Fictitious Projection Point) P; the r position, in an embodiment of the present invention, may be the back of the head at the diameter of the simulated sphere of the head 3.5 times the value. At this time, a formula (xH _x ) ² +(yH _y ) ² +(zH _z ) ² =r ^{2 is used to} simulate the outer shape of the customer’s head, and the virtual projection starting point P and the equation (xH _x ) ² +(yH _y ) ² +(zH _z ) ² = ^{the two tangent points A and B between the peripheral shape of the customer's head simulated by r 2} to form a two vector, that is, from the tangent point A and the virtual projection point P , A fifth vector formed by the tangent point B and the virtual projection point P

With a sixth vector

. Finally, according to the aforementioned second vector

With the third vector

At the intersection on a first virtual plane (screen 810), a simulated projection area (that is, the projection area 820) is formed accordingly.

跟該第六向量

）來模擬一圓錐狀的模擬投影照射區域，並在一或多個虛擬平面（例如圖11D之該第一虛擬平面（屏幕810）、一第二虛擬平面（屏幕812））形成該訪客之一或多個模擬投影區域（例如圖11D之該投影區域820、一投影區域822）。若該物品（標的）的部份或全部為該圓錐狀模擬投影照射區域內，則表示該物品（標的）為該顧客所關注的標的，其中，若該物品（標的）的位置越靠近中心線（例如圖11C中之由點H與點O所形成一第一中心線及由點H與點G所形成一第二中心線），則表示該顧客不僅關注該物品（商品）且關注該物品（商品）的程度越高。According to another embodiment of the present invention, referring to research in the field of Human Factors Engineering, the visual fields of the human eyes are about 120˚ in the vertical direction when the head is stationary and not rotating. The field of view of 140˚ and the field of view of about 150˚ in the horizontal direction are about 60˚ and 90˚ of the field of vision shared by both eyes in the vertical and horizontal directions, respectively. If you want to look at an object (target) more closely (which also means that the degree of concentration on the object (target) is higher), the angle of the field of view will inevitably be narrower. Therefore, in an embodiment of the present invention, one or more virtual projection lines formed by an angle of 30° in the vertical direction and an angle of 30° in the horizontal direction are used to define the common field of view of the visitor’s eyes, wherein the one or more The virtual projection line extends backward and meets at a point (a virtual projection point P, as shown in Figure 11C) at about 6r (r is the radius of the head) behind the head of the visitor. From the point (the virtual projection point P) along the one or more virtual lines (for example, the fifth vector shown in FIG. 11C)

With the sixth vector

) To simulate a cone-shaped simulated projection irradiation area, and one or more virtual planes (for example, the first virtual plane (screen 810) and a second virtual plane (screen 812) in FIG. 11D) form one of the visitors Or multiple simulated projection areas (for example, the projection area 820 and a projection area 822 in FIG. 11D). If part or all of the item (target) is within the conical simulated projection irradiation area, it means that the item (target) is the target that the customer pays attention to, and if the position of the item (target) is closer to the center line (For example, in Figure 11C, a first centerline formed by points H and O and a second centerline formed by points H and G), it means that the customer not only pays attention to the item (commodity) but also pays attention to the item (Commodity) the higher the degree.

。According to another embodiment of the present invention, the rotation matrix formula may be a two-dimensional rotation matrix formula as follows:

.

。According to another embodiment of the present invention, the rotation matrix formula may be a three-dimensional rotation matrix formula as follows:

.

In the above description, according to an embodiment of the present invention, only when customers stop and the length of time they watch an item (object) exceeds a set time threshold, will they be inferred for their Gazing Area (Gazing Area) ; The set time may be, for example: 2 seconds, but not limited to this, it can be any set time length. According to another embodiment of the present invention, if the customer is moving and his gaze continues on the same item (subject) for more than a set time threshold (for example, more than 2 seconds, but not limited to this), Will speculate on their Gazing Area (Gazing Area). Except for the situation where the head direction of the customer cannot be recognized, each customer in the picture should be able to estimate the probability of being looked at in its gaze area (projection area) and its sub-projection area. The reasons for the above "unrecognizable head direction of the customer" may be: (1) The head image of a customer in the picture is incomplete, for example: the head image of the customer has a large area blocked by other customers or objects; (2) ) The pixels contained in the image range of a customer's head in the picture are too small, so that the customer's head cannot be identified. According to the technology disclosed in the present invention, even if the head image of the customer in the picture does not include the eyes of the face (for example, the head image on the back side), the head direction of the customer can still be recognized. According to another embodiment of the present invention, when the customer’s head image in the picture clearly shows the eye characteristics of the face, the customer’s line of sight direction can be calculated and referenced, and the projection area can be further restricted to increase the estimated customer The correctness of the gaze area. The constricted gaze area after calculating the gaze direction may be as shown in 820E in FIG. 11B. After step 740 is executed, jump to step 750 for execution.

Returning to the above step 705, when step 705 recognizes the part of the specific picture of the image data that has a specific feature as a "thing (commodity)", then step 715 is entered.

In step 715, it is a step of entering the subject identified as "thing (commodity)" in the picture. The present invention is based on the CNN architecture. During the training phase, deep learning is performed through various "things (commodity)" data sets (Data Sets) in the database 450. After that, the image analysis application 460 can learn the topics as various "things (commodities)". )" "feature mapping (feature mapping)", so in step 715, the subject of the "thing (commodity)" in the picture (frame) can be identified. Except for the unrecognizable products, all the products in the picture should be identifiable. The reasons for the above "unidentifiable product" may be: (1) the image data of a product in the picture is incomplete, for example: the image range of the product is largely blocked by other customers or items; (2) the product in the picture The pixels contained in the image range of is too small, making the product unrecognizable. Since the image ranges of the products contained in the pictures vary in size, products with a too small image range may not be effectively identified. In an embodiment of the present invention, the pixel size at which the product can be successfully detected and recognized is 40×40 pixels (inclusive) or more. The threshold value of 40X40 pixels is only an implementation reference provided by the present invention, and the present invention is not limited to this. In fact, the lowest recognizable pixel threshold value may be the pixel size of any other value. Depending on the capabilities of the hardware. Generally speaking, there are many types of commodities in shopping malls. In order to correctly identify each commodity, in the training phase, the data set of each commodity must be provided to the image analysis application 460 for deep learning. . In an embodiment of the present invention, in the Data Set that defines each product, the image data also presents "the status of the product under different rotational degrees of freedom in a three-dimensional space", as shown in Figure 7B and Figure 7B. As shown in 7C; even in the label (Label) of the image data, the values of the three rotation angles of roll (Roll), pitch (Pitch) and yaw (Yaw) representing the "degree of freedom of rotation in space" are recorded. Step 715 is executed and then step 745 is executed.

According to another embodiment of the present invention, step 715 may not be executed, that is, no detection is performed on the part of the image data related to the product. For the product-related part, a method of inputting product data is adopted for reference and comparison by the image analysis application 460, the details of which will be explained later. In this embodiment, during the training phase, the data set of the product may not be provided for the image analysis application 460 to perform deep learning.

In step 745, the position of the product in the environment (store) is estimated or obtained. According to an embodiment of the present invention, in step 745, the position of the product is estimated by identifying the product in the image data (picture). For the same reason as step 735, before converting the position of the specific point (object) in the 2D picture in the 3D space, the image capturing device 400 needs to be calibrated. Therefore, according to an embodiment of the present invention, the position of a product in the picture estimated in step 745 can be represented by 3D spatial coordinates, for example: (200, 20, 525); or represented by 2D planar coordinates, for example : (550, 200).

According to another embodiment of the present invention, in step 745, the "placement position of each product in the environment (sales) where the image capture device 400 is located" can be obtained by referring to the input product information. The above-mentioned "entered product information" refers to: inputting the placement position of each product that has been set in the database 450 in advance, for example: which space point/range in the environment (sales) where each product is placed, Or which shelf in which position in the store... etc. In this case, the image analysis application 460 does not need to identify which products are in the image data, nor does it need to identify the products in the image data to estimate its location. When it is necessary to know which products are in the customer's gaze direction (or within the range of interest), the purpose can be achieved by referring to the product location data entered in the database 450 in advance.

In addition, it must be noted that in steps 735 and 745 of the present invention, for the "estimation of the position of the customer's head" and the "estimation of the position of the product", no matter which position in the 3D space is to be calculated, there is no need to At the same time, through the two camera lenses (Lens) of the two image capture devices 400, this function can be achieved by only one camera lens inside the image capture device 400. Although a general 3D video camera is only a single video camera in appearance, it contains at least two or more camera lenses inside. Therefore, the technology of the present invention does not need to rely on multiple lenses of a 3D image camera, and only needs to use one lens of an ordinary image capture device 400 to calculate the position of each point (representing a person or an object) in the 2D image in the environment. 3D position.

In step 750, the information of the "customer gaze area and its probability" in step 740 and the "product location" information in step 745 are aggregated to estimate which products in the picture are covered by the customer's gaze area and the probability of obtaining customer attention (interest). As shown in the flowchart of FIG. 9, the results after the execution of step 745 and step 740 are all output to step 750. After step 740 is executed, it is possible to know the possible coverage of the customer's gaze area in the picture and the probability that the gaze area is the customer's line of sight; after step 745 is executed, it is possible to know the position of the goods in the environment (sales) in the 3D space . Therefore, based on the information provided in steps 740 and 745, step 750 can infer which products are covered by the customer's gaze area, that is, "which products the customer in the picture is looking at". Among them, the customer's gaze area itself is further divided into various sub-areas with different probabilities. Therefore, the above "which products the customer is looking at in the picture" can be expressed by "what products are covered by the sub-regions of the customer's gaze area and the probability of being looked at", or you can choose to "fall in the customer's gaze area the highest Probability of products in one or more sub-regions". If the customer's gaze area or its sub-areas covers 2 or more types of products, these products will all be included. According to another embodiment of the present invention, the product that each customer looks at in the picture is not represented by the item (category) of the product, but is represented by the plane coordinates of the product in the picture, or the product in the picture. It is expressed by coordinates in space.

In step 755, the above information about customers and products is recorded in the database 450, and a visualization effect is generated on the picture. According to an embodiment of the present invention, the so-called "visual effect" herein refers to displaying the identified customer ID and the product they are looking at in the picture with words, numbers or symbols. According to another embodiment of the present invention, the visual effect of the product in the picture can be displayed with the "coordinate value" of the product instead of words, numbers, or symbols. In addition, the position of the customer's face/head and the position of the product in the picture can also be added with a box (or other shapes) to highlight the visual effect, as shown in FIG. 8B. According to another embodiment of the present invention, the so-called "visual effect" here is in addition to displaying the identified customer ID (ID) and the product they are looking at in the picture in words, numbers or symbols, but also The position of the customer's face/head and the position of the product in the picture are achieved by "enhancing the brightness of the pixels in these position areas". According to another embodiment of the present invention, the so-called "visualization effect" here, in addition to displaying the identified customer ID and the product they are looking at, in the original picture in words, numbers or symbols, it is more "In the picture, except for the position of the customer's face/head and the position of the product, the pixels in other areas are reduced in brightness (darkening)"; while the position of the customer's face/head and the position of the product in the picture are The brightness can be maintained as normal. Also, please refer to Figure 8C. The various products in Figure 8C are not all placed in a completely frontal manner, and some products have a certain angle of rotation. However, after using the Data Set in the database 450 for deep learning in the training phase, the image analysis application 460 in step 755 can still identify each product and mark it with a colored box, and display it in the picture Add text or symbols representing the name of the product next to the product.

Please refer to FIG. 10. FIG. 10 shows an example of recording customers and their products in step 755. In step 755, the customer-related data, product-related data, and the time that the customer continues to look at the product obtained in the previous steps are recorded (stored) in the database 450. According to an embodiment of the present invention, the various related information recorded in the database 450 includes data in each field shown in FIG. 10. The table in Figure 10 contains multiple items. Each item records the identity (ID) of a customer and its related attributes, such as gender, age, and/or occupation; and records the customer's attention (interested) Product related information, such as: "product name", "product location", "continued attention time", "attention level"... etc.; in addition, the "date/time" will also be recorded for reference. Among them, "continuous attention time" is the duration of time a customer pays attention to a certain product. Basically, the duration should be greater than a set minimum gaze time length, for example: 2 seconds, but not limited to this. "Product location" refers to the actual location of the product in the environment (sales). It can be represented by three-dimensional coordinates in 3D space (as shown in Figure 10), or represented by a pre-defined code (such as 123-456). Represents the 456 shelf in aisle 123). According to an embodiment of the present invention, the location of a product can be used to assist in confirming the name of the product. The customer's attribute data such as "age", "sex", and occupation (not shown in Figure 10) should be recorded (stored) in the database 450 according to the actual situation. If the functional modules related to this type of identification data are not enabled when identifying the customer ID, the data of "age" and "gender" in Figure 10 can be omitted and not recorded. The data of "attention level" refers to the intensity of the product's interest in the customer (Intensity), which can be a level represented by a value, for example: 1 to 5 represent five levels from weak to strong, but not based on this Is limited. The "attention level" can be related to the "continuous attention time". For example, the larger the value of the "sustained attention time", the higher the degree of customer interest the product arouses; in addition, if the product is located in the customer's gaze area with a higher probability Area, will also increase the "attention level." When recording customer ID data, if the image analysis application 460 cannot find the corresponding customer data from the database 450, it will replace it with a temporary code, such as the "F0103" data in FIG. 10. In this case, in addition to using a temporary code as the customer’s ID record in step 755, it is also necessary to add the customer’s face/head image data to the database 450 to provide the same customer again next time Used to identify their identity when shopping at the store. According to another embodiment of the present invention, the spatial coordinates in the "product location" in FIG. 10 can also be replaced by plane coordinate data.

In step 760, various relevant data obtained in the above steps are updated to the database 450, and the record content is accumulated for statistical analysis. The amount of data for statistical analysis can be a period of time (for example: 1 hour or 1 day) of records, such as the chart shown in Figure 10; or a longer period of accumulated data (for example: one week, one month, one quarter, Or one year). The statistical analysis results of different amounts of data have their own meanings, depending on the needs of users. Statistical analysis items can be, for example: "Which products in the store are likely to arouse the interest of shopping customers", "Gender and age group of certain products that arouse the interest of shopping customers", "Products that are least likely to arouse the interest of shopping customers" , "Where is a product placed in the store most likely to arouse the interest of shopping customers", "Which color of the same product but in different colors is likely to arouse the interest of shopping customers"... etc. are available for users to use Information.

In step 765, based on the statistical analysis result in step 760, an analysis report and/or suggestions are generated to the customer. The generated analysis report may include image data after adding visualization effects, useful information obtained after analyzing the data, suggestions after analysis, and so on. Among them, the suggestions after analysis can be, for example: "a certain product's replenishment quantity and estimated replenishment time", "a certain product that is not favored by customers should be replaced in a certain area for display", "a certain product The timing of the promotion with its price, the “best placement of a certain product”... etc. are useful suggestions for customers that can help sales based on big data analysis.

The execution of the "output analysis report and recommendations" in step 765 may be omitted due to the particularity of each industry or based on the actual needs of customers. After step 765 is executed, the flow of FIG. 9 is also ended.

Please refer to FIG. 11A. FIG. 11A is a schematic diagram of the present invention referring to the head direction (Head Direction) to estimate the area that the customer may look at (interested) by simulating the projection light. Figure 11A includes two schematic diagrams of heads 800A and 800B showing different angles. The head direction 1 (Head Direction 1) of the head 800A is represented by (Roll1, Pitch1, Yaw1) to represent the three in the space. Rotation angles. The three angle values in (Roll1, Pitch1, Yaw1) can represent the deflection of the head direction 1 (Head Direction 1) in the three-dimensional space. Similarly, the head direction 2 (Head Direction 2) of the head 800B is represented by (Roll2, Pitch2, Yaw2) to represent the three rotation angles in the space. The three angle values in (Roll2, Pitch2, Yaw2) can represent the deflection of the head direction 2 (Head Direction 2) in the three-dimensional space. According to an embodiment of the present invention, in the present invention, the area that the customer may look at (interested) is estimated by simulating the projected light, which is described as follows. Assume that there is a light source (Light Source) directly behind the heads 800A and 800B in the three-dimensional space, and the light is projected toward the front of the head with the head direction 1 (Head Direction 1) and the head direction 2 (Head Direction 2) respectively. This simulated light source projects light from directly behind the head to the front of the head, and the simulated projected light forms a cone or cylinder in a three-dimensional space. When the simulated light is projected forward to the place where the object or commodity is located, it is assumed that there is a simulated screen (Screen) 810 there, and the projection of the cone or cylinder of the simulated light on the simulated screen 810 (Projection) ) Is the area that customers may look at (interested). As shown in FIG. 11A, after light is projected in different head directions (Head Direction 1 and Head Direction 2), different projections 815A and 815B are respectively generated on the simulated screen 810 at the same place. The projection 815A and the projection 815B are the calculated gaze areas of the head 800A and the head 800B, and the items located within the area of the projection 815A and the projection 815B can be presumed to be items that the customer is watching or interested. (commodity).

Under normal circumstances, people seldom look at objects with the peripheral light of their eyes. In most cases, they look at objects with a line of sight similar to the direction of their heads. When the visual field area of the eyes is set to be the same width as the head, as shown by the dashed lines 805C and 805D in Fig. 11B, the estimated possible fixation area will be very limited, as shown in the area range 820 in Fig. 11B ; Conversely, when the visual field area of the eye is expanded to the area where the peripheral light of the eye can reach, as shown in the visual field area represented by the dashed lines 805A and 805B in Fig. 11B, the estimated possible fixation area will become much larger, as shown in Fig. Area range 815A in 11B. Furthermore, the projections 815A and 815B shown in FIG. 11A may be any of the elliptical regions 820, 830, and 840 shown in FIG. 11B. According to the technology of the present invention, the location of the customer (head 800A and 800B) in the three-dimensional space can be calculated, and the distance from the customer (head 800A and 800B) to the virtual screen 810 (that is, where the item is located) can also be calculated. The inferred value, plus the "feature mapping" obtained by deep learning to determine the angles (Head Direction 1 and Head Direction 2) of the customer's head 800A and 800B in the three-dimensional space (Head Direction 1 and Head Direction 2) After inputting various data into the image analysis application 460, the area (projection 815A and projection 815B) that the customer may look at (interested) can be calculated.

Please refer to FIG. 11B. FIG. 11B is a schematic diagram showing the possible fixation area and the visual field of the eye formed by the simulated projection in FIG. 11A from another angle. In FIG. 11B, the dashed lines 805A and 805B indicate the visual field of the eye and define the projection range 840; and the dashed lines 805C and 805D indicate the visual field of the same width as the head 800A and define the projection range 820. In view of the foregoing, the area 840 formed by the field of vision 805A and 805B with the peripheral light of the eye is the largest; the area 820 formed by the field of view projection with the same width as the head 800A is the smallest. In general, the line of sight of the eyes always looks forward in the same direction as the head. Therefore, the area 820 located in the center is the area where the customer is most likely to look at (interested). For example, suppose that there is a 95% chance that customers are looking at the items (commodities) in the most central area 820, and then the outer area 830 removes the ring part of the area 820 may have a 65% probability. The area range 840 of the outer ring may only have a 15% chance of removing the ring part of the area range 830. According to another embodiment of the present invention, when the eyes of the face in the picture can be clearly recognized, the calculation method can be adjusted according to the recognized line of sight of the eyes, and a more accurate area range 820E can be obtained, and the area range 820E The probability percentage of 820E should be higher than 95% in the area range 820. The probability percentage of 820E in the area range 820E shown in Figure 11B is 98%.

Compared with the prior art, the technology disclosed in the present invention must record the center line of a person’s face and the direction of the eye (eyeball direction) through two lenses at the same time to know that the person is watching. What is the object. The technology disclosed in the present invention only needs to capture the head image of a person through one lens, and even if the face is not seen, it is possible to know what kind of object the person is looking at. Therefore, the method disclosed in the present invention is suitable for use in situations where the image data contains many people/objects, and the method only needs to use a single image camera (lens), and the pixel (resolution) requirement of the image data does not need to be high. To the extent that the eyeballs on the human face can be seen clearly. Therefore, the method of the present invention can simply achieve the purpose of identifying which people in the picture are looking at which objects.

Please refer to FIG. 12A and FIG. 12B at the same time. According to an embodiment of the present invention, FIG. 12A is an architecture diagram of an Intelligent Electric Signage System (IESS) 100M. The system 100M includes an image processing device (Image Processing Device) 300M, a video/audio display device (Video/Audio Display Device) 850, an image capture device (Image Capture Device) 400M, and a content server (Content Server, CS). ) 870 and a Media Player 880. Among them, the image processing device 300M and the image capturing device 400M, the image processing device 300M and the media player 880, and the media player 880 and the audio-visual display device 850 are connected to each other through a transmission line 855. ; The image processing device 300M, the media player 880 and the content server 870 are connected to each other through a network 860 or a transmission line to transmit data. The network 860 may be a local area network (LAN), a wide area network (WAN), an Internet (Internet), or a wireless network (Wireless Network), etc. According to another embodiment of the present invention, the image processing device 300M and the image capturing device 400M do not transmit data through the transmission line 855, but are connected to each other through the network 860 or another network to transmit data. According to another embodiment of the present invention, the image processing device 300M and the media player 880 do not transmit data through the transmission line 855, but are connected to each other through the network 860 or another network to transmit data. According to an embodiment of the present invention, the image capturing device 400M can be independently installed near the audio-visual display device 850; according to another embodiment of the present invention, the image capturing device 400M is integrated with the audio-visual display device 850 into a single device .

According to an embodiment of the present invention, the image processing device 300M and/or the content server 870 may be a virtual machine (Virtual Machine, VM) in the cloud data storage unit 150 in FIG. 2, and an operating system running on the virtual machine It is also provided by the supplier of the cloud data storage unit 150. Please refer to FIG. 13B, where the image analysis application 460M can be executed in the operating system on the virtual machine provided by the cloud data storage unit 150; that is, the user uploads the image analysis application 460M to the cloud data storage unit 150, Then the image analysis application 460M is executed through the virtual machine. In this embodiment, the image data captured by the image capturing device 400M is first transmitted to the cloud data storage unit 150 via the network 860, so as to be provided to the virtual machine located in the cloud (ie, the image processing device 300M) for analysis and processing .

In FIG. 12A, the image capturing device 400M is used to capture people and/or objects located in the front. The image capturing device 400M in FIG. 12A and the image capturing devices 400A-400N in FIG. 2 are similar devices, and their functions are not very different. For the details of the image capturing device 400M, please refer to the previous section on Figure 2 Description of the image capture device 400A-400N. In an embodiment of the present invention, when the image capture device 400M and the audio-visual display device 850 are integrated into a single device, the image capture device 400M can be placed above, on the side, or below the audio-visual display device 850 to record People watching the video data in the audio-visual display device 850. Basically, the image capturing device 400M contains only one image lens to implement the technology disclosed in the present invention; but in other embodiments, the image capturing device 400M can also be equipped with multiple lenses to meet the requirements of special applications. need. Through the image data captured by the image capturing device 400M, the image processing device 300M can detect the sight direction of people standing in front of the image capturing device 400M (ie, the audio-visual display device 850) to determine whether there are people and how many People and who are watching or watching the content displayed by the audio-visual display device 850. Therefore, the installation position of the image capture device 400M only needs to achieve the above purpose (that is, it can record the position of people's face (line of sight)), and it is not limited to be installed above the audio-visual display device 850 or other specific positions. .

In FIG. 12A, the image processing device 300M of the present invention is used to process and analyze the image data from the image capture device 400M. The hardware and software architecture of the image processing device 300M is similar to the image analysis server 300 in FIG. 3A and FIG. 3B. Please refer to FIGS. 13A and 13B. Compared with the image analysis server 300 of FIGS. 3A and 3B, the image processing device 300M only needs one physical storage device 385 and a single operating system, which is enough to implement the smart electronic signage system of the present invention. In another embodiment of the present invention, the image analysis server 300 of FIGS. 3A and 3B can also be used as the image processing device 300M. For the various hardware and software components of the image processing device 300M in FIGS. 13A and 13B, please refer to the description of each hardware/software component in FIGS. 3A and 3B in the foregoing. The image processing device 300M receives the image data from the image capturing device 400M, and performs detection and analysis through the image analysis application 460M on the image processing device 300M to determine whether the video content (such as advertisements) played by the video and audio display device 850 is being watched. ) Of the various attributes of people (Attributes of People). In addition, it is synchronized with the playback progress of the media player 880 and/or the content server 870 regarding the Attribute of Contents data of the plurality of audiovisual advertisements, as a reference for further adjusting the playback content and order of the plurality of audiovisual advertisements later. . When the image processing device 300M needs to record the various attributes of the people watching the audio-visual advertisements, the playback progress of the media player 880, and the content attributes of multiple audio-visual advertisements, etc., and these statistical data need to pass through a handheld device (Hand Hold Device) output, the basic structure of the image processing device 300M in FIG. 13A further includes a wireless transmission device (Wireless Transmission Device, not shown), such as a WiFi device or a Bluetooth device. The aforementioned handheld device may be, for example, a smart phone (Smart phone), a tablet PC (Tablet PC), a notebook PC (Notebook PC), etc.

The Media Player 880 in Figure 12A decodes the Streaming Video/Audio Data from the content server 870 through the network 860, reassembles it into video data and audio data, and outputs it through the transmission line 855 To the audio-visual display device 850; meanwhile, the media player 880 will also transmit the real-time playback progress to the image processing device 300M. According to another embodiment of the present invention, the media player 880 only outputs the video data and audio data to the audio-visual display device 850 through the transmission line 855, and does not transmit the real-time playback progress to the image processing device 300M. The above-mentioned transmission line 855 for transmitting video data and audio data to the audio-visual display device 850 may be, for example, a transmission line matched with an HDMI terminal, a transmission line matched with an A/V terminal, etc. The media player 880 in FIG. 12A is implemented by hardware. According to another embodiment of the present invention, the media player 880 can also be implemented by software. For example, a multimedia player program 465M in FIG. 13B is used in the image processing device 300M to perform streaming video and audio playback with software. method.

The audio-visual display device 850 in FIG. 12A is used to display the streaming audio-visual data transmitted by the media player 880, and basically includes a display (panel) 890 and a speaker system (not numbered). The streaming audio-visual data is transmitted by the content server 870 through the network 860, and after being decoded by the audio-visual decoder in the media player 880, the video data is output to the display 890 to display the audio-visual advertisement, and then decoded The sound data is sent to the speaker system to play the sound. According to another embodiment of the present invention, the audio-visual display device 850 further has a network interface, not through the transmission line 855 but through the network 860 to receive the video data and audio data from the media player 880.

According to another embodiment of the present invention, the image processing device 300M and the media player 880 in FIG. 12A can be integrated into a single device; in addition, the image capturing device 400M and the audio-visual display device 850 can also be integrated into a single device. According to another embodiment of the present invention, the aforementioned image processing device 300M, media player 880, image capturing device 400M, and audio-visual display device 850 can be integrated into a single device.

The content server 870 is a server with a network interface. Basically, the hardware architecture of the content server 870 is not much different from the general server; but in terms of software content, the content server 870 will provide a variety of different types of audiovisual files (for example, advertisements). The contents of these various types of audiovisual files (such as advertisements) are designed for people of different ethnic groups, among which the so-called people of different ethnic groups can be, for example: elderly men/women, young men/women, children, obese people Men/women, men/women who wear glasses, men/women who keep pets... and other ethnic groups with different attributes. The content of audio-visual files (for example, advertisements) involves products or services in various fields including food, clothing, housing, transportation, education, and music. In addition, various information such as the ethnic group, playback sequence, video length, etc., targeted by each of the above-mentioned audiovisual files (for example, advertisements) are the "Attribute of Contents" of the audiovisual file (for example, advertisements). The content server 870 can circulate the above-mentioned various types of audio-visual files (such as advertisements) in a 24-hour uninterrupted manner or at a specific time, and convert these audio-visual files (such as advertisements) into the format of audio-visual streaming data and connect them to the Internet. Route 860 to the media player 880. The content server 870 in FIG. 12A transmits the content attributes and other related data of the audiovisual file (for example: advertisement) to the image processing device 300M via the network 860. In addition, the content server 870 will also receive instructions from the image processing device 300M through the network 860 to change the playback behavior of video files (such as advertisements), such as interrupting playback, changing the playback order, fast forwarding images, volume adjustment, display specific Text...etc.

Please refer to FIG. 12B, which is an application scenario diagram of the smart electronic signage system 100M shown in an embodiment of the present invention. Figure 12B shows that 2 customers are watching the current advertisement content in front of the audio-visual display device 850. The image capture device 400M located above the audio-visual display device 850 is capturing the expressions and reactions of the two visitors. The images are captured at the same time. The device 400M also sends the recorded image data to the image processing device 300M for analysis. The image processing device 300M uses the internal image analysis application 460M to analyze and determine the gender, age, facial expression, clothing, height, etc. attributes of the two visitors in real time. The above-mentioned various attribute information about visitors has a very high correlation with the content of the broadcasted video and audio advertisements. This kind of connection will make the broadcasted video and audio advertisements attract the attention of the most consistent customer group to maximize the advertising benefits. .

Please refer to FIG. 13A. FIG. 13A is a block diagram of the basic hardware structure of the image processing device 300M in FIG. 12A according to an embodiment of the present invention. The hardware architecture of the image processing device 300M in FIG. 13A is almost the same as the hardware architecture of the image analysis server 300 in FIG. 3A. The difference is that the image processing device 300M may further include a wireless transmission device (Wireless Transmission Device, not shown), which allows the image processing device 300M and a handheld device to transmit data wirelessly. For each hardware architecture in FIG. 13A, please refer to the related description in FIG. 3A in this specification.

Please refer to FIG. 13B. FIG. 13B is a diagram illustrating the software and hardware architecture relationship of the image processing device 300M in FIG. 12A according to an embodiment of the present invention. The hardware architecture of the image processing device 300M in FIG. 13A is almost the same as the hardware architecture of the image analysis server 300 in FIG. 3A, except for the number of physical storage devices and the hardware specifications. In addition, with regard to the software architecture in FIG. 13B, in addition to the absence of a virtual machine monitor 470, the operating system 475 and image analysis application 460 of the image processing device 300M are almost identical to the operating system 475 and image analysis application 460 in FIG. 3A. It's all the same. In addition, in FIG. 13B, the operating system 475 may also include a multimedia player program 465M. The multimedia player program 465M is drawn with a dotted line, which means that when the media player 880 is not used, the multimedia player program 465M (software ) Can replace the function of the media player 880 (hardware). Because the multimedia player program 465M may not necessarily be executed in the operating system 475, the multimedia player program 465M in FIG. 13B is a software module shown by a dotted line. In addition, according to an embodiment of the present invention, the application modules in the image analysis application 460 in FIG. 13B, in addition to the various application modules shown in FIG. 4, include expression detection modules and height detection modules. Measurement module, clothing detection module, fat and thin detection module, portable tool detection module, portable pet detection module, occupation prediction module... etc. For the hardware and software architecture of the image analysis server 300 in FIG. 13B, please refer to the descriptions of FIGS. 3B and 4 described above.

Please refer to FIG. 14, which is a flowchart of the overall operation of the smart electronic signage system 100M according to an embodiment of the present invention. The overall operation flow chart includes a smart electronic signage person detection training phase 900, a smart electronic signage person survey/statistical analysis phase 910, and a smart electronic signage application phase 920. The person detection training phase 900 of the smart electronic signage is the training phase shown in FIG. 5 and FIG. 6 (step 510 in FIG. 5, step 520 in FIG. 6, and steps 610 to 640). Please refer to step 510 of FIG. 5, step 520, and step 610 to step 640 of FIG. 5 for the person detection training phase 900 of the smart electronic signage.

In the present invention, the detection of people in image data is based on the "feature mapping" of various attributes of people obtained through deep learning using the Data Set during the training phase under the CNN architecture, and in In the application stage, the learned "feature mapping" is used to detect the part of the image data captured by the image capture device 400M that meets the feature mapping, and then analyze the "various attributes of people" . In particular, for the person detection training stage 900 of the smart electronic signage system in Figure 14, it is basically only related to "person recognition" such as "face/head recognition" and "body recognition", and Does not include "identification of objects/commodities". Therefore, in the training phase, for various image data, the main data set (DataSet) related to "people" is provided. This person-related data set not only provides the file data shown in Figure 7A, but also includes: data sets of male/female faces and bodies of various age groups (elderly/middle-aged/young/children...) A data collection of postures (fat/slim/moderate...), a data collection of facial expressions (happy/angry/sorrow/joy/fear/disappointment...) etc. In addition, data sets related to "people" can further include data sets of "babies" and "pets". If you need to estimate a person’s occupation, you need to provide various data sets including "clothes (uniforms for various industries)", "tools (tools for various industries)"... and so on. For the "Smart Electronic Signage Person Detection Training Phase" 900 in FIG. 14, please refer to the descriptions of steps 510 and 520 in FIG. 5 and steps 610 to 640 in FIG. In addition, the detailed flow of the implementation of the "Smart Electronic Signage Person Survey/Statistical Analysis Phase" 910 in Figure 14 will be illustrated in Figure 15; the detailed flow of the implementation of the "Smart Electronic Signage Application Phase" 920 in Figure 14 will be shown in Figure 14 Explained in 16. In addition, step 910 in FIG. 14 is shown with a dotted line. The reason is that if the system 100M does not need to output and analyze the statistical data of step 910, then step 910 in FIG. 14 "Smart Electronic Signage Characters" "Investigation/Statistical Analysis Phase" can be omitted, so step 910 is shown by a dotted line.

Please refer to FIG. 15. FIG. 15 shows a flowchart based on the "personal survey/statistical analysis stage of the smart electronic signage system" 910 in FIG. 14. Following the description of step 910 in FIG. 14, when the system 100M does not need to output and analyze the statistical data of step 910, the flow in FIG. 15 can be omitted. The process in FIG. 15 starts at step 1000.

In step 1000, the image processing device 300M obtains image data to be analyzed. In FIG. 12A, when the audio-visual display device 850 is playing audio-visual advertisements with different "content attributes", the image capturing device 400M also simultaneously records various aspects of the person in front of the audio-visual display device 850. According to an embodiment of the present invention, the so-called "person" here only refers to the part of the image data related to the "person", and does not include other objects, such as cars, buildings, traffic signs, etc., in the following description The "persons" referred to in the middle are all the same.

In step 1010, the part of the image data with a subject is identified; the so-called "subject" is a subject with specific characteristics that can be identified by the image analysis server 300M after deep learning. Each frame of the image data may contain one or more objects with specific characteristics. These objects may be objects with the characteristics of a person's "face/head" or a person's "body". "" and other characteristics of the subject. According to an embodiment of the present invention, since each picture in the image data is composed of a large number of pixels, in step 1010, it is necessary to determine which targets of the "face/head" feature of a person are included in the picture and which ones are persons. In addition, it is necessary to know where each object is distributed in the picture; in other words, to judge every object with human "face/head" characteristics in the picture and every object with human The target of the "body" feature, and the area (pixel size) and position of each individual in the picture are the main purpose of performing step 1010. Generally speaking, although the dynamic specification of image data is composed of pictures from a few to dozens of pictures per second per second, according to an embodiment of the present invention, it is not necessary to detect every picture in practice. It includes how many objects with the characteristics of a person's "face/head" and objects with the characteristics of a person's "body". If you take the most common 30 frame/second (second) specification as an example, you can choose, but not limited to this, select a frame (picture) to detect every 1/6 second interval. Measure how many objects with human "face/head" characteristics and objects with human "body" characteristics are included in the picture, as well as their respective positions and ranges in the picture. In other words, under this assumption, in step 1010, every 1 second, it is necessary to discriminate between 6 pictures with a person's "face/head" feature and a person's "body" feature. The foregoing detection of a frame of pictures every 1/6 second is only an embodiment of the present invention, in fact, the time interval can be any other numerical time interval. It must be noted that the detection of the person's "face/head" or the person's "body" in the picture in step 1010 is not achieved through the comparison of the Data Base, but by the deep learning in the training process The learned "feature mapping" is used to identify the person's "face/head" and/or the person's "body" in the picture.

After the image data is recognized in step 1010, the target of the recognized person's "face/head" and the person's "body" can be divided into two parallel processes: "face/head" and "body" for processing . Therefore, after step 1010, the target of the person's "face/head" is recognized to step 1020 and the following steps are performed; the target of the person's "body" is recognized to step 1030 and the following steps are performed.

Furthermore, according to an embodiment of the present invention, the above-mentioned so-called "face/head" of a person includes parts of the head in addition to the face; the so-called "body" here refers to parts of the body other than the "face/head" other parts. According to another embodiment of the present invention, the image processing device 300M analyzes the content of the image data, in addition to "face/head" and "body", it also includes all "objects" extremely related to "people". For example: personal clothing (clothes, hats...), personal objects (babies, glasses, jewelry, pets, purses...), etc.

Continuing from the above description, step 1020 is a step in which the target identified as a person's "face/head" in the picture is entered, and step 1020 is the detection of the person's face/head in the picture. Since the range of the face/head shown in the picture is large or small, if there is a small area (pixel) of the face/head in the picture, it may not be effectively detected. In an embodiment of the present invention, the pixel size that can be detected is more than 40×40 pixels (inclusive). The threshold value of 40X40 pixels is only an implementation reference provided by the present invention, and the present invention is not limited to this. The lowest recognizable pixel threshold value may be any other value of pixel size, depending on the hardware and software. Depends on the ability. After determining the target of the "face/head" recognized as a person in step 1020, proceed to "detection of the direction of the human head" in step 1040, "estimate the position of the human head" in step 1050, and step 1060 "Detection of Human Gender/Age/Expression" and other 3 steps. The above three steps are all related to the "face/head" of a person, and these three steps will be explained later.

In step 1040, the head direction of the person in the picture is detected. Following the description of the previous step 1020, after the execution of step 1020 is completed, the "detection of human head orientation" in step 1040, the "estimation of human head position" in step 1050, and step 1060 will be executed in parallel processing. In "Detection of Human Gender/Age/Expression". According to an embodiment of the present invention, before step 1040 is performed, the data set (DataSet) shown in FIG. 7A must be used in the training phase of FIG. 14 to let the image analysis application 460M do deep learning, with the learned topic as the head The "feature mapping" of the direction, and then the "feature mapping" of the head direction to determine the most likely state of the person's head direction in the picture. Take Figure 7A as an example. The pictures numbered 1-91 show images of the same woman with different head orientations. Each image has a corresponding label, and the information recorded in each label At least include "subject" (e.g. female head), and "subject's roll, pitch, and yaw three values". From the above, it can be seen that the pictures No. 1-91 show a female face/head in different "head directions". In actual application, the direction of the customer’s head in the picture (that is, the degree of freedom of turning in the space formed by the head roll (Roll), pitch (Pitch), yaw (Yaw), etc.) can roughly correspond to the learning Number 1-91 is one of the conditions, so the image analysis application 460M can use the "feature mapping" obtained by deep learning in the past training phase to determine the roll corresponding to the customer's head direction in the picture. Pitch, Yaw and other values. If the person’s head orientation in the picture does not completely match one of the above numbers 1-91, step 1040 will also perform a comparison of the person’s head orientation according to the learned "feature mapping" The closest judgment, estimate its most likely Roll, Pitch, Yaw and other values. In addition, according to an embodiment of the present invention, step 1040 may further calculate the position and range of the human head in the picture. If it can be identified, if a picture includes 10 men and women, the head directions of the 10 men and women will be detected. In addition, people in front of the image capturing device 400M may have different behaviors: they may be watching video advertisements that are being broadcast, they may be talking, they may be looking at distant buildings, and other different behaviors. The behavior presented by the "person's head direction" will also be different.

In step 1050, it is an estimation of the head position of the person in the picture. In the present invention, before executing the application process shown in FIG. 15, the image capturing device 400M must be calibrated to obtain the relevant parameters (for example: Intrinsic/Extrinsic matrix) of the image capturing device 400M. After learning and training, step 1050 can use the parameters of the image capturing device 400M and the image data of the specific object in the picture to calculate the distance and relative position of the specific object in the picture and the image capturing device 400M. According to an embodiment of the present invention, the relative distance between the specific object and the image capturing device 400M can be regarded as the distance between two points in space. Generally, the image capturing device 400M can be set as the origin, so the coordinates of the image capturing device 400M in the 3D space are expressed as (0, 0, 0). Then the coordinates of the specific object in the 3D space can be calculated, for example: (100, 552, 211), the distance of the specific object relative to the origin (the image capturing device 400M) is a calculable length. According to another embodiment of the present invention, the origin in the 3D space can also be set as the triangle intersection point between two adjacent walls of the store in the picture and the ground. In summary, the present invention first performs calibration on the image capturing device 400M to know the relative distance of the specific point in the 2D picture to the image capturing device 400M or the reference point (the set origin) in the 3D space . Accordingly, in step 1050, the calibrated image capturing device 400M can be used to estimate the position of the human head in the 3D space.

In step 1070, it is the estimation of the gaze area of the person in the picture. According to the results (data) after performing steps 1040 and 1050, the gaze area of the person in the picture can be further known.

According to an embodiment of the present invention, after detecting the head direction of the person in step 1040 and estimating the head position of the person in step 1050, it is possible to know the "head direction" (roll, pitch) of a person in the picture. , Yaw three values) and the position of the head of a certain person in three-dimensional space. Through these two kinds of information, the following simulation is performed: the light projected in the same direction as the direction of the head from right behind the head of the person in the picture, and the projected light is set to extend until the audio-visual display device 850 Position, and then at the position of the audio-visual display device 850, suppose there is a simulated screen and the projected light forms a simulated projection on it. The projection area formed by the simulated projection is the possible gaze area of the certain person, such as the elliptical projection areas 815A and 815B as shown in FIGS. 11A and 11B. According to an embodiment of the present invention, the projection area is further divided into a plurality of sub-projection areas, where the sub-projection area with the highest probability is the most likely focus range of the person’s line of sight, and it can also be inferred from this that the person is watching Whether the object is the screen 890 on the audio-visual display device 850, to determine whether the person is watching the audio-visual advertisement that is being broadcast. Please refer to the description of step 740 in FIG. 9, FIG. 11A, and FIG. 11B. In step 1070, each person in the picture has its own projection area according to its different head orientation, and each person's respective projection area represents the gaze area of each customer.

According to an embodiment of the present invention, in step 1070, only when the person in the picture has watched an object for more than a set time threshold, will it be inferred for them whether their gazing area (Gazing Area) is the audio-visual display The screen 890 on the device 850; the set time threshold may be, for example, 5 seconds, but it is not limited to this, and can be any set time length. According to another embodiment of the present invention, if the person in the picture is moving and their gaze continues on the same object for more than a set time threshold (for example: 5 seconds, but not limited to this), they will also be targeted It is inferred whether the gaze area is the screen 890 on the audio-visual display device 850. In step 1070, except for the situation where the head direction of the person in the picture cannot be recognized, each person in the picture should be able to estimate the size of their gaze area (projection area) and the probability of being looked at in the sub-projection area. The reasons for the above "unrecognizable head direction of the person in the picture" may be: (1) The head image of the person in the picture is incomplete, for example: a large area of the person's head image is blocked by other people or objects; (2) ) The pixels contained in the image range of a certain person's head in the picture are too small, so that the certain person's head cannot be identified. According to the technology disclosed in the present invention, even if the head image of the person in the picture does not include the eyes of the face (for example, the head image on the back side), the head direction of the person can still be recognized. According to another embodiment of the present invention, when the head image of the person in the picture clearly shows the eye characteristics of the face, the direction of the customer’s line of sight can be calculated and referred to, and the projection area mentioned above can be further reduced to increase the estimated picture. The correctness of the middle person's gaze area. After step 1070 is executed, skip to step 1080 for execution.

Go back to step 1060. In step 1060, it is the detection of the gender, age, and expression of the person in the picture. According to an embodiment of the invention, the "face" part of the person in the picture should be clearly recognized outside the head direction, and the person in the picture who fails to meet this standard will be ignored and not detected. According to another embodiment of the present invention, when the area of the face of the person in the picture that is covered by other people or objects exceeds the upper limit that the image analysis application 460M can determine, the person who is covered will be ignored. I detect. Basically, the area of the person’s face in the picture must contain at least 40×40 pixels or more. The value of 40×40 is only an embodiment of the present invention and is not a limitation of the present invention. Therefore, the picture In the area of the person's face, the pixels contained in it can be pixels of any size. The detection of "gender", "age", and "expression" in step 1060 uses the Data Set in the database 450 during the training phase of step 920 in Figure 14 for deep learning, and the topics obtained are each "gender" "Feature mapping" of "Age", "Expression", so in the training phase of step 920 in Figure 14, the database 450 should be provided, for example: "Young male/female", " "Middle-aged male/female", "Older male/female", "Middle-aged male/female", etc., data sets of various age groups (Data Set) to learn to detect the "gender" of the person in the picture, "age". For detecting the "age" of the person in the picture in step 1060, as long as more DataSets are provided during the training phase, the "age" of the person in the picture can be detected more accurately in step 1060. In addition, for the detection of human "emoji" in the picture, the DataSet in the database 450 during the training phase of step 920 in FIG. 14 should contain various emotions, such as: smile, happiness, anger, sadness, happiness, tears, surprise , Fear, doubt... and other data sets that show emotions. Although it is shown in step 1060 that the detection of the "gender", "age", and "expression" are listed in the same step, the system and steps for the detection of the "gender", "age" and "expression" are implemented 1040. Step 1050 is processed in a parallel process. In addition, the above-mentioned "gender", "age", and "expression" of the person in the picture in step 1060 can be regarded as one of the "person's attributes".

Returning to the above step 1010, in step 1010, when the subject of identifying the characteristics of the person in the picture is "body", then step 1030 is entered.

In step 1030, it is the detection of the body part of the person in the picture, especially the detection of the "person's height", "person's posture (fat/thin)", "person's clothing" and other parts. For the height of a person, it is necessary to clearly display all the image data of the person "head to shoes" in the picture, otherwise the height of the person who fails to meet this standard will be ignored and not calculated. According to an embodiment of the present invention, since the head position of the person in step 1050 can be estimated, in the same way, in step 1030, the height of the person in the picture can also be estimated in this way. Basically, "adult male/female" and "child male/female" can be identified through height detection. In addition, for the detection of "human body (fat/thin)" and "human clothing" in step 1030, the same is done in the training phase of step 920 in Figure 14 using the Data Set in the database 450 for deep learning , The themes obtained are "feature mapping" of "human posture" and "human clothing". Therefore, in the training phase of step 920 in Figure 14, the database 450 needs to provide data sets of various postures (height, fat, thin, etc.) of men and women, and data sets of "uniforms worn by various occupations" (Data Set). ) To learn to detect the posture of the person in the picture (tall, short, fat, thin, etc.), and to determine the occupation by detecting the clothes (uniform) worn by the person in the picture. For the detection of "human posture" and "human clothing" in the picture in step 1030, as long as the more Data Sets provided during the training phase, the better the learning result; in other words, it can have a stronger ability to accurately Determine the attributes of the "person's posture" and "person's clothing" in the picture. According to another embodiment of the present invention, inferring the occupation of the person in the picture further includes analyzing the person's "carry-on items". For example: when the person’s personal belongings in the picture is a "briefcase", the occupation of the person in the picture can be presumed to be an "office worker"; when the person's personal belongings in the picture is a "school bag", the occupation of the person in the picture It can be inferred that it is a "student"...etc. Since it is not easy to guess the occupation of the person in the picture, in practice, the estimation of the occupation of the person in the picture will basically analyze several situations, in addition to the above-mentioned "clothing" and "personal items", it also includes "accessories" , "Time", "Location", "Weather", ... and other conditions are added to the analysis to infer the "occupation" of the person in the picture. Although it is shown in step 1030 that the detections of the "height", "fat/slim", and "clothing" are listed in the same step, in practice, the "person's height" and "person's posture (fat/slim) The detection of "clothing" is processed in parallel with step 1040, step 1050, and step 1060.

In step 1070, it is the estimation of the gaze area of the person in the picture. Following the execution result (output) of step 1040 and step 1050, step 1070 can estimate where the area that a person is looking at in the picture is. After step 1070 is executed, it can be known "whether a person in the picture is watching the screen 890 on the audio-visual display device 850", that is, whether the person in the picture is watching the audio-visual advertisement being played. Therefore, the most important process is to collect relevant information (attributes) of the "people in the picture who are watching the video advertisement being played". For this process, it must also include the condition of "continue watching the video and audio advertisement being broadcast for more than a period of time." The condition for this period of time is 5 seconds. In addition, the value of 5 seconds is only an embodiment of the present invention, and is not a limitation of the present invention. The above-mentioned 5 seconds time length can be any number of seconds. Continuing from the output results of step 1070, step 1060, and step 1030, the basic condition of the target person (Target Person) that needs to be counted in the present invention is: "The person in the picture continues to watch the video advertisement being broadcast for more than a preset period of time. Over time". After the step 1070, step 1060, and step 1030 are executed, the judgment of step 1080 is executed.

In step 1080, it is determined whether the person in the picture is looking at the screen 890 of the audio-visual display device 850 and the time length of looking at the screen 890 continuously exceeds a time threshold, for example, more than 5 seconds. According to an embodiment of the present invention, the talents in the picture who meet the above two conditions at the same time meet the targets that need to be counted, and the other people in the picture who do not meet the above two conditions will be ignored. If the result of the judgment in step 1080 is true (Yes), step 1110 is executed, otherwise, step 1120 is executed.

In step 1090, the media player 880 provides the video processing device 300M with the playback progress of the video and audio advertisement currently being played, and the content server 870 provides the video processing device 300M with the content attributes of the video and audio advertisement currently being played. According to another embodiment of the present invention, the playback progress of the current audio-visual advertisement and the content attributes of the currently-played audio-visual advertisement may both be provided by the content server 870 or by the media player 880. In step 1090, the media player 880 provides the playback progress of the audiovisual advertisement currently being broadcast, and the playback progress basically refers to the number of seconds in the audiovisual advertisement currently being played. The content attributes of the audio-visual advertisement include: the number of the audio-visual advertisement currently being played, the playing sequence of the audio-visual advertisement, the target product of the audio-visual advertisement (for example: ice cream, car, glasses, health food... etc.), the audio-visual advertisement suitable for The ethnic group of the audience (for example: women, the elderly, children, male office workers... etc.), the preset broadcast period of the video and audio advertisement, the total length (time) of the video and audio advertisement... and other information.

In step 1110, the related information (content attribute) of the person in the picture and the content attribute of the video and audio advertisement are synchronously recorded. In step 1110, relevant information such as the output results in step 1060, step 1030, and step 1090 will be recorded. The aforementioned "record" is a statistical record (not shown in the figure), and the statistical record accumulates the record data in the database 450 in a newly added manner. The statistical record includes the current date/time, the gender, age, height, fat, thin, occupation, facial expressions of the person watching the video advertisement and the content attributes (number, target product, preset Various field information such as the duration of the broadcast, the total length (time) of the broadcast, and the current playback progress. In addition, the statistical record can be in a record format such as a database file, a table, and a text file. The statistical record can be stored in the hard disk 385 of the image processing device 300M or in the non-volatile memory 330. According to an embodiment of the present invention, when the statistical record reaches a predetermined number of records or the audio-visual advertisement reaches a predetermined test time, the statistical record stored in the image processing device 300M will be transmitted to others through the network 860 Server (not shown). After step 1110 is executed, it will jump back to step 1010 to continue the analysis of other people's image data in the picture, and continue to execute iteratively. According to an embodiment of the present invention, basically, each person in the picture will detect their gender, age, height, weight, occupation, expression and other attributes (personal attributes) in a parallel processing manner.

In addition, following the execution result in step 1080 above, when the result is false (No), step 1120 is executed.

In step 1120, it is determined whether the predetermined test time has been reached or the amount of data accumulated in the statistical record is sufficient. Under normal circumstances, the person survey/data statistics phase of step 910 in FIG. 14 has a predetermined period when it is executed, and the predetermined period can be 1 day, 3 days, 7 days... and other different time intervals. When the time interval is longer, the amount of accumulated data is larger, and the analysis result of the statistical record will be closer to the goal predetermined by the present invention in actual application. In addition, when the cumulative data volume of the statistical record has reached a certain number, the correlation between the statistical record and the audio-visual advertisements can be further analyzed. When the judgment result of step 1120 is true (Yes), step 1130 is executed, otherwise, it jumps back to step 1010 to continue execution.

In step 1130, the statistical record (not shown) is output to other devices via the network 860 for analysis. Basically, the statistical record will be output through the network 860, and then the correlation between the statistical record and the audio-visual advertisements will be analyzed. According to another embodiment of the present invention, the statistical record (not shown) is output to a handheld device via WiFi or a Bluetooth device.

In step 1140, the correlation between the statistical record and the content of each video and audio advertisement is analyzed. The execution of the analysis behavior is performed by the manufacturer of the video and audio advertisement content, or by the provider of the image processing device 300M. Continuing to analyze the statistical records (not shown) accumulated after the execution of step 1110 above, the results of the analysis may be: "The ethnic group attracted by Advertisement A is the elderly over 70 years old", "The ethnic group attracted by Advertisement B It is an office worker between the ages of 35-45", "The content of the advertisement in Advertisement C is not attractive, and most people's attention time is almost 1.5 seconds", "The content of Advertisement D is between 1:30 and 2:05 It’s most likely to cause everyone to laugh"... and other analysis results. These many analysis results will be used as the information basis of the "smart electronic signage application stage" in step 920 of Figure 14 when it is executed, and the conclusion also represents the audiovisual An unmanned questionnaire for the manufacturer of the advertising content.

The completion of step 1140 means that the process in FIG. 15 has ended, and it also means that the "smart electronic signage character survey/statistical analysis stage" of step 910 in FIG. 14 has been executed.

The analysis result after the execution of the above step 1140 can achieve the purpose of "which video and audio advertisement content will attract which ethnic groups". Therefore, the analysis result after the execution of step 1140 can be added as part of the "content attribute" of the video advertisement.

Furthermore, the execution of the above steps 1040, 1050, and 1070 is to determine whether the person in the picture is looking at the screen 890 on the audio-visual display device 850. If the person in the picture is not looking at the screen 890 on the audio-visual display device 850, then People in this situation in the picture will not be included in the statistics. According to another embodiment of the present invention, when it is not necessary to determine whether the person in the picture is looking at the screen 890 on the audio-visual display device 850, step 1040, step 1050, and step 1070 can be omitted. In addition, in this case, there is no need to step 1080 to determine whether the person in the picture looks at the screen 890 on the audio-visual display device 850 for a time longer than a preset time threshold. In this case, step 1080 will also be omitted simultaneously.

Please refer to FIG. 16. FIG. 16 shows a flowchart based on the "application stage of the smart electronic signage system" 920 in FIG. 14. The process starts at step 1200.

In step 1200, the image data from the image capturing device 400M is obtained. Step 1200 in FIG. 16 is the same as step 1000 in FIG. 15. Please refer to the description of step 1000 in FIG. 15.

In step 1205, relevant information about the attributes of the audio-visual advertisement content is provided. The attributes of the advertisement content refer to the content attributes and other data of the audio-visual advertisement provided by the content server 870 or the media player 880. The content attributes of the above-mentioned audio-visual advertisements include: the serial number of each audio-visual advertisement, the playing sequence of each audio-visual advertisement, the number of the audio-visual advertisement currently being played, and the target product of each audio-visual advertisement (for example: ice cream, car, glasses, health care) Food... etc.), the ethnic group that each video advertisement is suitable for (for example: women, the elderly, children, male office workers... etc.), the preset broadcast period of each video and audio advertisement, the total playing time of each video and audio advertisement... And other information. Among them, the ethnic group that each audiovisual advertisement is suitable for has a very high degree of relationship with the various "human attributes" in the pictures of the subsequent detection image data. The "human attributes" are, for example: gender, height, Fat, thin, age, occupation... etc.

In step 1210, identify the subject part of the image data. Step 1210 in FIG. 16 is the same as step 1010 in FIG. 15. Please refer to the description of step 1010 in FIG. 15.

In step 1220, it is the detection of the face/head of the person in the picture. Step 1220 in FIG. 16 is the same as step 1020 in FIG. 15. Please refer to the description of step 1020 in FIG. 15.

In step 1240, it is the detection of the head direction of the person in the picture. Step 1240 in FIG. 16 is the same as step 1040 in FIG. 15. Please refer to the description of step 1040 in FIG. 15.

In step 1250, it is an estimation of the head position of the person in the picture. Step 1250 in FIG. 16 is the same as step 1050 in FIG. 15. Please refer to the description of step 1050 in FIG. 15.

In step 1270, it is the estimation of the gaze area of the person in the picture. Following the results (outputs) after steps 1240 and 1250 are executed, the gaze area of a person in the picture can be further known to determine whether the person is looking at the screen 890 on the audio-visual display device 850. Step 1270 in FIG. 16 is the same as step 1070 in FIG. 15. Please refer to the description of step 1070 in FIG. 15.

In step 1260, it is the detection of the gender, age, and expression of the person in the picture. According to another embodiment of the present invention, the step 1260 only detects the gender and age of the person in the picture, and does not include the detection of expressions. Step 1260 in FIG. 16 is the same as step 1060 in FIG. 15. Please refer to the description of step 1060 in FIG. 15.

Returning to the above step 1210, in step 1210, when the subject of identifying the characteristics of the person in the picture is "body", then step 1230 is entered.

In step 1230, it is the detection of the body part of the person in the picture, especially the detection of the "person's height", "person's posture (fat/thin)", "person's clothing" and other parts. Although it is shown in step 1230 that the detections of the "person's height", "person's posture (fat/thin)", and "person's clothing" are listed in the same step, in practice, for the "person's height", The detection of "human posture (fat/thin)" and "human clothing" is processed in parallel with step 1240, step 1250, and step 1260. Step 1230 in FIG. 16 is the same as step 1030 in FIG. 15. Please refer to the description of step 1030 in FIG. 15.

In step 1270, it is the estimation of the gaze area of the person in the picture. Following the execution results (outputs) of steps 1240 and 1250, step 1270 can estimate where the area that a person is looking at in the picture is. After step 1270 is executed, it is possible to know "whether a person in the picture is watching the screen 890 on the audio-visual display device 850", that is, whether the person in the picture is watching the video advertisement being played. Therefore, the most important process is to collect relevant information (attributes) of the "people in the picture who are watching the video advertisement being played". For this process, it must also include the condition of "continue to watch the video and audio advertisement being broadcast for more than a period of time", and the condition for this period of time is 5 seconds. In addition, the value of 5 seconds is only used as an embodiment of the present invention, and is not a limitation of the present invention. The above-mentioned time length of 5 seconds can be any number of seconds. Continuing from the output results of step 1270, step 1260, and step 1230, the basic condition of the target person (Target Person) that needs to be counted in the present invention is: "The length of time that the person in the picture continues to watch the video advertisement being broadcast exceeds one The preset time threshold". After the steps 1270, 1260, and 1230 are executed, the judgment of step 1280 is executed.

In step 1280, it is judged whether the person in the picture is looking at the screen 890 of the audio-visual display device 850 and the duration of looking at the screen 890 continuously exceeds a time threshold, for example, more than 5 seconds. In order to make according to an embodiment of the present invention, the talents in the picture who meet the above two conditions at the same time are the targets in the disclosed method of the present invention, and other people will be ignored. If the result of the judgment in step 1280 is true (Yes), then step 1290 is executed; otherwise, step 1330 is executed.

In step 1290, the person who is looking at the screen 890 in the picture and its attributes are counted in real time. The attributes of the person looking at the screen 890 in the above picture refer to the "gender", "age", "height", "body (fat/slim)", "accessories", and "clothing" of the person looking at the screen 890 in the picture. "", "Occupation"... and other features, step 1290 performs real-time statistics on the attributes of the person who is looking at the screen 890 in the picture (real-time attribute statistics, not shown). The statistical results of the real-time attribute statistics may be "women aged 20-26: 6 people", "elders aged 70: 6 people, children aged 6-10: 2 people", "women with glasses: 10 people" ", "45-60 year old male office worker: 16 people, 45-55 year old female office worker: 3 people", "obese female: 9 people, moderate female: 2 people, thin female : 4 people"...It is equal to the attribute statistics of various people in the picture. The result of the above real-time attribute statistics will be used as a basis for whether to change the content of the video and audio advertisement currently being played.

In step 1300, it is determined whether all people in the picture have been detected/statistically completed. According to an embodiment of the present invention, the image analysis application 460M processes the detection of everyone in the picture almost in parallel. Following the above description, the image analysis application 460M includes: detection of head orientation, estimation of head position, judgment of gender, estimation of age, estimation of height, estimation of fatness and thinness, estimation of occupation (clothing)... It belongs to the detection of face/head and body. When the image analysis application 460M completes the above tasks, it executes step 1310, otherwise, it jumps back to step 1210 for execution.

In step 1310, the most suitable (appropriate) video and audio advertisements are selected based on the results of real-time human attribute statistics and the content attributes of each video and audio advertisement. The above-mentioned real-time person attribute statistics is the statistics of all people who are looking at the screen 890 in the picture in step 1290 and their attributes; the audio-visual advertisement attributes are related data of the content attributes of each audio-visual advertisement provided in step 1205. The image analysis application 460M uses intelligent comparison and analysis. The image analysis application 460M will select the most suitable (suitable) audio-visual advertisements of "the person watching the audio-visual advertisement in the current picture". The selected audio-visual advertisement may only be one or more than one. When the selected audio-visual advertisement is more than one, it will further include determining the order of its broadcast. The above-mentioned intelligent comparison and analysis may be: when more than two-thirds of the people included in the picture are males aged 30-65, choose to broadcast audio-visual advertisements about cars; four of the people included in the picture When more than three-thirds of women are 12-50 years old, choose to broadcast audio-visual advertisements about female physiological products; when more than one-half of the people included in the picture wear glasses, choose to broadcast audio-visual advertisements about myopia glasses …And other video and audio advertisements that belong to different content attributes.

In step 1320, the content server 870 is notified to change the currently playing video and audio advertisement and its sequence. Since the file of the video and audio advertisement is stored in the content server 870, the playback of the video and audio advertisement and the output of the video and audio stream are also completed by the content server 870. Therefore, the content server 870 can determine the content (content attributes) of the current video advertisement to be played and its play order. Therefore, the video processing device 300M notifies the content server 870 to play the video and audio advertisements with "in step 1310, the video and audio advertisements and their order selected after intelligent comparison and analysis". In addition, at this time, for the audio-visual advertisement that has not been played so far, the content server 870 may immediately interrupt the audio-visual advertisement and switch to a new sequence of audio-visual advertisements to broadcast. After switching the new order, the advertisement attribute of the video advertisement is the most relevant to the attribute of the person currently watching the video advertisement broadcast in the picture, so it is very likely that the picture is currently watching the video advertisement broadcast. People are most interested in video and audio ads. In the case of the audio-visual advertisement, the audio-visual advertisement aroused the interest of the most people in the picture. Therefore, the smart electronic signage system 100M of the present invention can maximize the advertising benefits of the audio-visual advertisement.

In step 1330, the image processing device 300M determines whether the pause/shutdown signal sent by the content server 870 is received. In order to reduce the cost of broadcasting video and audio advertisements, such video and audio advertisements are not necessarily played in a loop for 24 hours, but are played at a certain time of the day, for example, from 07:00 to 24:00. When the image processing device 300M receives the pause/shutdown signal sent by the content server 870, it ends the process of the smart electronic signage application stage in FIG. 16; otherwise, it jumps back to step 1210 to continue execution.

Moreover, the execution of the above steps 1240, 1250, and 1270 is to determine whether the person in the picture is looking at the screen 890 on the audio-visual display device 850. If the person in the picture is not looking at the screen 890 on the audio-visual display device 850, then People in this situation in the picture will not be included in the statistics. According to another embodiment of the present invention, when it is not necessary to determine whether the person in the picture is looking at the screen 890 on the audio-visual display device 850, step 1240, step 1250, and step 1270 can be omitted. In addition, in this case, step 1280 is not required to determine whether the person in the picture looks at the screen 890 on the audio-visual display device 850 for a period of time that exceeds a preset time threshold. In this case, step 1280 will also be omitted simultaneously.

Furthermore, although step 910 and step 920 are shown in FIG. 14 to be executed in sequence, and the two sub-processes of FIG. 15 and FIG. 16 are used to represent the execution details of step 910 and step 920 respectively; however, according to another aspect of the present invention In an embodiment, step 910 and step 920 in FIG. 14 can also be executed independently in parallel, that is, the two sub-processes representing the execution content of step 910 and step 920 (ie, FIGS. 15 and 16) can be executed at the same time .

In the embodiment of the present invention, after the image processing device 300M obtains the real-time image data, it first identifies/analyzes the attributes of the person watching the audio-visual advertisement in the picture, and then performs a smart analysis comparison with the content attributes of each audio-visual advertisement. Yes, in order to get the latest video and audio advertisement play order and change the playback of the video and audio advertisement in real time, so the advertisement content of the latest video and audio advertisement is very likely to arouse the interest of the people watching the video and audio advertisement in the picture, and it also means that it is currently being broadcast. 'S video ads are all watched by the most relevant people at the moment. Therefore, the use of the smart electronic signage system and method disclosed in the present invention will maximize the advertising benefits of the audio-visual advertisement, so as to save advertising expenses. The foregoing descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention shall fall within the scope of the present invention.

40: gaze at sight 50: Target 60, 70: Video camera 100: Image data acquisition and analysis system 150: Cloud data storage unit 180, 190: Internet 300: image analysis server 300M: image processing device 310: Central Processing Unit 320: dynamic random access memory 330: Non-volatile memory 340: Read Only Memory 350: storage interface controller 360: network interface controller 370: Graphics Processing Unit 380: Physical Storage Device Array 385: physical storage device 390: Bus 400, 400A, 400B, 400C, 400N, 400M: image capture device 405: People Flow Analysis Module 410: Head Detection Module 415: visitor identification module 420: Commodity Detection Module 425: Gaze Area Analysis Module 430: Gaze Item Analysis Module 435: Interest Analysis Module 440: Moving Path Analysis Module 445: visitor attribute analysis module 450: database 455: Return customer analysis module 460: Image Analysis Application 465: Cloud Channel Service 465M: Multimedia player 470: Ultra Controller 475: Operating System 480: hardware 800A, 800B: head 805A, 805B, 805C, 805D: tangent 810: screen 815A, 815B: projection 820: Projection area (95% probability) 820E: Projection area (98% chance) 830: Projection area (65% chance) 840: Projection area (15% chance) 850: audio-visual display device 855: Cable 860: Network 870: Content Server 880: Media Player 890: screen 510, 520, 530, 550, 560, 570, 610, 620, 630, 640,700, 705, 710, 715, 720, 725, 772A, 722B, 722C, 730,735, 740, 745, 750, 755, 760,765, 900, 910, 920,1000, 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, 1090,1110, 1120, 1130, 1140, 1200, 1205, 1210, 1220, 1230, 1240, 1250, 1260, 1270, 1280, 1290, 1300, 1310, 1320, 1330: steps

FIG. 1A is a schematic diagram of detecting objects that a woman is looking at by using two cameras in the prior art. Fig. 1B is a schematic diagram of the face and eyes of a female when looking at an object recorded by Camera 1 and Camera 2 in Fig. 1A. FIG. 2 is a block diagram of the system architecture of the image data acquisition and analysis system according to an embodiment of the present invention. FIG. 3A is a block diagram based on the basic hardware architecture of the image analysis server in FIG. 2. FIG. 3B is a block diagram based on the structural relationship between the software and hardware of the image analysis server in FIG. 2. FIG. 4 is a block diagram of each functional module in the analysis and application of the image data in FIG. 3B. FIG. 5 is a flowchart of the overall application concept of recognizing people and objects in image data based on the CNN architecture according to an embodiment of the present invention. FIG. 6 is a flowchart based on the training phase shown in FIG. 5. FIG. 7A is a data set of people based on the CNN architecture in an embodiment of the present invention. FIG. 7B is a data set (1) of objects based on the CNN architecture in an embodiment of the present invention. FIG. 7C is a data set (2) of objects based on the CNN architecture according to an embodiment of the present invention. FIG. 8A is a schematic diagram of identifying the customer's face/head in the image picture according to the analysis of the image data and the application execution in an embodiment of the present invention. FIG. 8B is a schematic diagram of identifying the customer identity and product name in the image picture according to the analysis of the image data and the application execution in an embodiment of the present invention. FIG. 8C is a schematic diagram of identifying the product name in the image picture according to the analysis of the image data and the application execution in an embodiment of the present invention. FIG. 9 is a flow chart of analyzing the most interesting products of shopping customers in a store according to an embodiment of the present invention. FIG. 10 is a schematic diagram based on the information record disclosed in step 755 of FIG. 9. FIG. 11A is a schematic diagram of the relationship between the head direction (1), the head direction (2) of the person and the projected light in an embodiment of the present invention. FIG. 11B is a schematic diagram of a region formed between the direction of the human head and the projected light in FIG. 11A. 11C is a schematic diagram of the relationship between the calculation of the rotation angle of the human head direction (3) and the human head direction (3) and the projected light according to an embodiment of the present invention. FIG. 11D is a schematic diagram of the relationship between the calculation of the rotation angle of the person's head direction (4) and the person's head direction (4) and the projected light according to an embodiment of the present invention. FIG. 12A is a system architecture diagram of a smart electronic signage system according to an embodiment of the present invention. FIG. 12B is a schematic diagram of a scenario of a smart electronic signage system according to an embodiment of the present invention. FIG. 13A is a block diagram based on the basic hardware structure of the image processing device in FIG. 12A. FIG. 13B is a diagram based on the relationship between the software and hardware architecture of the image processing device in FIG. 12A. FIG. 14 is a flowchart of the overall operation of the smart electronic signage system in an embodiment according to the present invention. FIG. 15 is a flowchart based on the person detection/statistical analysis stage of the smart electronic signage system in FIG. 14. FIG. 16 is a flowchart based on the application phase of the smart electronic signage system in FIG. 14.

700,705,710,715,720, 725,722A,722B,722C,730, 735,740,745,750,755,760, 765: step

Claims (88)

A visitor's interest level analysis system is used to analyze a visitor's interest level in at least one item. The system includes: At least one image capturing device is set at a place to capture an image data of the place, the image data records a first head image of the visitor; and An image analysis server connected to the at least one image capture device for calculating and analyzing the image data from the at least one image capture device, the image analysis server further includes: A data processing center for executing an image analysis application program can determine a first head image corresponding to a first feature mapping (feature mapping) obtained from the first head image Head direction, and calculate a first projection area according to the first head direction; and A memory unit for temporarily storing the image data, the first head image, the first feature mapping, and other related data required or produced by the data processing center during the operation; Wherein, the calculation of the first projection area uses a virtual light source that is placed behind the visitor's head and projects light in a direction consistent with the direction of the first head to form the simulated first projection area When the first projection area covers the location of the at least one item, it means that the visitor has a degree of interest in the at least one item. The system according to claim 1, wherein the image analysis server further has a storage unit composed of at least one physical storage device for providing storage space of the image analysis server. The system according to claim 1, wherein the data processing center is composed of one or both of a central processing unit and a graphics processing unit. The system according to claim 1, wherein the first feature mapping (feature mapping) is obtained by the image analysis server by analyzing the relationship between pixels in the first head image, which can describe the first head image. A set of data on the characteristics of a head image. The system according to claim 1, wherein the first head direction refers to a degree of freedom of movement of the visitor's head in space in the first head image. The system according to claim 5, wherein the degree of freedom is expressed in three rotational dimensions of roll, pitch, and yaw. The system according to claim 5, wherein the degree of freedom is represented by a three-dimensional rectangular coordinate system representing front and rear, left and right, and up and down. The system according to claim 1, wherein the at least one image capturing device and the image analysis server are directly connected or connected through a network. The system according to claim 1, wherein the image data is only captured by a single lens of the at least one image capturing device. The system according to claim 1, wherein the image analysis application is a software or a module with both software and hardware for analyzing the image data. The system according to claim 1, which further includes a cloud data storage unit connected to the at least one image capture device and the image analysis server, and the cloud data storage unit is used to store the at least one image capture device The captured image data. The system according to claim 11, wherein the image analysis server further includes a cloud channel service application program as an intermediary service program for data transmission between the image analysis server and the cloud data storage unit. The system according to claim 11, wherein the image analysis server is a virtual machine located in the cloud data storage unit. The system according to claim 1, wherein the first projection area is the same as the cross-sectional area represented by the first head image in space. The system according to claim 1, wherein the first projection area is further divided into a plurality of sub-areas, and each of the plurality of sub-areas corresponds to a probability, which represents that when the position of the at least one item is covered by the sub-area, the at least one item Probability of being watched. The system according to claim 15, wherein the probability corresponding to each of the plurality of sub-areas represents the degree of interest of the visitor to the at least one item. The system according to claim 1, wherein it is further calculated whether the length of time that the first projection area covers the location of the at least one item exceeds a time threshold, as a method for judging whether the visitor has the degree of interest in the at least one item in accordance with. The system according to claim 1, wherein the position of the at least one item is obtained by referring to an input location data, or calculated by analyzing all or part of the image of the at least one item in the image data. The system according to claim 1, wherein words, numbers or symbols are added to the image data to indicate the identity (ID) or code of the visitor. The system according to claim 1, wherein words, numbers or symbols are added to the image data to indicate the name or code of the at least one item. The system according to claim 1, wherein the image data further includes all or part of the image of the at least one item, and a visual effect is added to or around all or part of the image of the at least one item for Highlight the visitor's degree of interest in the at least one item. A visitor interest level analysis system, connected to at least one image capturing module and receiving an image data from the at least one image capturing module, is used to analyze a visitor's degree of interest in at least one subject. The system includes: A data processing center executes an image analysis application program, which can determine a first head image corresponding to a first feature mapping (feature mapping) obtained from a first head image in the image data A head direction, and a first projection area is calculated according to the first head direction; and A memory unit for temporarily storing the image data, the first head image, the first feature mapping, and other related data required or produced by the data processing center during the operation; Wherein, the calculation of the first projection area uses a virtual light source that is placed behind the visitor's head and projects light in a direction consistent with the direction of the first head to form the simulated first projection area When the first projection area covers the location of the at least one target, it means that the visitor has a degree of interest in the at least one target. The system according to claim 22, wherein there is a storage unit composed of at least one physical storage device for providing storage space. The system according to claim 22, wherein the data processing center is composed of one or both of a central processing unit and a graphics processing unit. The system according to claim 22, wherein the first feature mapping (feature mapping) is obtained by the data processing center by analyzing the relationship between pixels in the first head image and can describe the first A set of data on the characteristics of the head image. The system according to claim 22, wherein the first head direction refers to a degree of freedom of movement of the visitor's head in space in the first head image. The system according to claim 26, wherein the degree of freedom is represented by three rotational dimensions of roll, pitch, and yaw. The system according to claim 26, wherein the degree of freedom is represented by a three-dimensional rectangular coordinate system representing front and rear, left and right, and up and down. The system according to claim 22, wherein the system is connected to the at least one image capturing module directly or through a network. The system according to claim 22, wherein the image data is only captured by a single lens of the at least one image capturing module. The system according to claim 22, wherein the image analysis application is a software or a module with both software and hardware for analyzing the image data. The system according to claim 22, wherein the system is further connected to the at least one image capturing device through a cloud data storage unit, and the cloud data storage unit is configured to store the at least one image capturing device captured by the at least one image capturing device. video material. The system according to claim 32, which further includes a cloud channel service application program as an intermediary service program for data transmission with the cloud data storage unit. The system according to claim 32, wherein the system is a virtual machine located in the cloud data storage unit. The system according to claim 22, wherein the first projection area is the same as the cross-sectional area represented by the first head image in space. The system according to claim 22, wherein the first projection area is further divided into a plurality of sub-areas, and each of the plurality of sub-areas corresponds to a probability, which represents that when the position of the at least one target is covered by the sub-region, the at least one target Probability of being watched. The system according to claim 36, wherein the probability corresponding to each of the plurality of sub-areas represents the degree of interest of the visitor to the at least one subject. The system according to claim 22, wherein it is further calculated whether the length of time that the simulated first projection area covers the at least one target position exceeds a time threshold value, to determine whether the visitor has the interest in the at least one target The basis of the degree. The system according to claim 22, wherein the position of the at least one target is obtained by referring to an input position data, or is calculated by analyzing all or part of the image of the at least one target in the image data. The system according to claim 22, wherein words, numbers or symbols are added to the image data to indicate the identity (ID) or code of the visitor. The system according to claim 22, wherein words, numbers, or symbols are added to the image data to describe the name or code of the at least one subject. The system according to claim 22, wherein the image data further includes all or part of the image of the at least one item, and a visual effect is added to or around all or part of the image of the at least one item, It is used to highlight the visitor's degree of interest in at least one target. According to claim 22, the system further includes: A content management module, which is connected to the data processing center and provides a stream of audiovisual data according to the analysis result of the data processing center, wherein the stream of audiovisual data further includes a content attribute; and A playback module is connected to the content management module and decodes the streaming video and audio data from the content management module, and transmits the decoded streaming video and audio data to a display device for playback. The system according to claim 43, wherein the display device is used to receive the streaming video and audio data decoded by the playback module. The system according to claim 43, wherein the display device includes a screen. The system according to claim 43, wherein the playback module and the data processing center are integrated. The system according to claim 43, wherein the content management module, the playback module, and the data processing center are integrated. The system according to claim 45, wherein the image capturing module is integrated in the display device. The system according to claim 43, wherein the content management module, the playback module, the image capturing module, and the data processing center are connected to each other through at least one network. The system according to claim 43, wherein the data processing center analyzes the number and attributes of the visitors in the image data, and adjusts the content of the streaming audio-visual data according to the analysis result. The system according to claim 43, wherein the data processing center determines whether the visitor is looking at the display device according to the first head direction, and adjusts the content of the streaming video and audio data according to the analysis result. The system according to claim 43, wherein the first projection area is further divided into a plurality of sub-areas, and each of the plurality of sub-areas corresponds to a probability, which represents that when the position of the display device is covered by the sub-area, the playback of the display device Probability of content being watched. The system according to claim 52, wherein the probability corresponding to each of the plurality of sub-regions represents the degree of interest of the visitor to the content played on the display device. The system according to claim 43, wherein it is further calculated whether the length of time that the simulated first projection area covers the position of the display device exceeds a time threshold to determine whether the visitor has the content of the display device. The basis for the degree of interest. The system according to claim 22, wherein the image capturing module captures the appearance of the visitor to generate the image data. The method according to claim 22, wherein the virtual light source is located behind the position of the first head image in a three-dimensional space about 3 times the diameter of the simulated sphere of the first head image. A method for analyzing the degree of interest of visitors. The method is executed by an image analysis server to determine the degree of interest of a visitor in at least one target. The method includes: Provide an image analysis application in the image analysis server; The image analysis server obtains an image data; The image analysis server detects a first head image with a first head feature in the image data; The image analysis server analyzes the first head image, and determines a first head direction corresponding to the first head image through a first feature mapping obtained by the analysis; The image analysis server calculates the position of the first head image in a three-dimensional space, and calculates a simulated first head image based on the position of the first head image, the first head direction, and a virtual light source. A projection area; and The image analysis server determines whether the visitor corresponding to the first head image has a degree of interest in the at least one object according to the coverage of the first projection area and the position of the at least one object. The method according to claim 57, wherein the image analysis application is a software or a module with both software and hardware for analyzing the image data. The method according to claim 57, wherein the method for the image analysis server to obtain the image data is that the image analysis server receives the image data transmitted by an image capture device through a network or directly. The method according to claim 59, wherein the image data is only captured by a single lens of the image capturing device. The method according to claim 57, wherein the image analysis server obtains the image data in a manner that the image analysis server downloads the image data from a cloud data storage unit through a network. The method according to claim 61, wherein a cloud channel service application is further provided as an intermediary service program for data transmission between the image analysis server and the cloud data storage unit. The method according to claim 57, wherein the image analysis server further includes: A data processing center for executing the image analysis application program can determine the first head image corresponding to the first head image based on the first feature mapping obtained from the first head image Head direction, and calculate the first projection area according to the first head direction; and A memory unit is used to temporarily store the image data, the first head image, the first feature mapping, and other related data required or produced by the data processing center during the operation. The method according to claim 63, wherein the data processing center is composed of one or both of a central processing unit and a graphics processing unit. The method according to claim 57, wherein the first feature mapping (feature mapping) is obtained by the image analysis server by analyzing the relationship between pixels in the first head image, which can describe the first head image. A set of data on the characteristics of a head image. The method according to claim 57, wherein the first head direction refers to a degree of freedom of movement of the visitor's head in space in the first head image. The method according to claim 66, wherein the degree of freedom is expressed in three rotation dimensions: roll, pitch, and yaw. The method according to claim 66, wherein the degree of freedom is represented by a three-dimensional rectangular coordinate system representing front and rear, left and right, and up and down. The method according to claim 57, wherein the first projection area and the first head image represent the same cross-sectional area in space. The method according to claim 57, wherein the first projection area is further divided into a plurality of sub-areas, and each of the plurality of sub-areas corresponds to a probability, which represents that when the position of the at least one target is covered by the sub-region, the at least one target Probability of being watched. The method according to claim 70, wherein the probability corresponding to each of the plurality of sub-areas represents the degree of interest of the visitor to the at least one subject. The method according to claim 57, wherein it is further calculated whether the length of time that the first projection area covers the position of the at least one target exceeds a time threshold, and is used to determine whether the visitor has the degree of interest in the at least one target in accordance with. The method according to claim 57, wherein the position of the at least one target is obtained by referring to an input position data, or is calculated by analyzing all or part of the image of the at least one target in the image data. The method according to claim 57, wherein words, numbers or symbols are added to the image data to indicate the identity (ID) or code of the visitor. The method according to claim 57, wherein words, numbers, or symbols are added to the image data to describe the name or code of the at least one subject. The method according to claim 57, wherein the image data further includes all or part of the image of the at least one item, and a visual effect is added to or around all or part of the image of the at least one item for Highlight the visitor's degree of interest in at least one target. The method according to claim 57, wherein the image analysis server is a virtual machine located in a cloud data storage unit. The method according to claim 57, wherein the image analysis server is further connected to a storage unit, and the storage unit is composed of at least one physical storage device for providing storage space of the image analysis server. The method according to claim 57, wherein the virtual light source is located behind the position of the first head image in a three-dimensional space about 3 times the diameter of the simulated sphere of the first head image. The method according to claim 57, wherein the at least one target is a display device. The method described in claim 80 further includes the following steps: When the image analysis server determines that the visitor has the degree of interest in the display device, it counts the number and attributes of the visitor; The image analysis server determines a stream of audiovisual data based on the statistical results of the number of visitors and their attributes, where the streamed audiovisual data further includes a content attribute; and The image analysis server notifies a content server and requests it to use the determined streaming video and audio data as the playback content of the display device. The method according to claim 81, wherein the display device further includes a multimedia player program for receiving and decoding the streaming video and audio data. The method according to claim 81, wherein the image analysis server further refers to the content attribute of the streaming video and audio data, and determines that the visitor has the degree of interest in the content attribute. The method described in claim 81 further includes the following steps: The image analysis server determines whether the visitor is looking at the display device according to the first head direction, and uses the result of statistical analysis as a basis for adjusting the streaming audio and video data. The method according to claim 81, wherein the first projection area is further divided into a plurality of sub-areas, and each of the plurality of sub-areas corresponds to a probability, which represents that when the position of the display device is covered by the sub-areas, the streaming audio-visual data Probability of being watched. The method according to claim 85, wherein the probability corresponding to each of the plurality of sub-regions represents the degree of interest of the visitor to the content played on the display device. The method described in claim 81 further includes the following steps: The image analysis server calculates whether the length of time that the first projection area covers the position of the display device exceeds a time threshold, as a basis for judging whether the visitor has the degree of interest in the playback content of the display device. The method according to claim 80, wherein the attributes of the visitor are composed of part or all of the following combinations, including: gender, age, height, posture, accessories, clothing, and occupation.

Images