WO2021020500A1

WO2021020500A1 - Information processing device and marketing activity assistance device

Info

Publication number: WO2021020500A1
Application number: PCT/JP2020/029208
Authority: WO
Inventors: 三郎山内
Original assignee: アースアイズ株式会社
Priority date: 2019-07-31
Filing date: 2020-07-30
Publication date: 2021-02-04
Also published as: JP6593949B1; JP2021026336A

Abstract

The present invention provides a means for efficiently extracting and analyzing only useful data from an enormous volume of recorded images. An information processing device 1 is provided with: a category designation unit 10; an object extraction unit 20 that extracts an object to be analyzed from recorded images 2; an object analysis unit 30 that analyzes the attribute and/or the motion of the object to be analyzed; an analysis unit 40 that analyzes the statistical amount of the attribute and/or the motion of the object to be analyzed; and a dashboard 50 that displays statistical data configured by including the analyzing result, wherein the extraction of the object to be analyzed by the object extraction unit 20 and the analysis of the attribute and/or the motion of the object to be analyzed by the object analysis unit 30 are executed by a machine-learning type image recognition means having a neural network.

Description

Information processing equipment and marketing activity support equipment

The present invention relates to an information processing device and a marketing activity support device. More specifically, the present invention relates to an information processing device that extracts useful information from a recorded image, analyzes and displays it, and a marketing activity support device including the information processing device.

A device for market research is being developed to detect the behavior of the purchaser from the surveillance image taken inside the store and acquire what kind of product the purchaser is interested in as marketing data (patent). Reference 1).

In addition, in order to make more effective use of the acquired marketing data, it is determined which customer group (customer attribute) the purchaser belongs to, and further, the area analysis data of the customer group is acquired to obtain the customer attribute. A system has also been proposed that enables the planning of marketing strategies that take into consideration the region and region. (See Patent Document 2).

Alternatively, by detecting and collating each product in the image such as a display shelf using image recognition technology, it is possible to improve the efficiency of inventory management and assist the customer to easily shop in the actual store. An image recognition system to be used has also been proposed (see Patent Document 3).

Japanese Unexamined Patent Publication No. 2006-293786 JP-A-2009-151408 Japanese Unexamined Patent Publication No. 2014-218318

The device described in Patent Document 1 is useful as a proposal for a process of extracting marketing data from an image. The system described in Patent Document 2 is also useful as a specific means of utilizing the data obtained in this way. Further, the system described in Patent Document 3 is also useful as a means for grasping the status of inventory and the like, which changes from moment to moment, in real time by image processing technology.

Here, in recent years, with the increase in the number of surveillance cameras for crime prevention installed in various places in public spaces, the amount of recorded images accumulated has become enormous. However, for example, a means for efficiently extracting and analyzing only useful data from these enormous recorded images by going back to a certain period (several days to several years) in the past is mentioned in any of the above documents. It has not been.

It has been desired to develop a means for efficiently extracting and analyzing useful information in a short time from the huge amount of accumulated images as described above. An object of the present invention is to provide a means for efficiently extracting and analyzing only useful data from a huge amount of recorded images.

The present invention solves the above-mentioned problems by the following solutions. In addition, in order to facilitate understanding, the description will be given with reference numerals corresponding to the embodiments of the present invention, but the present invention is not limited thereto.

(1) A category designation unit that can specify a specific analysis target category, and an object extraction unit that extracts analysis target objects belonging to the analysis target category specified by the category designation unit from recorded images. , An object analysis unit that analyzes the extracted attributes and / or movements of the analysis target object, and an analysis unit that analyzes the analyzed attributes and / or movement statistics of the analysis target object, and the analysis unit and the statistics. A dashboard that displays statistical data including analysis results is provided, and the analysis target object is extracted by the object extraction unit, and the attributes and / or movement of the analysis target object by the object analysis unit. The analysis is executed by a machine learning type image recognition means having a neural network, and the object analysis unit has a position in the recorded image which is a two-dimensional image and a three-dimensional image to be recorded. An information processing device including a coordinate setting unit that sets coordinates associated with an actual position in space in the recorded image.

The invention of (1) omits the process of visually reproducing a huge amount of recorded images in extracting and analyzing image data for obtaining useful data from a huge amount of recorded images, and has a neural network. It was decided to perform the above extraction and analysis completely and automatically using a machine learning type image recognition means (so-called deep learning type image recognition means). This makes it possible to obtain useful data that has been processed into a format that is easy for the user to understand in a short processing time. Further, according to the invention of (1), for example, even from an image having only two-dimensional information acquired only by a monocular camera that can be acquired at a low cost without introducing a distance measuring device, a 3D camera, or the like. The motion analysis of the object to be analyzed can be efficiently executed with high accuracy only by the automatic processing by the coordinate setting unit.

(2) The recorded image is input to the object extraction unit as digital data, and the analysis target object is directly extracted from the digital data without going through a conversion process into a two-dimensional image that can be visually recognized by humans. The information processing apparatus according to (1).

The invention of (2) has a configuration in which no work that requires human visibility is involved in the process of extracting and analyzing necessary data from the image in the invention of (1). This makes it possible to obtain useful data that has been processed into a format that is easy for the user to understand in an extremely short processing time.

(3) The object analysis unit is configured to include a face recognition information acquisition unit capable of analyzing the age and gender of the person from image information related to the person's face, according to (1) or (2). The information processing device described.

The invention of (3) is configured to further include a face recognition information acquisition unit that acquires unique face recognition information of the object (person) to be analyzed in the invention of (1) or (2). As a result, the analysis of the attributes of the object to be analyzed can be efficiently executed with high accuracy only by automatic processing.

(4) The object analysis unit is configured to include a skeleton extraction unit that extracts the skeleton of the object to be analyzed, which is composed of skeleton lines connecting a plurality of feature points, and is individually composed of changes in the positions of the feature points. The information processing apparatus according to any one of (1) to (3), which recognizes the movement of the object to be analyzed.

In the invention of (4), for example, by using an image analysis means such as "OpenPose" described later, a plurality of feature points of the analysis target object (person) are connected, especially when the analysis target object is a person. By extracting the skeleton and analyzing the position and speed of each of these feature points, the movement of the object to be analyzed can be recognized. According to this, various movements of the analysis target object can be recognized with higher accuracy regardless of the body shape (shape) of the analysis target object.

(5) A marketing activity support device according to any one of (1) to (4), wherein the statistical data is marketing data.

According to the invention of (5), useful marketing data processed into a format that is easy for the user to understand can be obtained in an extremely short processing time from the enormous amount of image information that has already been accumulated.

(6) A category designation unit that can specify a specific analysis target category, and an object extraction unit that extracts analysis target objects belonging to the analysis target category specified by the category designation unit from recorded images. An object analysis unit that analyzes the attributes and / or movements of each of the extracted objects to be analyzed, and an analysis unit that analyzes the analyzed attributes and / or movement statistics of the analysis target object. A machine equipped with a dashboard for displaying statistical data including analysis results of statistics, and the extraction by the object extraction unit and the analysis by the object analysis unit both have a neural network. Executed by a learning type image recognition means, the object analysis unit obtains coordinates that relate a position in the recorded image, which is a two-dimensional image, to an actual position in the three-dimensional space to be recorded. An information processing system including a coordinate setting unit set in the recorded image.

The invention of (6) omits the process of visually reproducing a huge amount of recorded images in extracting and analyzing image data for obtaining useful data from a huge amount of recorded images, and has a neural network. It was decided to perform the above extraction and analysis completely and automatically using a machine learning type image recognition means (so-called deep learning type image recognition means). This makes it possible to obtain useful data that has been processed into a format that is easy for the user to understand in a short processing time. Further, according to the invention of (6), for example, even from an image having only two-dimensional information acquired only by a monocular camera that can be acquired at a low cost without introducing a distance measuring device, a 3D camera, or the like. The motion analysis of the object to be analyzed can be efficiently executed with high accuracy only by the automatic processing by the coordinate setting unit.

(7) A marketing activity support system according to (6), wherein the statistical data is marketing data.

According to the invention of (7), useful marketing data processed into a format that is easy for the user to understand can be obtained in an extremely short processing time from the enormous amount of image information that has already been accumulated.

(8) In the category designation section, the category designation step for designating a specific analysis target category and the object extraction section extract the analysis target object belonging to the analysis target category specified in the category designation step from the recorded image. , The object extraction step and the object analysis unit analyze the attributes and / or movements of the extracted individual objects to be analyzed, and the object analysis step and the analysis unit analyze the attributes and / or movement statistics of the analyzed objects. The analysis step for analyzing the quantity and the statistical data display step for displaying the statistical data in which the dashboard includes the analysis result of the statistic are provided, and the analysis target object by the object extraction step is provided. Both the extraction and the analysis of the attributes and / or movements of the analysis target object by the object analysis step are executed by the machine learning type image recognition means having a neural network, and in the object analysis step, the coordinate setting unit Is an information processing method in which coordinates for associating a position in the recorded image, which is a two-dimensional image, with an actual position in the three-dimensional space to be recorded are set in the recorded image.

The invention of (8) omits the process of visually reproducing a huge amount of recorded images in extracting and analyzing image data for obtaining useful data from a huge amount of recorded images, and has a neural network. It was decided to perform the above extraction and analysis completely and automatically using a machine learning type image recognition means (so-called deep learning type image recognition means). This makes it possible to obtain useful data that has been processed into a format that is easy for the user to understand in a short processing time.

(9) In the object extraction step, the recorded image is input to the object extraction unit as digital data, and the digital data is converted from the digital data to a two-dimensional image that can be visually recognized by humans. The information processing method according to (8), wherein the object to be analyzed is directly extracted.

The invention of (9) omits the process of visually reproducing a huge amount of recorded images in extracting and analyzing image data for obtaining useful data from a huge amount of recorded images, and has a neural network. It was decided to perform the above extraction and analysis completely and automatically using a machine learning type image recognition means (so-called deep learning type image recognition means). This makes it possible to obtain useful data that has been processed into a format that is easy for the user to understand in a short processing time.

(10) The information processing method according to (8) or (9), wherein in the object analysis step, the face recognition information acquisition unit analyzes the age and gender of the person from the image information related to the person's face.

The invention of (10) further includes a face recognition information acquisition unit that acquires unique face recognition information when the object to be analyzed is a person in the invention of (9) or (10). As a result, the analysis of the attributes of the object to be analyzed can be efficiently executed with high accuracy only by automatic processing.

(11) In the object analysis step, the skeleton extraction unit extracts the skeleton of the analysis target object composed of skeleton lines connecting a plurality of feature points, and the individual analysis target objects are extracted from the position fluctuations of the feature points. The information processing method according to any one of (8) to (10), wherein the movement of is recognized.

In the invention of (11), for example, by using an image analysis means such as "OpenPose" described later, a skeleton formed by connecting a plurality of feature points of an object to be analyzed is extracted, and the position and velocity of each feature point are extracted. By analyzing the above, the movement of the object to be analyzed) can be accurately recognized. According to this, various movements of the analysis target object can be recognized with higher accuracy regardless of the body shape (shape) of the analysis target object.

(12) A marketing activity support method according to any one of (12) (8) to (11), wherein the statistical data is marketing data.

According to the invention of (12), useful data processed into a format that is easy for the user to understand can be obtained in an extremely short processing time from the enormous amount of image information that has already been accumulated.

(13) A category designation unit that can specify a specific analysis target category, and an object extraction unit that extracts analysis target objects belonging to the analysis target category specified by the category designation unit from recorded images. , An object analysis unit that analyzes the extracted attributes and / or movements of the analysis target object, and an analysis unit that analyzes the analyzed attributes and / or movement statistics of the analysis target object, and the analysis unit and the statistics. The object analysis unit includes a dashboard for displaying statistical data including analysis results, and the object analysis unit has a position in the recorded image, which is a two-dimensional image, and a three-dimensional space to be recorded. In an information processing apparatus that includes a coordinate setting unit that sets coordinates associated with an actual position in the recorded image in the recorded image, the analysis belonging to the specific analysis target category from the recorded image. An object extraction step of extracting a target object by a machine learning type image recognition means having a neural network, and a machine learning type image having a neural network of the attributes and / or movements of each extracted object to be analyzed. Display on the dashboard statistical data including the object analysis step analyzed by the recognition means, the analysis step for analyzing the analyzed attribute and / or motion statistics, and the analysis result of the statistics. A program for causing the information processing apparatus to execute the statistical data display step.

The invention of (13) omits the process of visually reproducing a huge amount of recorded images in extracting and analyzing image data for obtaining useful data from a huge amount of recorded images, and has a neural network. It was decided to perform the above extraction and analysis completely and automatically using a machine learning type image recognition means (so-called deep learning type image recognition means). This makes it possible to obtain useful data that has been processed into a format that is easy for the user to understand in a short processing time.

(14) A program for supporting marketing activities according to (13), wherein the statistical data is marketing data.

According to the invention of (14), useful data processed into a format that is easy for the user to understand can be obtained in an extremely short processing time from the enormous amount of image information that has already been accumulated.

According to the present invention, it is possible to provide a means for efficiently extracting and analyzing only useful data from a huge amount of recorded images.

It is a block diagram which shows the structure of the information processing apparatus of this invention. It is a figure which shows typically an example of the dashboard which comprises the information processing apparatus of this invention. It is a figure which shows the state which the analysis target object (person H and object M) of a recorded image is extracted by the object extraction unit provided in the information processing apparatus of this invention. It is a figure which shows the state which the feature point of the skeleton of the object to be analyzed is superposed on the coordinates including three-dimensional information (depth information) by the object analysis part provided in the information processing apparatus of this invention. It is a figure which shows the state which the movement of the analysis target object is recognized based on the information about the change of the position of the feature point. It is a figure which shows the state of the velocity vector of the object to be analyzed analyzed by the object analysis part.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings as appropriate.

<Information processing device (marketing activity support device)>
The information processing device of the present invention is an information processing technique that can be widely applied to all information processing devices that take a recorded image as input data and output the analysis result in an arbitrary format that is easy for the user to use. According to the information processing apparatus of the present invention, by selecting and designating an arbitrary target category from various objects included in the recorded image, useful analysis results for the category can be automatically obtained.

Such an information processing apparatus of the present invention is useful for marketing activities (for example, a specific area) from a huge amount of recorded images taken by a surveillance camera or the like installed in a public space. An embodiment used as a "marketing activity support device" that extracts, analyzes, and displays the traffic volume of a person with a specific attribute and the degree of attention to a specific item in the above can be mentioned as an example of the preferred embodiment. Hereinafter, an embodiment in which the information processing device of the present invention is used as a “marketing activity support device” will be described in detail as the best mode of the present invention.

[overall structure]
The marketing activity support device 1 uses a recorded image 2 that has been recorded for a certain period from the past to the present and is held in a playable state as input data for analysis. Then, statistical data that can be obtained from the recorded image 2 and is useful in marketing activities is output as analysis result (marketing data) 3 as audiovisual information that is easy for humans to understand. The "marketing data" in the present specification is statistical data that can be obtained by analyzing a characteristic quantity related to the movement of a person or an object within a predetermined area, and is used for marketing activities. So, it refers to any data that can be used as a basis for judgment or reference information.

The basic configuration of the marketing activity support device 1 is as shown in FIG. The marketing activity support device 1 has a category designation unit 10 that can select and specify a specific analysis target category, an object extraction unit 20 that extracts individual objects to be analyzed from the recorded image 2, and an object extraction unit 20. The object analysis unit 30 that analyzes the attributes and / or movements of the individual analysis target objects extracted by, the analysis unit 40 that analyzes the attributes and / or movement statistics of the analyzed analysis target objects, and the statistics. It is configured to include a dashboard 50 that displays marketing data, which is statistical data that includes analysis results. Hereinafter, in the present specification, the category designation unit 10, the object extraction unit 20, the object analysis unit 30, and the analysis unit 40 are collectively referred to as a “calculation processing unit”.

The "calculation processing unit" is connected to a playback device or the like capable of outputting the image data of the recorded image 2 so that the image data of the recorded image 2 can be input. This connection can be a wired connection using a dedicated communication cable or a wired LAN connection. Further, the connection is not limited to a wired connection, and may be a connection using various wireless communications such as a wireless LAN, short-range wireless communication, and a mobile phone line.

In the marketing activity support device 1, the recorded image 2 is sent to the arithmetic processing unit (object extraction unit 20) not as a visible image form but as a digital data form that can be arithmetically processed by an information processing device. It is preferable that the configuration is such that it is directly input. By configuring the analysis target object to be directly extracted from the input data in the digital format without going through the conversion process to the two-dimensional image format that can be seen by humans and the display process of such a two-dimensional image. Objects to be analyzed can be automatically extracted from a huge amount of image data in a shorter time.

Then, in the marketing activity support device 1, among the above-mentioned units constituting the arithmetic processing unit, the object extraction unit 20 and the object analysis unit 30 are all machine learning type image recognition means (so-called) having a neural network. , Deep learning type image recognition means), the processing related to each extraction and analysis is executed. The details of the operation of each of these parts will be described later.

Further, in the marketing activity support device 1, at least the component including the dashboard 50 for displaying the analysis result to the user is arranged as an independent device in a different place away from the other components. Moreover, both of these components can be implemented as a form of a decentralized "marketing activity support system" in which both of these components are connected by a wired or wireless line as illustrated above.

Alternatively, in the marketing activity support device 1, the component including the dashboard 50 described above is composed of a plurality of information processing terminals, and the function of one arithmetic processing unit is shared by the plurality of dashboards 50. It can also be implemented as a form of "activity support system". For example, a part or all of the plurality of dashboards 50 may be a small portable information processing terminal. By implementing each of these forms, each part constituting the marketing activity support device 1 can be distributed and arranged in an optimum location in consideration of economic efficiency, user convenience, and the like. ..

[Recorded image]
The recorded image 2 input to the marketing activity support device 1 is not limited to an image having a specific content, format, and amount of information. Any image can be used that contains data that may be suitable for the intended use by data analysis. Surveillance images in public spaces and the like, where an enormous amount of images have been accumulated in recent years, are an example of an optimal data source for recorded images 2. Such a monitoring image is a treasure trove of marketing data in which a large amount of images capable of grasping the flow of people, the movement of products, the movement of a clerk, etc. are accumulated, and by using the marketing activity support device 1, This can be used efficiently and effectively.

The marketing activity support device 1 obtains only useful analysis results for an image containing a huge amount of data at random, such as the above-mentioned monitoring image, without performing a reproduction process that can be visually recognized by an analysis worker. be able to. Therefore, it is possible to obtain only useful information for marketing activities without infringing on the privacy of the photographed person to be extracted.

[Calculation processing unit]
The "arithmetic processing unit" including the category designation unit 10, the object extraction unit 20, the object analysis unit 30, and the analysis unit 40 can be configured by using, for example, a personal computer, a tablet terminal, a smartphone, or the like. .. Alternatively, the "arithmetic processing unit" can be configured by a dedicated device specialized for image processing operations. In any of these configurations, the "arithmetic processing unit" includes hardware such as a CPU, memory, and communication unit.

Then, the "arithmetic processing unit" having the above configuration concretely executes various operations of the marketing activity support device and the marketing activity support method described below by executing the "program" for the computer. Can be done.

[Category designation section]
The category designation unit 10 designates a specific analysis target category to be analyzed by the analysis unit 40. This designation may be configured to manually set an arbitrary target each time the worker uses the marketing activity support device 1. Alternatively, the category designation unit 10 may be configured such that a specific analysis target category is set in advance by default, and the setting is manually changed only when necessary. In any case, the analysis target category selected in the category designation unit 10 is transmitted to the object extraction unit 20 as a command, and the objects belonging to the analysis target category are extracted from the recorded image 2 according to the command.

For example, when the object extraction unit 20 can recognize "people" and "bottles" individually, and intends to obtain marketing data by analyzing and analyzing their movements and attributes. If so, the analysis target category 1 may be designated as "person" and the analysis target category 2 may be designated as "bottle" in the category designation unit. Alternatively, if the object extraction unit 20 has a function capable of individually recognizing the gender and age of a person by the face recognition function, for example, in the category designation step, the analysis target category is set to "female in her thirties". It is also possible to specify.

[Object extraction unit]
The object extraction unit 20 extracts the analysis target object belonging to the analysis target category designated by the category designation unit 10 from the recorded image 2. The extraction by the object extraction unit 20 is executed at high speed by a machine learning type image recognition means having a neural network, a so-called deep learning type image recognition means.

The object extraction unit 20 analyzes the "people and animals and plants" and "objects" (hereinafter, collectively referred to as "objects") existing in the recorded image 2 and designated by the category designation unit 10. Objects belonging to the target category (objects to be analyzed) are extracted by deep learning type image recognition means. For example, as shown in FIG. 3, the object extraction unit 20 extracts the analysis target objects (person H and object M) existing in the recorded image 2. However, FIG. 3 is a conceptual diagram relating to extraction, and actually reproducing such an image in a real-time visible state is not an indispensable constituent requirement in the apparatus, system, and method of the present invention.

The algorithm of the image recognition processing means for extracting the analysis target object from the recorded image 2 is not particularly limited, but "You only look once (YOLO)" can be preferably used. For example, by using "You only view (YOLO)" as an image recognition means for extracting and specifying an analysis target object in the object extraction unit 20, for example, about 1000 types of analysis target objects can be individually processed in parallel at the same time. It is also possible to extract. According to this, only useful objects required for analysis at the present time are extracted from a huge amount of image information accumulated over a certain period of time in the past at high speed and more accurately than manual work by human visual inspection. be able to.

[Object Analysis Department]
The object analysis unit 30 analyzes the attributes and / or movements of the individual objects to be analyzed extracted by the object extraction unit 20, and the analysis by the object analysis unit 30 is also a machine learning type image recognition means having a neural network. It is executed by a so-called deep learning type image recognition means. An image recognition technique using a machine learning type image recognition means (image recognition means using deep learning) having a neural network is disclosed below, for example.
"Deep Learning and Image Recognition, Operations Research"
(Http: //www.orsj.o.jp/archive2/or60-4/or60_4_198.pdf)

Further, it is preferable that the object analysis unit 30 further includes a face authentication information acquisition unit 31, a coordinate setting unit 32, and a skeleton extraction unit 33 as its internal configuration.

The algorithm of the machine learning type image recognition means (image recognition means using deep learning) having a neural network that analyzes the attributes and / or movements of the object to be analyzed is not limited to a specific algorithm. However, when the object analysis unit 30 is configured to include the skeleton extraction unit 33, a technique called "OpenPose" disclosed in the following document is used as the algorithm of the image recognition processing means for extracting the skeleton. Is preferable.
"Zhe Cao et al. Realtime Multi-Person 2D Human Pose Estimation using Part Affinity Fields, CVPR 2017"

(Face recognition information acquisition department)
It is preferable that the object analysis unit 30 includes a face recognition information acquisition unit 31 capable of analyzing the age and gender of the person from the image information related to the person's face. As the face recognition information acquisition unit 31, various conventionally known face recognition information acquisition devices can be used. By providing the face authentication information acquisition unit 31 in the object analysis unit 30, when the analysis target category is "person", the attributes such as age and gender of the analysis target object (person) belonging to the analysis target category can be highly accurate. Can be analyzed automatically with.

(Coordinate setting part)
The object analysis unit 30 preferably includes a coordinate setting unit 32 that sets identifiable coordinates by associating the position of the object to be analyzed in the image with the actual position in the three-dimensional space to be recorded. .. As a result, even when the recorded image 2 is composed of image data having only two-dimensional information, the analysis of the movement of the object to be analyzed can be efficiently processed with high accuracy only by the automatic processing by the coordinate setting unit 32. Can be executed.

The coordinate setting unit 32 performs a process of setting identifiable coordinates by associating the position corresponding to the floor surface in the image of the recorded image 2 which is a two-dimensional image with the actual size. The coordinates set by the coordinate setting unit 32 are, when a certain arbitrary position is specified in the recorded image 2 and the position is on the floor surface, the floor surface is in the space of the actual monitoring area. It is a coordinate that can identify which position it corresponds to. That is, the position on the coordinate to be set is set in association with the actual size. Since the recorded image 2 captured by the photographing unit 120 is two-dimensional image information, even if a position in the recorded image 2 is selected (specified), it is any position in the actual three-dimensional space. I can't identify. However, if the position to be selected (specified) in the recorded image 2 is limited to the floor surface after setting the coordinates associated with the actual dimensions on the floor surface, it is selected (specified) in the recorded image 2. ) It becomes possible to identify the position of the floor surface in the real space (monitoring area). Therefore, the coordinate setting unit 32 sets the coordinates corresponding to the floor surface.

(Skeletal extraction section)
Further, as described above, the object analysis unit 30 preferably includes a skeleton extraction unit 33 that extracts the skeleton of the object to be analyzed composed of skeleton lines connecting a plurality of feature points. The skeleton extraction unit 33 is composed of a plurality of feature points and a skeleton line connecting the plurality of feature points of the object to be analyzed (for example, the person H and the object M in FIG. 3) extracted by the object extraction unit 20. The process of extracting the skeleton of each analysis target is performed.

In the present specification, the "skeleton" of the object to be analyzed is a linear figure formed by connecting a plurality of feature points of the object to be analyzed. FIG. 4 is a diagram showing a state in which the skeleton is extracted from the person H to be analyzed. In FIG. 4, the positions corresponding to the crown of the person H, the left hand H ₂ , and the tips of the other limbs and the main joints, which are the objects to be analyzed, are grasped as feature points (h ₁ , ..., H _n ). The "skeleton" of the analysis target H formed by these plurality of feature points and the line segments connecting them is recognized as the "skeleton" of the analysis target object in the recorded image 2. ..

When the object analysis unit 30 includes the skeleton extraction unit 33, it is possible to recognize the "movement" of the object to be analyzed based on the information related to the fluctuation of the "position" of the feature points constituting the extracted skeleton. The "movement" referred to here includes all movements of the analysis target object that can be grasped by the position change of the feature point of the skeleton, such as the position change of the analysis target object and the posture change without the position change. (See FIGS. 5 and 6).

Further, when the analysis target category includes "person" and "object", the difference between the velocity vector of the feature points constituting the skeleton of "human" and the velocity vector of the feature points constituting the skeleton of "object". Is used as an input value, and the characteristics related to the movement of the analysis target can also be analyzed by comparing this input value with a predetermined threshold value. This makes it possible to comprehensively analyze the movement of the object to be analyzed, including the movement of the "person" with respect to the "object", such as "a person grabs an object and takes it away as it is" (see FIGS. 5 and 6). ).

Specifically, the skeleton of the object to be analyzed can be extracted by any of various conventionally known methods or a combination thereof. As an example, by using the above-mentioned "OpenPose", the skeleton of "human" can be extracted from the two-dimensional recorded image 2.

In FIG. 4, for the person H and the object M extracted as the objects to be analyzed by the object extraction unit 20, the coordinate points (h1, h2, ... h5) and (m1) of the extracted skeletons are set. It is a figure which shows the state which is superposed on the coordinates including the three-dimensional information (depth information) set by the part 32.

Then, if the standing position of the person H can be specified on the coordinates including the three-dimensional information (depth information), for example, from the size and shape of the person H in the recorded image 2, the person H in the actual three-dimensional space It is possible to calculate and grasp the actual size and three-dimensional shape of. That is, it is possible to acquire three-dimensional data related to the position and movement of the person H and the object M from the two-dimensional image data (position information of the feature points superimposed on the coordinates in the recorded image 2).

FIG. 6 is a diagram showing a state in which the movement of each analysis target object is recognized based on the information related to the fluctuation of the three-dimensional position of each feature point constituting the skeleton of each analysis target object. Here, the position of the left hand H2 of the person H specified as the analysis target object is from the position h2 ₀ (xh2 ₀ , yh2 ₀ , zh2 ₀ ) to the position h2 ₁ (xh2 ₁ , yh2 ₁ , yh2 ₁ ) in the actual three-dimensional space. zh2 ₁₎ it has moved in, and the position _m1 0 for M thing is also analyzed object _{_{(xm1 0, ym1 0, zm1}} 0) that is stationary and movement of the left hand H2 of the "human H a position h2 ₁ after, and the position m1 ₀ of the object M, that are consistent in actual three-dimensional space "is recognized.

Note that the object analysis unit 30 can also detect the line-of-sight direction of the monitored object (person) by including the skeleton extraction unit 33. For example, from the triangular "three-dimensional position information" formed by connecting the "feature points for detecting the direction of the line of sight of three places" corresponding to the positions of both ears and the nose of the object (person) to be monitored, the monitoring target is concerned. The line-of-sight direction of an object (person) can be detected. Specifically, the direction from the midpoint of the base, which connects the points corresponding to the positions of both ears in the above triangle, to the apex, which is the point corresponding to the position of the nose, is within the three-dimensional space of the person to be monitored. It can be detected as the line-of-sight direction at. The line-of-sight direction detection is not limited to the above method, and other conventionally known line-of-sight detection means can be appropriately combined with the present invention.

[Analysis Department]
The analysis unit 40 analyzes the attribute and / or movement statistic of the object to be analyzed analyzed by the object analysis unit 30 and converts it into data. Then, the numerical data related to the statistics obtained as the analysis result of the attributes and movements of the object to be analyzed is output to the dashboard 50. As a specific example of the statistic that is the analysis result of the object to be analyzed, for example, the flow and residence time of people by age and gender at a specific position in a specific sales floor, and the sales of products displayed at that position. Correlation with and the like can be mentioned.

The analysis unit 40 may be configured in a server independent of the dashboard, or may be configured on the client side by using a personal computer, a tablet terminal, a smartphone, or the like. In any configuration, the analysis unit 40 performs the above analysis process by providing hardware such as a CPU, a memory, and a communication unit.

[Dashboard]
The dashboard 50 displays statistical data (for example, 3a to 3e in FIG. 2) including the analysis result of the statistic by the analysis unit 40. The dashboard 50 is a device that visualizes marketing data with a graph or the like for analysis and displays management numerical values or the like in an easy-to-analyze manner.

The dashboard 50 includes a commercially available desktop personal computer in which a business application for managing marketing data (for example, a Web application) is installed, a commercially available notebook personal computer, a PDA (Personal Digital Assistants), a smartphone, and a tablet personal computer. It can also be configured by a portable information processing device such as a computer.

[Operation of marketing activity support device]
The information processing method (marketing activity support method) of the present invention executed by the operation of the marketing activity support device 1 includes a category designation step executed by the category designation unit 10, an object extraction step executed by the object extraction unit 20, and an object. The object analysis step executed by the analysis unit 30, the analysis step executed by the analysis unit 40, and the statistical data display step executed by the dashboard are sequentially performed to be executed as an entire process.

In the information processing method (marketing activity support method) of the present invention, among the above steps, the object extraction step and the object analysis step are machine learning type image recognition means having a neural network (deep learning type image recognition). Means).

(Category specification step)
In the category designation step, the analysis unit 40 designates a specific analysis target category to be analyzed. The analysis target category is specified by the object extraction unit 20 selecting an arbitrary category that can be individually classified and extracted in the recorded image 2. As described above, the analysis target category can be specified by manually designating an arbitrary target each time the worker uses the marketing activity support device 1, or can be specified in advance. It is also possible to set the analysis target category of the above by default, and manually change and specify the setting only when necessary.

(Object extraction step)
In the object extraction step, the object extraction unit 20 extracts individual analysis target objects belonging to the analysis target category specified in the category designation step from the recorded image 2. As described above, the object extraction step is executed by a machine learning type image recognition means having a neural network.

(Object analysis step)
In the object analysis step, the attributes and / or movements of the individual analysis target objects extracted in the object extraction step are analyzed by the object analysis unit 30. The object analysis step is also performed by a machine learning type image recognition means having a neural network.

In the object analysis step, it is preferable that the coordinate setting process by the coordinate setting unit 32 is performed in advance prior to the analysis of the object to be analyzed. However, if the recorded image 2 is given three-dimensional information in advance by a distance measuring means such as a distance sensor or a 3D camera, the coordinate setting process by the coordinate setting unit 32 is not always performed in the monitoring method of the present invention. This is not a required process.

(Analysis step)
In the analysis step, the analysis unit 40 analyzes the attribute and / or motion statistics analyzed in the object analysis step for the object to be analyzed extracted in the object extraction step.

(Statistical data display step)
In the statistical data display step, statistical data including the analysis result of the statistic analyzed by the analysis unit 40 is displayed on the dashboard 50.

In the information processing method (marketing activity support method) of the present invention, an appropriate analysis target category may be appropriately specified in the analysis target category designation step according to the content of the statistical data to be finally displayed on the dashboard 50. Good.

For example, if you want to obtain the analysis result of the movement of the clerk of the store, specify "person" as the analysis target category and register the unique biometric information such as the face authentication information of the clerk in the object analysis department in advance. By setting, only the clerk is identified from the extracted "people", the movement is analyzed, and the analyzed movement is statistically analyzed, so that the movement of the clerk of the store can be obtained from the recorded image. The analysis result of the above can be obtained, and this result can be displayed on the dashboard 50 in an arbitrary format such as a graph that is easy for the user to understand. Alternatively, by analyzing the attributes and movements of a specific item (product) and a "person" located near the item, statistical data related to the flow of people in the vicinity of the item (specific attribute at a specific place). The passing rate of people, average staying time, etc.) can be displayed in the same easy-to-understand graph or table format as above.

1 Information processing device (marketing activity support device)
10 Category designation unit 20 Object extraction unit 30 Object analysis unit 31 Face recognition information acquisition unit 32 Coordinate setting unit 33 Skeleton extraction unit 40 Analysis unit 50 Dashboard 2 Recorded

images

3, 3a, 3b, 3c Analysis results (marketing data)

Claims

A category specification section that allows you to specify a specific analysis target category,
An object extraction unit that extracts an analysis target object belonging to the analysis target category designated by the category designation unit from the recorded image, and an object extraction unit.
An object analysis unit that analyzes the attributes and / or movements of the extracted object to be analyzed.
An analysis unit that analyzes the analyzed attributes and / or motion statistics of the analyzed object,
A dashboard and a dashboard that displays statistical data composed of the analysis results of the statistics.
With
The extraction of the analysis target object by the object extraction unit and the analysis of the attributes and / or movements of the analysis target object by the object analysis unit are both executed by a machine learning type image recognition means having a neural network. ,
The object analysis unit sets coordinates in the recorded image that associate the position in the recorded image, which is a two-dimensional image, with the actual position in the three-dimensional space to be recorded. It is composed of parts,
Information processing device.
The recorded image is input to the object extraction unit as digital data, and the object to be analyzed is directly extracted from the digital data without going through a conversion process into a two-dimensional image that can be seen by humans.
The information processing device according to claim 1.
The object analysis unit includes a face recognition information acquisition unit that can analyze the age and gender of the person from image information related to the person's face.
The information processing device according to claim 1 or 2.
The object analysis unit is configured to include a skeleton extraction unit that extracts the skeleton of the object to be analyzed, which is composed of skeleton lines connecting a plurality of feature points, and the individual analysis is performed from the position variation of the feature points. Recognize the movement of the target object,
The information processing device according to any one of claims 1 to 3.
A marketing activity support device according to any one of claims 1 to 4, wherein the statistical data is marketing data.
A category specification section that allows you to specify a specific analysis target category,
An object extraction unit that extracts an analysis target object belonging to the analysis target category designated by the category designation unit from the recorded image, and an object extraction unit.
An object analysis unit that analyzes the attributes and / or movements of each of the extracted objects to be analyzed.
An analysis unit that analyzes the analyzed attributes and / or motion statistics of the analyzed object,
A dashboard and a dashboard that displays statistical data composed of the analysis results of the statistics.
With
The extraction by the object extraction unit and the analysis by the object analysis unit are both executed by a machine learning type image recognition means having a neural network.
The object analysis unit sets coordinates in the recorded image that associate the position in the recorded image, which is a two-dimensional image, with the actual position in the three-dimensional space to be recorded. It is composed of parts,
Information processing system.
The information processing system according to claim 6, wherein the statistical data is marketing data, a marketing activity support system.
In the category specification section, the category specification step that specifies a specific analysis target category and
An object extraction step in which the object extraction unit extracts an analysis target object belonging to the analysis target category specified in the category designation step from the recorded image, and an object extraction step.
An object analysis step in which the object analysis unit analyzes the attributes and / or movements of the extracted individual objects to be analyzed.
An analysis step in which the analysis unit analyzes the analyzed attribute and / or motion statistics.
A statistical data display step and a statistical data display step in which the dashboard displays statistical data composed of the analysis results of the statistic.
With
The extraction of the analysis target object by the object extraction step and the analysis of the attributes and / or movements of the analysis target object by the object analysis step are both executed by a machine learning type image recognition means having a neural network. ,
In the object analysis step, the coordinate setting unit sets coordinates in the recorded image that associate the position in the recorded image, which is a two-dimensional image, with the actual position in the three-dimensional space to be recorded. Set to,
Information processing method.
In the object extraction step, the recorded image is input to the object extraction unit as digital data, and the object to be analyzed is analyzed from the digital data without going through a conversion process into a two-dimensional image that can be seen by humans. Is extracted directly,
The information processing method according to claim 8.
In the object analysis step, the face recognition information acquisition unit analyzes the age and gender of the person from the image information related to the person's face.
The information processing method according to claim 8 or 9.
In the object analysis step, the skeleton extraction unit extracts the skeleton of the analysis target object composed of skeleton lines connecting a plurality of feature points, and the movement of each of the analysis target objects is generated from the position change of the feature points. Recognized,
The information processing method according to any one of claims 8 to 10.
A marketing activity support method according to any one of claims 8 to 11, wherein the statistical data is marketing data.
A category specification section that allows you to specify a specific analysis target category,
An object extraction unit that extracts an analysis target object belonging to the analysis target category designated by the category designation unit from the recorded image, and an object extraction unit.
An object analysis unit that analyzes the attributes and / or movements of the extracted object to be analyzed.
An analysis unit that analyzes the analyzed attributes and / or motion statistics of the analyzed object,
A dashboard and a dashboard that displays statistical data composed of the analysis results of the statistics.
With
The object analysis unit sets coordinates in the recorded image that associate the position in the recorded image, which is a two-dimensional image, with the actual position in the three-dimensional space to be recorded. It is composed of parts,
In information processing equipment
An object extraction step of extracting the analysis target object belonging to the analysis target category from the recorded image by a machine learning type image recognition means having a neural network.
An object analysis step that analyzes the attributes and / or movements of each of the extracted objects to be analyzed by a machine learning type image recognition means having a neural network.
An analysis step and an analysis step that analyzes the analyzed attribute and / or motion statistics.
A statistical data display step that displays statistical data including the analysis result of the statistic on a dashboard, and
Is executed by the information processing apparatus.
The program according to claim 13, for supporting marketing activities, wherein the statistical data is marketing data.