JP2020096255A - Information processing method, program, information processing apparatus, learned model generation method, and learned model - Google Patents

Abstract

To provide an information processing method and the like that can accurately grasp a passerby passing through a public space.SOLUTION: An information processing method causes a computer to execute processing of acquiring an image of a passerby passing through a public space from an imaging device attached to a power distribution facility installed in the public space, inputting the learned model so as to classify the passerby when the image is input, and acquiring the classification result of the passerby from the learned model.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing method, a program, an information processing device, a learned model generation method, and a learned model.

Various systems have been proposed for performing remote monitoring using an imaging device such as a monitoring camera installed on the street or in a facility. For example, Patent Document 1 discloses a photographic information providing device and the like that delivers in real time an image captured by an imaging device installed on a street pole or the like. Further, Patent Document 2 discloses a remote monitoring system or the like for performing remote monitoring by providing an imaging device on a fixed point such as a store, a station, or a movable object such as a railroad vehicle or an automobile.

JP 2002-185953 A [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-313695

However, the inventions of Patent Documents 1 and 2 merely distribute and share images captured by the image capturing apparatus, and have not yet analyzed the subject included in the captured images.

In one aspect, it is an object of the present invention to provide an information processing method and the like that can accurately grasp a passerby passing through a public space.

In one aspect, the information processing method includes acquiring an image of a passerby passing through the public space from an image pickup device attached to a power distribution facility installed in the public space, and when the image is input, the traffic passing is performed. The acquired image is input to a trained model that has been trained to classify a person, and the computer is caused to execute a process of acquiring a classification result of the passerby from the trained model.

In one aspect, it is possible to accurately understand the passersby who pass through public spaces.

It is a schematic diagram which shows the structural example of a passer-by analysis system. It is a block diagram which shows the structural example of a server. It is explanatory drawing which shows an example of a record layout of a telephone pole DB and a passerby DB. It is explanatory drawing regarding the production|generation process of a classification model. It is an explanatory view about classification processing of a passerby. It is an explanatory view showing an example of a display screen of traffic information. It is a flowchart which shows the process sequence of a classification model generation process. It is a flow chart which shows a processing procedure of classification processing of a passerby. It is a flow chart which shows a processing procedure of output processing of traffic information. FIG. 9 is an explanatory diagram showing an outline of the second embodiment. It is explanatory drawing which shows an example of the display screen of flow line distribution information. It is a flow chart which shows a processing procedure of flow line specific processing. It is explanatory drawing which shows the outline|summary of Embodiment 3. 10 is a flowchart showing a processing procedure of passerby classification processing according to the third embodiment.

Hereinafter, the present invention will be described in detail with reference to the drawings showing an embodiment thereof.
(Embodiment 1)
FIG. 1 is a schematic diagram showing a configuration example of a passerby analysis system. In the present embodiment, a passerby analysis system will be described in which a passerby is imaged by a camera 2 installed in a public space such as a road, and attributes of the passerby are analyzed from the captured image. The passerby analysis system includes an information processing device 1, a camera 2, a sensor 3, and a terminal 4. Each device is communicatively connected via a network N such as the Internet.

The information processing device 1 is an information processing device capable of various information processing and information transmission/reception, and is, for example, a server device, a personal computer, or the like. In the present embodiment, the information processing device 1 is assumed to be a server device, and will be read as server 1 for simplicity in the following description. The server 1 acquires from the camera 2 an image of a passerby who passes by on the road, analyzes the acquired image, and attributes such as age and sex, or actions such as walking speed, direction, movement, or personal belongings. Passers are classified according to various classification criteria, and big data in which passerby information is accumulated is constructed. In the present embodiment, the server 1 classifies passers by using a classification model (learned model) that has been learned by machine learning so as to classify (identify) passers from the captured image.

The camera 2 is an image pickup device installed in a public space, and picks up an image of a passerby passing through a road. In the present embodiment, the camera 2 is attached to a distribution facility installed along a road, specifically, a utility pole. The cameras 2 are respectively attached to telephone poles installed at a plurality of points along the road, and continuously capture images of passers-by who pass by the road from the points. The server 1 acquires the image captured by the camera 2 and analyzes the attributes of the passerby in the acquired image. Further, depending on the measurement environment, a human sensor for detecting a person by a laser or an infrared ray may be installed in place of the camera 2 and data about the flow line information of the person such as the number of people, speed and dwell time may be acquired. Good.

In the present embodiment, a power pole is taken as an example of the power distribution equipment, but the power distribution equipment is not limited to the power pole, and may be, for example, a ground equipment (transformer) of an underground electric wire. Further, the place (point) where the power distribution equipment is installed is not limited to the road, and may be any public space where passersby can pass. Further, the “road” in the present specification may include both a sidewalk and a roadway.

The sensor 3 is a sensing device attached to the telephone pole together with the camera 2, and is, for example, a thermometer, a hygrometer, a rainfall meter, an illuminance meter, an anemometer, or the like. The sensor 3 measures weather information such as temperature, humidity, and rainfall at the point where the camera 2 is installed. The server 1 acquires the weather information of the image capturing point from the sensor 3 and stores it in association with the passerby classification result.

The terminal 4 is a terminal device used by a user who receives the big data provided from the server 1, and is an information processing terminal such as a personal computer, a smartphone, or a tablet terminal. Although the user who uses this system is not particularly limited, for example, the user is a business operator who markets the relevant point, and the terminal 4 receives the traffic information regarding the passerby at the relevant point from the server 1 and displays the traffic information. ..

FIG. 2 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14.
The control unit 11 has an arithmetic processing unit such as one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), and GPUs (Graphics Processing Units), and stores the programs P stored in the auxiliary storage unit 14. By reading and executing, various information processing, control processing, and the like related to the server 1 are performed. The main storage unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), and a flash memory, and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. Remember. The communication unit 13 is a communication module for performing processing relating to communication, and transmits/receives information to/from the outside.

The auxiliary storage unit 14 is a large-capacity memory, a hard disk, or the like, and stores the program P and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a classification model 141, a telephone pole DB 142, and a passer-by DB 143. The classification model 141 is a classifier (classifier) that classifies a passerby in a captured image, and is a learned model generated by machine learning. The utility pole DB 142 is a database that stores information on utility poles in which the camera 2 is installed. The passerby DB 143 is a database that stores the classification results of passersby classified by using the classification model 141.

The auxiliary storage unit 14 may be an external storage device connected to the server 1. Moreover, the server 1 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software.

Further, in the present embodiment, server 1 is not limited to the above-described configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that receives an operation input, a display unit that displays an image, and the like. ..

FIG. 3 is an explanatory diagram showing an example of a record layout of the telephone pole DB 142 and the passer-by DB 143. The utility pole DB 142 includes a utility pole ID sequence, a position information sequence, a camera information sequence, and a sensor information sequence. The telephone pole ID column stores telephone pole IDs for identifying each telephone pole installed on the road. The position information sequence, the camera information sequence, and the sensor information sequence are associated with utility pole IDs, and store the location information of the point where the utility pole is installed, the information of the camera 2 installed on the utility pole, and the information of the sensor 3. There is. The information of the camera 2 may include information such as a height at which the camera 2 is attached, an imaging angle, a magnification, an imaging range, and the like. The information of the sensor 3 may include information about the type of the sensor 3 such as a thermometer, a hygrometer, a rainfall meter, and the like.

The passerby DB 143 includes a telephone pole ID column, a date and time column, a weather column, and a passerby column. The electric pole ID stores the electric pole ID for identifying each electric pole installed on the road. The date/time column, the weather column, and the passerby column are respectively associated with the telephone pole ID, and the date and time when the image is captured by the camera 2 attached to the telephone pole, the weather information around the telephone pole at the date and time, and the passers-by classified by the image. It stores the classification result.

FIG. 4 is an explanatory diagram regarding the generation process of the classification model 141. FIG. 4 conceptually illustrates a process of performing the machine learning to generate the classification model 141. The generation process of the classification model 141 will be described with reference to FIG.

In the present embodiment, the server 1 performs the deep learning for learning the image feature amount regarding the appearance, behavior, belongings, etc. of the passerby appearing in the image captured by the camera 2 as the classification model 141, thereby capturing the captured image. Generates a neural network that outputs the classification result obtained by classifying the passersby. The neural network is, for example, a CNN (Convolution Neural Network), specifically, an R-CNN (Regions with CNN), and has an input layer that receives an input of a captured image, an output layer that outputs a classification result, and image characteristics of the captured image. And an intermediate layer for extracting an amount.

The input layer has a plurality of neurons that receive the input of the pixel value of each pixel included in the captured image, and transfers the input pixel value to the intermediate layer. The intermediate layer has a plurality of neurons, recognizes an object (passerby, etc.) from the captured image, and extracts the feature amount of the image area (indicated by a dotted rectangular frame in FIG. 4) in which the recognized object is reflected. And pass it to the output layer. For example, when the classification model 141 is CNN, the middle layer alternately includes a convolution layer that convolves the pixel values of each pixel input from the input layer and a pooling layer that maps the pixel values that are convolved by the convolution layer. It has a connected configuration and finally extracts the image feature amount while compressing the pixel information of the image region of the object. The output layer is a classifier related to SVM (Support Vector Machine), has one or a plurality of neurons that output the classification result of the passerby, and classifies the passerby based on the image feature amount output from the intermediate layer. ..

Although the classification model 141 is described as a CNN in the present embodiment, the classification model 141 is not limited to CNN, and may be constructed by another learning algorithm such as a neural network other than CNN, a Bayesian network, a decision tree, or the like. It may be a trained model.

The server 1 performs learning using teacher data in which a plurality of images of the passers-by are associated with the correct values of the classification result when the passers-by in each image are classified according to attributes and the like. For example, as shown in FIG. 4, the teacher data is data obtained by labeling the captured image of the passerby with the coordinate range of the image area in which the passerby is shown and the classification result of the passerby.

Note that, in the drawings of this specification, one image of one passer-by is shown for simplification, but it is needless to say that a plurality of passers-by may be shown in one image. ..

The server 1 inputs the captured image, which is the teacher data, into the input layer, performs the arithmetic processing in the intermediate layer, and acquires the passerby classification result from the output layer. The classification result output from the output layer may be a discrete value (for example, a value of “0” or “1”), and a continuous probability value (for example, a range from “0” to “1”). Value). For example, the server 1 includes, as the classification result output from the output layer, the attributes of the passerby such as gender and age, the walking speed, the walking direction, and the action of the passerby such as entering and leaving the facility (for example, a roadside store). Also, obtain the classification results classified according to your belongings. The details of the above classification criteria will be described later.

The server 1 compares the classification result output from the output layer with the information labeled on the captured image in the teacher data, that is, the correct value, and the intermediate layer so that the output value from the output layer approaches the correct value. Optimize the parameters used for the calculation processing in. The parameter is, for example, a weight between neurons (coupling coefficient), a coefficient of an activation function used in each neuron, or the like. The method of optimizing the parameters is not particularly limited, but the server 1 optimizes various parameters using the error back propagation method, for example.

The server 1 performs the above processing on each captured image included in the teacher data to generate the classification model 141. When the captured image is acquired from the camera 2, the server 1 classifies the passers-by using the classification model 141.

FIG. 5 is an explanatory diagram relating to a passer-by classification process. FIG. 5 conceptually illustrates how a passerby is classified based on various criteria from a captured image. The passer-by classification process will be described with reference to FIG.
The server 1 acquires an image of a passerby from each of the cameras 2, 2, 2... Attached to each power pole installed at a plurality of points along the road. For example, the camera 2 continuously captures images, and the server 1 acquires in real time a moving image captured by the camera 2 of a road on which a passerby passes, or periodically acquires recorded data of the moving image.

Further, the server 1 acquires weather information such as temperature, humidity, and rainfall amount at the time of capturing an image from each sensor 3 attached to the same utility pole as each camera 2. It should be noted that the server 1 may acquire the weather information of the point from a predetermined external API (Application Programmable Interface) instead of using the sensor 3. That is, it is not essential to install the sensor 3 on the utility pole.

The server 1 inputs the image acquired from the camera 2 into the classification model 141, and acquires the classification result obtained by classifying the passers-by shown in the image as an output. For example, the server 1 sequentially inputs, into the classification model 141, the frame images at the respective image capturing times that form the moving image captured by the camera 2. The server 1 performs arithmetic processing to extract the feature amount of the frame image in the middle layer of the classification model 141, inputs the extracted feature amount to the output layer of the classification model 141, and is imaged at each point at each imaging time. The classification result obtained by classifying the passersby in the frame image is acquired as an output.

In the present embodiment, the server 1 classifies passers by various criteria. As a first classification criterion, the server 1 classifies according to the attributes of passersby. The passerby attributes are, for example, sex and age. For example, the server 1 classifies the males and females of passersby and the ages of minors in their 20s, 40s, 50s, and 60s or more into a total of 8 classifications.

Note that the attributes of the passerby are not limited to gender and age, and for example, emotions that can be recognized from the face (expression) of the passerby may be detected (classified) as attributes.

As a second classification criterion, the server 1 classifies according to the behavior of the passerby. The behavior of the passerby is, for example, the walking speed of the passerby, the walking direction (destination), the walking position on the road (whether it is on the right side or the left side, etc.) and other actions.

In particular, in the present embodiment, the server 1 detects an entry/exit action of a passerby to a store located on the side of a road, that is, a facility existing in a public space, and classifies the passerby according to the detected enter/exit action. .. The entry/exit operation is an operation in which a passerby enters (enters) the entrance/exit of the store or exits (exits) from the entrance/exit of the store. It should be noted that the server 1 may detect only one of entry and exit. Further, the entry/exit operation is not limited to the operation at the time of actual entry/exit of the store, and may be an operation indirectly involved in the entry/exit such as a stop operation in front of the store and an operation of directing a line of sight to the store. The server 1 classifies the passers-by who enter and exit the store and the passers-by who pass without entering the store among passers who pass in front of the store according to the detected entry-exit operation.

As a third classification criterion, the server 1 classifies according to the belongings of the passerby. The belongings of the passerby are, for example, an umbrella, a paper bag, a bag, a suitcase, a container for drinks that the passerby possesses, clothes of the passerby, pets (dogs, etc.) taken by the passerby. The server 1 generates a classification model 141 in which the image feature amounts of these belongings are learned during learning, detects the belongings of the passerby using the classification model 141, and determines the type of the detected belongings.

The server 1 not only detects the belongings of the passerby from the captured image, but also estimates the gender, age, etc. of the passerby according to the belongings, thereby accurately classifying the attributes of the passerby. For example, when the weather is fine and the user has an umbrella, the server 1 determines that the parasol is a parasol and classifies the passerby as a woman. Also, for example, the server 1 classifies a passerby as a woman when the passerby wears high heels, a skirt, or the like. In this way, the server 1 detects the belongings using the classification model 141 and classifies the attributes of passers-by according to the detected belongings.

The server 1 inputs each frame image in the moving image acquired from each camera 2 to the classification model 141, and classifies the passers-by who pass through each point at each image capturing time according to each of the above classification criteria. It should be noted that the server 1 may perform classification according to each of the above-described classification standards with one classification model 141, and a plurality of classification models 141 are prepared according to each classification standard, and the plurality of classification models 141 are used. You may classify.

The server 1 stores the classification result obtained by classifying the passers-by shown in the frame image in the passer-by DB 143 in association with the image capturing time (date and time) at which the frame image was captured and the installation point of the camera 2 at which the frame image was captured. .. The server 1 continuously performs the above-mentioned processing, and stores the information of the passerby in the passerby DB 143.

FIG. 6 is an explanatory diagram showing an example of a display screen of traffic information. The server 1 outputs (delivers) the information of the passerby stored in the passerby DB 143 to the terminal 4 in response to the access from the terminal 4. The output processing of the traffic information will be described with reference to FIG.

When the output request is received from the terminal 4, the server 1 refers to the passer-by DB 143 to generate the pass information, and outputs the pass information to the terminal 4. For example, the server 1 first accepts, via the terminal 4, a designated input regarding the location of the traffic information to be output and the date and time (imaging time). Specifically, the server 1 accepts a designation input that designates a point at which the traffic information is desired to be provided and a desired date (for example, month and day of the week). The server 1 refers to the passer-by classification result stored in the passer-by DB 143 in association with the designated spot and date in the camera 2 installed at the designated spot, and specifies the designated date. Passage information is generated by analyzing the classification results of passers-by who have passed through the specified point and output to the terminal 4.

An example of a display screen of the traffic information output from the server 1 is illustrated on the lower side of FIG. For example, the terminal 4 displays a graph in which the number of passersby who pass through a designated point on a designated date is totaled by time zone and weather. Further, the terminal 4 displays a graph in which passers-by are classified by attributes such as gender and age. In addition, the terminal 4 displays a graph in which the number of passersby walking in each direction (direction) is tabulated according to the weather. In addition, the terminal 4 displays a graph showing the ratio of pedestrians who have various belongings by temperature.

Further, the server 1 detects the passage information of the entrance/exit information regarding the passers-by/passers entering/exiting the facility based on the entry/exiting action of the passer-by to/from the facility (store) existing at the spot detected by using the classification model 141. It is generated as one and is output to the terminal 4. For example, the server 1 generates a graph in which the number of passers-by who enter and exit the relevant facility is aggregated by time zone, and a graph showing the number of times the passer-by stays in the relevant facility by the number of people, and displays it on the terminal 4. .. In addition, for example, the server 1 may generate more detailed entrance/exit information, such as totaling the number of people who enter and leave the passerby and the staying time for each attribute. Thereby, not only the information of the passers-by on the road but also the information of the passers-by who enter and exit the store can be presented, and the data more suitable for marketing can be provided to the user.

As described above, the server 1 collects the number of passersby according to each classification criterion (attribute, behavior, belongings) to determine the distribution status of passersby at a designated point and date (imaging time). The traffic information shown is generated and output to the terminal 4. Thereby, the server 1 can provide reference information, for example, when the user performs marketing at the corresponding point.

As described above, the server 1 can classify the passers-by using the classification model 141 generated by machine learning and provide the information of the passers-by on the road. In particular, in the present embodiment, the camera 2 is attached to each power pole installed along the road, and the road is continuously captured by each camera 2, so that linear and planar observation such as street view can be performed. While making it possible, it is possible to perform observation in time series like a surveillance camera. That is, it is possible to realize a system having both characteristics and analyze a passerby suitable for marketing and the like. In particular, telephone poles have already been installed on many roads and are installed at predetermined intervals. By using this to install the camera 2 on each telephone pole, the area to be observed can be appropriately covered, and the information of passersby can be collected more accurately.

FIG. 7 is a flowchart showing a processing procedure of generation processing of the classification model 141. The contents of the machine learning process for generating the classification model 141 from the teacher data will be described based on FIG. 7.
The control unit 11 of the server 1 acquires teacher data in which an image obtained by capturing a passerby who passes by a road is associated with a correct value of a classification result when the passerby shown in the image is classified (step). S11). The teacher data is, for example, data in which information such as attributes such as sex and age, behavior, and belongings is labeled for each of one or more passers-by in the image.

The control unit 11 uses the teacher data to generate a classification model 141 (learned model) that outputs a classification result of classifying the passerby when a captured image of the passerby is input (step S12). Specifically, the control unit 11 inputs the captured image that is the teacher data to the input layer of the neural network, and acquires the classification result obtained by classifying the passers-by in the captured image from the output layer. The control unit 11 compares the obtained classification result with the correct value of the teacher data (information labeled on the captured image), and the classification result in the intermediate layer is adjusted so that the classification result output from the output layer approaches the correct value. Optimize parameters (weight, etc.) used for arithmetic processing. The control unit 11 stores the generated classification model 141 in the auxiliary storage unit 14, and ends the series of processes.

FIG. 8 is a flowchart showing the processing procedure of the passer-by classification. The contents of the process of classifying passers by using the classification model 141 will be described with reference to FIG.
The control unit 11 of the server 1 acquires an image of a passerby passing by from the camera 2 provided on a telephone pole installed along the road (step S31). Specifically, the control unit 11 acquires moving images in which the road is continuously imaged from the cameras 2, 2, 2... That are installed at the respective points along the road and correspond to the respective utility poles. The control unit 11 also acquires the date and time when the image was captured, the weather information of each point at each image capturing point, and the like (step S32).

The control unit 11 inputs the image acquired from the camera 2 into the classification model 141, and acquires from the classification model 141 the classification result obtained by classifying the passers-by shown in the image (step S33). Specifically, the control unit 11 inputs each frame image constituting the moving image acquired from each camera 2 into the classification model 141, and classifies the passers-by imaged at each imaging time point and each point. To get. For example, the control unit 11 obtains a classification result in which the passerby is classified by sex, age, and the like, actions including entrance/exit actions to and from the facility, and belongings. The control unit 11 stores the classification result of the passerby in the passerby DB 143 in association with the image pickup time point and the image pickup point (step S34), and ends the series of processes.

FIG. 9 is a flowchart showing the processing procedure of the output processing of the traffic information. The contents of the traffic information output process will be described with reference to FIG.
The control unit 11 of the server 1 receives, from the terminal 4, the designation input regarding the location, the date and time (imaging point) of the traffic information to be output, and the like (step S51). For example, the control unit 11 receives a designation input that designates a spot to which the traffic information is output and a date to be the output target. The control unit 11 is a classification result of passers-by obtained from an image captured by the camera 2 installed near a designated point, and the classification result of passers-by obtained from an image captured on a designated date. It is read from the person DB 143 (step S52). For example, the control unit 11 reads out the passerby classification result stored in the passerby DB 143 in association with the designated point and date, and the weather information at the designated point and date.

The control unit 11 generates the pass information regarding the passers-by who pass through the designated spot on the designated date based on the read-out classification result of the passers-by (step S53). For example, the control unit 11 may designate a graph showing the number of passers-by passing by the time zone and the weather as well as a graph summarizing the number of passers-by according to classification criteria such as attributes, actions, and belongings. Passage information indicating the distribution status of passersby at different points and dates (imaging time) is generated. The control unit 11 also uses the entry/exit information of the passers-by, who enter/exit the facility (store) on the side of the road, based on the entry/exit actions of the passers-by detected using the classification model 141. To generate. The control unit 11 outputs the generated traffic information to the terminal 4 (step S54) and ends the series of processes.

As described above, according to the first embodiment, by attaching the camera 2 (imaging device) to the telephone pole (power distribution equipment) which is already installed in many places on the road (public space), the passerby can be suitably imaged. By classifying the passers-by using the classification model 141 generated by machine learning, the features of the passers-by can be accurately analyzed. As a result, it is possible to accurately grasp the passersby who pass through the public space.

In addition, according to the first embodiment, the camera 2 is attached to each telephone pole installed on a large number of roads, and continuous imaging is performed by each camera 2, thereby enabling a wide spatial observation. , It is also possible to observe in time series.

Further, according to the first embodiment, by storing the classification result of the passerby in association with the point and the time (imaging time) at which the passerby passes, the information of the passerby passing through each point is accumulated. Big data can be constructed.

In addition, according to the first embodiment, by storing the classification result of the passers-by in association with the weather information, it is possible to grasp the passing amount according to the weather, the attributes of the passing people, and the like.

Further, according to the first embodiment, it is possible to provide the user with information useful for marketing and the like.

Further, according to the first embodiment, since the processing is performed based on the captured image, not only the attributes of the passerby can be grasped (classified), but also the behavior (behavior) of the passerby is performing. It will be possible to grasp even the highest. This enables a deeper analysis than the big data collection using a GPS (Global Positioning System) function of a smartphone or the like, and makes it possible to more accurately grasp the information of passersby.

In addition, according to the first embodiment, it is possible to detect the entry/exit action of a passerby to/from the facility (store) existing on the side of the road and provide the entry/exit information, so that the user can obtain more useful information for marketing. Can be provided to.

Further, according to the first embodiment, by identifying (classifying) the belongings of the passerby, it is possible to more accurately grasp the passerby information including attributes such as gender.

(Embodiment 2)
In the present embodiment, a mode in which the flow line of a passerby is specified from images captured at a plurality of points will be described. In addition, the same reference numerals will be given to the same contents as those in the first embodiment, and the description thereof will be omitted.
FIG. 10 is an explanatory diagram showing an outline of the second embodiment. Similar to the first embodiment, the server 1 acquires a moving image from each camera 2 attached to each power pole installed at each of a plurality of points, and captures a frame at each point at each image capturing time (time). Passers-by are recognized from the images and classified according to attributes such as age and sex, walking speed, behavior such as direction, or belongings. Further, in the present embodiment, the server 1 performs multi-camera cooperation, and the same passerby passes each point based on the classification result of the passerby classified (identified) from each frame image captured at each of a plurality of points. Identify the flow line.

For example, the server 1 performs anonymity authentication of passers-by based on the classification result output from the classification model 141 such as attributes and personal belongings. The server 1 identifies a point (for example, an installation point of the camera 2) at which a passerby having the same classification result is imaged and an image capturing time point, and maps the point at which the passerby is imaged and the image capturing time point. The server 1 arranges the points at which the passerby was imaged in the order according to the time of image pickup, and identifies the flow line estimated that the passerby has moved.

In the above, the server 1 allows the passerby to pass in front of which store based on the behavior such as the walking speed, the direction, the position, and the entry/exit operation of the passerby at each point specified from the images captured by the cameras 2. However, it may be possible to identify which store has stopped and which store has entered and exited. That is, the server 1 may specify not only the flow line of each passerby from the images of the plurality of cameras 2 but also the movement (behavior) of each passerby on the flow line. As a result, it is possible to grasp in more detail what kind of route and how the passerby was acting.

FIG. 11 is an explanatory diagram showing an example of a display screen of flow line distribution information. The server 1 generates flow line distribution information indicating how the passersby move in the target area (public space) based on the flow lines of the passers identified above and the flow line distribution information. Output to the terminal 4.

For example, the server 1 receives a designation input of the target area via the terminal 4. Specifically, the server 1 receives the input of the station name of the railway station. The server 1 identifies the flow line of each passerby in the area around the corresponding station, and superimposes the flow line of each passerby on the map to generate flow line distribution information. In FIG. 11, the thickness of the line indicates the size of the number of passersby. As shown in FIG. 11, how the passers-by move in the target area, especially from the railway station where many passers-by come and go in common, or the route toward the railway station. It is possible to grasp at a glance whether to move.

The server 1 does not simply generate the flow line distribution information by superimposing the flow lines of all the passers-by, but narrows down the target people based on, for example, the attributes of the passers-by, time zones, etc. to obtain the flow-line distribution information. Of course, it may be generated. Alternatively, the server 1 may narrow down the target persons by using the facility (store) as a key and generate the flow line distribution information of the passers-by who enter and exit the relevant facility. Alternatively, the server 1 may generate flow line distribution information to be displayed in a different manner, such as displaying the flow lines of passers-by who have entered or exited the relevant facility in a different color from the flow lines of other passers-by. Good. This makes it possible to provide more useful information for marketing measures for the relevant facility.

Further, in the display screen illustrated in FIG. 11, the size of the number of passersby is only represented by the thickness of the line, but for example, the terminal 4 displays each passerby in a time series at the time when each passerby is imaged. The flow line distribution information may be displayed by reproducing the movement (flow line) of the passerby. This allows the user to intuitively understand how many passers-by at each time zone exist at each point.

FIG. 12 is a flowchart showing the processing procedure of the flow line identification processing.
The control unit 11 of the server 1 receives a designation input of the target area for which the flow line distribution information is generated via the terminal 4 (step S201). In step S201, the control unit 11 may accept the designation input by narrowing down the attribute, date and time (time zone, etc.) of the target passerby, or the facility where the passerby enters and exits. The control unit 11 identifies the flow line of each passerby based on the passerby classification result obtained from the frame images corresponding to each of the plurality of points in the target area (step S202). Specifically, the control unit 11 performs anonymous authentication of passers-by based on the classification result output from the classification model 141, and specifies passers-by imaged at each of a plurality of points. The control unit 11 performs mapping based on each point where the passerby passes and the time when the passerby passes (imaging time), and identifies the flow line for each passerby.

The control unit 11 generates flow line distribution information indicating the distribution status of the flow lines specified for each of the plurality of passers-by (step S203). For example, the control unit 11 generates flow line distribution information in which the flow lines of the passersby are superimposed on the map of the target area. The control unit 11 outputs the generated flow line distribution information to the terminal 4 (step S204), and ends the series of processes.

As described above, according to the second embodiment, the passerby can be tracked (tracking), and the behavior of the passerby can be more accurately grasped.

Further, according to the second embodiment, it is possible to present the flow line distribution information of each passerby, and it is possible to provide more useful information in marketing and the like.

(Embodiment 3)
In the present embodiment, a mode will be described in which a classification model 141 used when classifying a passerby is selected in accordance with an imaging condition when capturing a passerby.
FIG. 13 is an explanatory diagram showing an outline of the third embodiment. In the present embodiment, the server 1 generates and holds a plurality of classification models 141 in advance according to the imaging conditions when imaging a passerby. The imaging condition is, for example, the weather (weather information) near the telephone pole at the time of imaging. For example, the server 1 includes a classification model 141 for fine weather, which is learned based on an image captured in fine weather, and a classification model 141 for rainy weather, which is learned based on an image captured in rainy weather. A type classification model 141 is prepared.

The mounting position of the camera 2 on the telephone pole is not particularly limited, but in many cases, it is assumed that the camera 2 is mounted on or near the arm (bracket) and the passerby is imaged from a high place. On the other hand, it is assumed that most passersby are holding umbrellas in the rain. Therefore, when a passerby is imaged in rainy weather, the upper half of the body of the passerby is not imaged, attributes of the passerby are not recognized, and proper classification may not be possible.

Therefore, in this embodiment, two types of classification models 141 for fine weather and for rainy weather are prepared. Note that the number of classification models 141 prepared by the server 1 is not limited to two, and may be three or more, such as preparing a classification model 141 for snow.

The server 1 performs learning based on an image taken in the rain, that is, an image in which many passersby are holding umbrellas, and generates a classification model 141 for rain. Specifically, the server 1 recognizes the umbrella that the passerby is holding from the captured image, and causes the classification model 141 to learn the image feature amount of the recognized umbrella, thereby determining the attribute of the passerby from the passerby umbrella. A classification model 141 for classification is generated. For example, the server 1 learns image feature amounts such as colors and shapes of the men's umbrella and the women's umbrella, and determines the gender of the passerby depending on whether it is a men's umbrella or a women's umbrella. A classification model 141 for classification is generated.

Further, for example, the server 1 may perform learning such that the height of a passerby is a learning target in addition to the umbrella described above, and whether the adult is a child or a child is estimated from the height so that the age of the passerby can be classified. .. Alternatively, the server 1 may add the lower body clothes such as pants and shoes to the learning target, and perform learning so that the sex, age, etc. can be classified from the lower body clothes. In this way, the classification model 141 for rainy weather is generated in case the upper body (especially face) of a passerby is hidden by the umbrella.

The server 1 selects the classification model 141 based on the weather at the time of image capturing, inputs an image into the selected classification model 141, and classifies passers-by. Specifically, the server 1 uses the classification model 141 to extract the feature amount of the passerby's umbrella, height, clothes of the lower body, and the like, and determines the attributes such as the gender and age of the passerby according to the extracted feature amount. Classify. The server 1 stores the classification result in the passer-by DB 143 in association with the weather (weather information) in addition to the shooting point, the shooting time, and the like.

Although weather is used as the imaging condition that defines the type of the classification model 141 in the above, other weather information such as temperature, humidity, and wind speed may be used as the imaging condition. Further, for example, as imaging conditions, time conditions (day or night), seasonal conditions such as seasons, or geographical conditions such as regions may be used. That is, the imaging condition may be any condition related to the imaging time point and the imaging point, and the content thereof is not particularly limited.

FIG. 14 is a flowchart showing the processing procedure of the passerby classification processing according to the third embodiment.
After acquiring the captured image from the camera 2 (step S31) and acquiring the date and time of the image capturing, the weather information of the image capturing point, and the like (step S32), the control unit 11 of the server 1 executes the following processing. The control unit 11 selects the classification model 141 according to the image capturing conditions when the image is captured (step S301). For example, the control unit 11 selects the classification model 141 according to the weather information acquired in step S32. The control unit 11 inputs the captured image into the selected classification model 141 and classifies the passerby (step S302). The control unit 11 shifts the processing to step S34.

As described above, according to the third embodiment, by using different classification models 141 depending on the imaging conditions, it is possible to more accurately grasp the passerby.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the meanings described above but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

1 server (information processing device)
11 Control Section 12 Main Storage Section 13 Communication Section 14 Auxiliary Storage Section P Program 141 Classification Model 142 Utility Pole DB
143 Passer DB
2 camera 3 sensor 4 terminal

Claims (16)

From an image pickup device attached to a power distribution facility installed in a public space, an image of a passerby passing through the public space is acquired,
In the learned model that has been trained to classify the passers-by when the image is input, input the acquired image,
An information processing method, comprising causing a computer to execute a process of acquiring the classification result of the passers-by from the learned model. From each of the imaging devices corresponding to each of the power distribution equipment installed in each of the plurality of points of the public space, to obtain a moving image continuously captured the public space,
The frame images at the time of each image forming the moving image are input to the learned model,
The information processing method according to claim 1, wherein a classification result of the passerby reflected in the frame image corresponding to each point and an imaging time point is acquired from the learned model. The information processing method according to claim 2, wherein the storage unit stores a classification result of the passers-by shown in the frame image in association with the point corresponding to the frame image and the image capturing time. Obtaining weather information of the point at the time of the imaging,
The information processing method according to claim 3, wherein the classification result of the passerby and the weather information are stored in the storage unit in association with the point and the image capturing time. Accept the specified input about the point and the time of imaging,
Based on a classification result of the passerby corresponding to the designated point and the image capturing time point, generate passer information about the passerby who has passed the point at the image capturing point point,
The information processing method according to claim 4, wherein the traffic information is output. The flow line of the same passerby is specified based on a classification result of the passerby obtained from the frame images corresponding to a plurality of points, respectively. The information processing method described in the section. Based on the flow lines of each of the plurality of passers-by, generate flow line distribution information indicating a distribution state of the flow lines in the public space,
The information processing method according to claim 6, wherein the generated flow line distribution information is output. The information processing method according to claim 1, wherein a classification result obtained by classifying the passers-by according to the behavior of the passers-by is acquired from the learned model. From the learned model, obtain a classification result that classifies the passers-by according to the entrance and exit actions of the passers-by to the facilities existing in the public space,
Generate entrance/exit information about the passers-by entering/leaving the facility based on the obtained classification result,
The information processing method according to claim 8, wherein the generated entrance/exit information is output. The information processing according to any one of claims 1 to 9, wherein a classification result in which the attributes of the passerby are classified according to the belongings possessed by the passerby is acquired from the learned model. Method. Acquiring information about the imaging conditions when the imaging device images the passerby,
Any one of the plurality of learned models that classify the passers-by according to the imaging conditions is selected based on the acquired information about the imaging conditions,
The information processing method according to any one of claims 1 to 10, wherein the image is input to the selected learned model. The information regarding the imaging conditions is weather information when the imaging device images the passerby,
The information processing method according to claim 11, wherein the learned model for classifying the passerby is selected from an umbrella held by the passerby according to the weather information. From an image pickup device attached to a power distribution facility installed in a public space, an image of a passerby passing through the public space is acquired,
In the learned model that has been trained to classify the passers-by when the image is input, input the acquired image,
A program for causing a computer to execute a process of acquiring the classification result of the passerby from the learned model. An acquisition unit that acquires an image of a passerby passing through the public space from an imaging device attached to a power distribution facility installed in the public space,
An input unit that inputs the acquired image into a learned model that has been learned to classify the passers-by when the image is input,
An information processing device, comprising: a classification unit that acquires a classification result of the passerby from the learned model. Acquiring teacher data in which an image of a passerby passing through a public space and a correct value of a classification result obtained by classifying the passerby are associated with each other,
A method of generating a learned model, comprising causing a computer to execute a process of generating a learned model that outputs a classification result obtained by classifying the passersby when the image is input based on the acquired teacher data. An input layer that receives an input of an image obtained by imaging a passerby from an imaging device attached to a power distribution facility installed in a public space,
An image feature amount is extracted from the image based on teacher data that is combined so that an output from the input layer is input and that associates the image with a correct value of a classification result of classifying the passerby. The middle layer that has already learned the parameters used for the arithmetic processing
An output layer that is coupled so that the output from the middle layer is input, and that outputs a classification result in which the passers-by are classified based on a calculation result in the middle layer,
A learned model for causing a computer to input the image to the input layer, perform arithmetic processing on the intermediate layer, and output the classification result of the passerby from the output layer.

Images