WO2020240851A1

WO2020240851A1 - System, method, or program

Info

Publication number: WO2020240851A1
Application number: PCT/JP2019/021811
Authority: WO
Inventors: 佐々木　雄一; ダニエルズ・アレックス・エドワード; 聞浩周; 山本　正晃; ジニトバット; 路威重松
Original assignee: ニューラルポケット株式会社
Priority date: 2019-05-31
Filing date: 2019-05-31
Publication date: 2020-12-03
Also published as: JPWO2020240851A1

Abstract

[Problem] One example of a system according to the present invention enables an image obtained from an imaging device to be more appropriately utilized. [Solution] A system comprising: an extraction unit for extracting, from an object image, some information in the object image that pertains to a moving portion in an moving-image that includes the object image; and a generation unit for generating, on the basis of the extracted some information, information about an object that pertains to the some information. A system in which the generation unit generates information about the object that pertains to the some information by applying the image that pertains to the some information to a machine learning unit having learned a relationship between the image and the object. A system in which the extraction unit extracts the some information using the motion of the object in the moving-image. A system in which the extraction unit extracts the some information using a difference between one image and other images out of a plurality of images constituting the moving-image.　

Description

[Name of invention determined by ISA based on Rule 37.2.] System, method, or program

The technology disclosed in this application relates to an information processing system, an information processing device, a server device, a program, or a method.

In recent years, methods for utilizing information in images have been studied.

JP-A-2019-48365 Japanese Unexamined Patent Publication No. 2012-164255 Japanese Unexamined Patent Publication No. 2013-239011 JP-A-2010-118862 JP-A-2009-110185

However, the inventions according to

Patent Documents

1 and 2 only improve the accuracy with respect to a part of the subject. Further, the invention according to Patent Document 3 is limited to improvement of clustering itself. Further, the invention according to Patent Document 4 can only move the background. Further, the invention according to Patent Document 5 points out a problem of using a camera, but requires a laser. Moreover, the invention according to Non-Patent Document 1 has not reached a concrete invention. Therefore, various embodiments of the present invention provide an information processing system, an information processing device, a server device, a program, or a method in order to solve the above problems.

The first system of one embodiment is
An extraction unit that extracts a part of information in the target image related to a moving part in a moving image including the target image from the target image.
A generation unit that generates target information related to the partial information based on the extracted partial information.
System with.

Further, in the first system of one embodiment, instead of the above,
An extraction unit that extracts some information from the target image,
A generation unit that generates target information related to the partial information based on the extracted image related to the partial information.
With
In the image related to the partial information, the amount of image information of a part of the background region is smaller than the amount of image information of the portion corresponding to the part in the target image.
It can be a system.

The second system of one embodiment
The generation unit generates information on the target related to the image related to the partial information by applying the image related to the partial information to the machine learning unit in which the relationship between the image and the target is machine-learned.
The first system.

The third system of one embodiment is
The extraction unit extracts a part of the information by using the movement of the target in the moving image.
The system of any one of the first and the second.

The fourth system of one embodiment is
The extraction unit extracts a part of the information by using the difference between one image and the other image among the plurality of images constituting the moving image.
The system of any one of the first to the third.

The fifth system of one embodiment is
The partial information includes a part corresponding to the movement of the target in the moving image.
The system of any one of the first to four.

The sixth system of one embodiment is
The generator
A transmitter that transmits an image related to the partial information,
An acquisition unit that acquires target information corresponding to an image related to the transmitted partial information, and an acquisition unit.
The system of any one of the first to fifth.

The seventh system of one embodiment is
The generation unit includes a machine learning unit that machine-learns the relationship between an image and an object.
By applying the image related to the partial information to the machine learning, the target information related to the image related to the partial information is generated.
The system of any one of the first to sixth.

The eighth system of one embodiment is
The partial information is a partial image of the target image.
The system of any one of the first to seventh.

The ninth system of one embodiment is
The partial information is information relating to feature points of a partial image of the target image.
The system of any one of the first to eighth.

The tenth system of one embodiment is
The partial information is information related to background subtraction of the target image.
The system of any one of the first to nine.

The eleventh system of one embodiment is
The partial information is an image obtained by binarizing the background difference of the target image.
The system of any one of the first to tenth.

The twelfth system of one embodiment is
The system
An estimation unit that estimates the number of targets based on predetermined rules for the cluster related to the partial information.
To prepare
The system of any one of the first to eleventh.

The thirteenth system of one embodiment is
The system further includes a statistical processing unit that generates statistical information.
The system of any one of 1st to 12th.

The fourteenth method of one embodiment is
The computer
A step of extracting a part of a moving part of a moving image including the target image from the target image, and
A step of generating target information related to the part of the image based on the extracted part of the image, and
How to do.

The fifteenth program of one embodiment
A program for operating a computer as any one of the first to thirteenth systems.

The sixteenth system of one embodiment
A clustering unit that generates clusters based on corresponding feature points in at least two images of a plurality of images constituting a moving image, and a clustering unit.
An estimation unit that estimates the number of objects in the cluster based on predetermined rules,
System with.

The seventeenth system of one embodiment
The predetermined rule is associated with the camera from which the image was acquired.
The system of any one of 1st to 16th

The eighteenth system of one embodiment
The extraction unit is used in the first target image and the second target image constituting the moving image.
The first part image in the first target image and
The second part image in the second target image and
Extract and
A determination unit that determines the relationship between the first part image and the second part image,
The system of any one of 1 to 17 comprising.

The nineteenth system of one embodiment
The system includes a statistical processing unit that performs statistical processing using the position related to the target.
The system of any one of 1st to 18th.

The twentieth program of one embodiment is
A program for operating a computer as any one of the first to nineteen systems.

According to one embodiment of the present invention, the data obtained from the image can be utilized more appropriately.

FIG. 1 is a block diagram showing a specific example of the function of the information processing apparatus according to the embodiment. FIG. 2 is a diagram schematically showing the extraction of information according to one embodiment. FIG. 3 is an example schematically showing the data possessed by the system according to the embodiment. FIG. 4 is a diagram schematically showing the extraction of information by the system according to one embodiment. FIG. 5 is an example schematically showing the data possessed by the system according to the embodiment. FIG. 6 is an example schematically showing the relationship of information processed in the system according to one embodiment. FIG. 7 is an example for explaining the number of clusters according to one embodiment. FIG. 8 is an example of a screen displayed by the system according to the embodiment. FIG. 9 is a diagram illustrating an example of a boundary handled by the system according to an embodiment. FIG. 10 is an example of a screen displayed by the system according to the embodiment. FIG. 11 is an example of a screen displayed by the system according to the embodiment. FIG. 12 is an example of a screen displayed by the system according to the embodiment. FIG. 13 is an example schematically showing the data possessed by the system according to the embodiment. FIG. 14 is a block diagram showing an overall picture including the configuration of the system according to one embodiment. FIG. 15 is an example schematically showing the facility corresponding to the system according to the embodiment. FIG. 16 is an example schematically showing the data possessed by the system according to the embodiment. FIG. 17 is an example of a screen displayed by the system according to the embodiment. FIG. 18 is an example of a flow processed by the system according to one embodiment. FIG. 19 is an example of a flow processed by the system according to the embodiment. FIG. 20 is an example of a flow processed by the system according to one embodiment.

1. 1. Functions of the system of this example The functions of the system of this example will be described with reference to FIG. This figure is a block diagram showing a specific example of the function related to the system of this example. An example system consists of a partial information extraction unit, a partial information relationship generation unit, a position identification unit, a target number estimation unit, an information generation unit, a boundary unit, a statistical processing unit, and a tracking unit. It may have some or all. All of these functions are not essential functions in the system, and even if they are a part, they have technical significance corresponding to each function, and even in their combinations, depending on each combination. It has technical significance.

1.1. Partial information extraction unit 11
The partial information extraction unit has a function of extracting partial information from the target image. The target image may be one of a plurality of images constituting the moving image. The partial information may relate to a moving part in the moving image including the target image, and includes, for example, a part corresponding to the movement of the target in the moving image and a part corresponding to the movement of the target in the moving image. It may be a thing, a part of a thing including a part corresponding to the movement of the object in the moving image, and the like. Further, the partial information may be a part of the target image or may be a feature point related to the target related to the target image. The area of the target image may be a part of the information, the image information of the target image may be a part of the information, or a combination thereof may be used as the partial information. When the image information is a part of the information, for example, if the image related to the part of the information is a bitmap image, the number of pixels may be reduced, or the number of bits of each pixel may be reduced. It may be made to. Further, if the image related to some information is a vector image, it may be a part of a constituent figure. The amount of image information may also be referred to as the amount of image information. Examples of the amount of image information include the number of pixels, the number of bits of each pixel, the number of colors, the number of figures, etc., as long as it indicates the amount of information in the image. , Anything can be used.

The partial information extraction unit of one example may have a partial image extraction unit and / or a feature point extraction unit in order to extract partial information.

1.1.1. Partial image extraction unit The partial image extraction unit has a function of extracting a part of an image from a target image. In the moving image composed of a plurality of images, the difference portion of one example may extract a part of the images that are the differences between the images due to the difference between one image and the other images. The part of the image may be a part of the one image or a part of the other image. The one image and the other image may be the next image in the time series of the moving image, or after a certain period of time such as several microseconds to several seconds has elapsed between the one image and the other image in the moving image. It may be a thing. When the time interval between one image and the other image is longer than a predetermined period, there is an advantage that the amount of calculation can be reduced. The partial image extraction unit may extract one or a plurality of partial images from the target image.

Taking the case where the image pickup device is a traffic camera that captures images on the road through which vehicles, people, animals (including pets), etc. (hereinafter, also referred to as "vehicles, etc.") pass, one image and others The difference between the images may be a vehicle or the like related to a portion of the moving image that moves due to the time difference between the one image and the other image.

Further, for example, when the image pickup device is a camera in a facility described later, the difference between one image and another image is related to a moving part in a moving image due to the time difference between the one image and another image. , Users, managers, service providers, pets, etc. (hereinafter, may be referred to as "users, etc.") in the facility.

Further, either one of the above-mentioned images or the other image may be a background image. For example, in an image captured by a traffic camera that captures a road on which a vehicle or the like passes, an image captured during a time when the vehicle or the like is absent may be used as a background image, and an image captured by a camera in a facility described later. In the above, an image taken in a state where a user or the like is absent in the facility may be used as a background image. Further, even if the image includes an automobile or the like, the image obtained by deleting these images may be used as the background image.

1.1.2. Feature point extraction unit The feature point extraction unit has a function of extracting feature points in a part of the target image from the target image. The feature point extraction unit of one example may extract feature points by the difference between the feature points related to one image and the feature points related to another image in a moving image composed of a plurality of images. The feature point may be a part of the one image or a part of the other image. The relationship between one image and the other image is the same as described above. The feature point extraction unit may extract one or a plurality of feature points from one target image.

The feature point may indicate a part of the object in the image. For example, if the target is a creature such as a person or a pet, it may be a part or all of the head, ears, eyes, nose, mouth, shoulders, elbows, hands, waist, knees, ankles, etc., and the target is a vehicle. If there is, it may be a part or all of bumpers, wheels, roofs, pillars, side schillers, vehicle contours, and so on. All of these are examples, and it is not necessary for each feature point to be specifically linked to a specific point of a person or vehicle, and the point obtained as a feature point by the method used is a feature. It can be a point. In addition, the feature point identification unit may specify other feature points in the image by a well-known method. When one or more feature points identified from the image are associated with one target A, such feature points may be referred to as "feature points of A".

1.2. Partial information relation generation unit 12
The partial information relationship generation unit has a function of generating a relationship of partial information in the target image. In the first target image and the second target image constituting the moving image, the partial information relation generation unit of one example includes the first part information in the first target image and the second part in the second target image. It may have a function of generating a relationship with information.

The relationship may be the presence or absence of a relationship between the first part information and the second part information, or may be an index of the height of the relationship. For example, it may be that there is a relationship between the first part information and the second part information, that there is no relationship between the first part information and the second part information, or that there is a relationship between the first part information and the first part information. The second part may be highly related to information. Whether or not there is a relationship is the relationship between whether the target related to the first part information in the first target image and the target related to the second part information in the second target image are the same in the real world. It may be. In the real world, this identity may be that a single object is the same, a plurality of objects may be the same, and in the case of a plurality of objects, the same type of combination may have different types. It may be a combination. In this sense, it may be referred to as "pseudo-identical object" hereafter. Examples of pseudo-identical objects include people, vehicles, pets, people to people, people to bicycles, people to pets, automobiles and people in them, vehicles carrying vehicles, etc., in short, an imaging device. It may move together as a certain area in the image of the moving image captured by. Further, regarding the height of the relationship, there is a high possibility that the target related to the first part information in the first target image and the target related to the second part information in the second target image are pseudo-identical targets. It may be an index indicating the height. The index may be a numerical value or one of a plurality of ranks given in advance.

Further, the relationship may be a specific correspondence relationship between the first part information and the second part information. For example, the correspondence between the 11th place of the 1st part information and the 21st place on the 2nd part, the 12th place of the 1st part information, and the 22nd place on the 2nd part. It may be a correspondence relationship of. Here, the 11th place, the 12th place, the 21st place, and the 22nd place may be information related to the first part information and the second part information, respectively, and may be a position in an image or a feature point. , Vector, cluster, etc.

The partial information relationship generation unit of the example may store the relationship in association with the partial information. For example, the partial information relationship generation unit of one example may store the relationship between the first part information and the second part information.

1.2.1. When the partial information relation generation unit of the image example can process the image as partial information, the partial information relation generation unit of the example is the first part image related to the first part information and the second part. The relationship with the second part image related to the image may be generated.

The method in which the partial information relationship generation unit of one example generates the relationship between the first part image in the first target image and the second part image in the second target image is a pattern matching between images. , Comparison of densities between images, closeness of positions between images, etc., various methods may be used to show that the first part image and the second part image correspond to the same object. There is no limit to. Note that the first part image and the second part image are both images of a part of the image captured by the image pickup apparatus, and since the area is small, calculations such as processing for generating the relationship are performed. There is an advantage that the burden of quantity is small.

The density between images may be determined by whether or not the difference in the proportion of black or white in the entire image due to the binary image such as the silhouette of the image or the black and white of the image is within a predetermined range. When a relationship is generated using an image expressed with a smaller number of bits than at the time of normal imaging, there are advantages that the calculation speed can be improved and restrictions due to image privacy can be reduced. In particular, when handling image data including people and vehicles in an information processing device, it meets the increasing demand in recent years to reduce restrictions due to privacy.

As the closeness of the positions of the images, for example, the closeness of the position of the partial image 02a in the image 01a and the position of the partial image 02b in the image 01b may be used.

FIG. 2 is an example in which the partial information extraction unit of one example schematically shows the extraction of partial information. In this example, the images 01a to 01d constituting the moving image captured by the imaging device are captured in chronological order in this order, and 02a to 02d, which are some images in each image, are This is an example in which an image related to a difference is extracted as a part of information in each adjacent image. 02a to 02d of some of these images may be a part of the images extracted as the portion related to the movement. The partial images 02a to 02d may be a part of the images 01a to 01d, respectively.

In this example, an example in which the partial information relationship generation unit of one example generates a relationship for a part of images as partial information will be described. For example, the relationship between 02a and 02b may be generated. The partial information relation generation unit of one example may indicate that both 02a and 02b correspond to the same target for some images by pattern matching and the density of the images. When it is determined that the partial information relation generation unit of one example is an image corresponding to the same target, the partial information relation generation unit of one example sets the partial image as the same target in association with one tracking ID. You may remember. FIG. 3 shows an example in which some image IDs 001 and 003006 are associated and stored with the tracking ID 01 of the tracking ID, and some images IDs 002, 004, 009 are associated and stored with the tracking ID 02 of the tracking ID. This is an example. With such a configuration, it is possible to determine where the target appearing in a part of the image is moving even if it moves in different images constituting the moving image.

As shown in FIG. 4, when a plurality of partial images as two or more partial information are extracted from the image 01b, the partial information relation generation unit of one example includes a partial image 02a and a partial image. The relationship between the image 02b and the image 02b and the relationship between the part image 02a and the part image 03b may be generated, and then a part of the plurality of images may be generated. A part of the images having a high relationship with the image 02a may be specified. This has the advantage of increasing the possibility that the target can be tracked more appropriately even when the moving speed of the target in the image is high.

Further, in the case where the above-mentioned plurality of candidate images exist, the partial information relationship generation unit of one example generates relationships in order from the one having the closest proximity, and when the height of the relationship is higher than a predetermined value, , It is not necessary to generate a relationship with other candidate images. In this case, there is an advantage that the amount of calculation is reduced.

1.2.2. When the partial information relation generation unit of an example of a feature point can process a feature point as partial information, the partial information relation generation unit of an example is associated with one or more first feature points related to the first part information. , The relationship with one or more second feature points related to the second part image may be generated. The relationship may be a correspondence relationship between the one or more first feature points and the one or more second feature points, or one generated from the first feature point and the second feature point. Alternatively, it may be a plurality of vectors, or may be one or a plurality of clusters generated from the one or a plurality of vectors.

1.2.2.1. Vector specifying part The vector specifying part has a function of specifying a vector of feature points. The feature point vector may be specified from the preceding and following images. For example, when the second image is next to the first image, a vector having a feature point A in the first image as a start point and a feature point B corresponding to the feature point A in the second image as an end point is specified. Good. For example, the vector unit determines the feature point B in the second image corresponding to the feature point A with respect to the feature point A in one image, and in the method of specifying the vector, the feature point A in the first image. The feature point B in the second image corresponding to may be determined. In addition, as the vector part, a well-known method for specifying a vector may be used.

Here, a vector whose vector is smaller than a predetermined value may be used as it is or may be excluded. The reason why the vector whose vector is smaller than the predetermined value is excluded is that it can be determined that the movement between the first image and the second image is minute, which has an advantage that the amount of calculation can be reduced.

1.2.2.2. Clustering unit The clustering unit has a function of clustering vectors. The clustering unit may generate clustered clusters. Clustering may be based on the position and length of the vector, the length of the vector, or the position and length of the vector. In addition, a well-known clustering method may be used for the clustering unit.

When the clustering unit processes clustering using vector positions, such as a clustering method that classifies the positions of vectors larger than a predetermined value into different clusters, the objects in the image have a certain area, so different objects can be used. There is an advantage that it can be presumed to be such a feature point.

In addition, the clustering unit classifies the length of a vector larger than a predetermined value into the same cluster, or classifies the length of a vector in a predetermined range into the same cluster, and the like. When is used, there is an advantage that feature points related to vectors having the same velocity can be determined to belong to the same object. For example, when the speeds of the first object and the second object are different by a predetermined value or more, even if the positions of the first object and the second object are close to each other, the vector and the first object related to the first object can be accurately measured from the viewpoint of speed. There is an advantage that the vector related to the two objects can be clustered in another classification.

Further, when the clustering unit uses both the position of the vector and the length of the vector, it has both the above-mentioned advantages, and there is an advantage that they can be combined to perform clustering with higher accuracy.

Since the vector is composed of the feature points in the first image and the feature points in the second image, using the position of the vector means the position of the feature points in the first image and / or the position in the second image. The position of the feature point will be used. The position of the vector may be the position of the start point of the vector or the position of the end point of the vector.

FIG. 5 is an example of a cluster generated by the clustering unit, and is an example of a table stored by the clustering unit. In this figure, a cluster is generated for one or a plurality of vectors composed of two feature points, and clusters C1 and C2 are attached.

The partial relationship generation unit of one example can generate a relationship between one or a plurality of first feature points and one or a plurality of second feature points by using the feature points. In particular, when some information of the target image corresponds to two or more objects in the real world, even when the two or more objects make different movements, the correspondence between the feature points is such different movements. There is an advantage that it can be classified by cluster according to the direction of the vector based on. For example, FIG. 6 is an example of a diagram schematically explaining how cluster IDs based on feature points related to some information are related to a time flow of t = 0 to 3. is there. Cluster ID 01 is based on the image at time t = 0, cluster ID 02 is based on the image at time t = 1,

cluster ID

03 and 05 are based on the image at time t = 2, and

cluster ID

04 and 06 are based on the image at time t = 4. Shows that it is based. Here, the cluster related to the feature point related to the image at time t = 1 was one cluster with the cluster 02, but the cluster related to the feature point related to the image at time t = 2 is 2 of

clusters

03 and 05. Since there are two clusters, the cluster 02 is divided into a cluster 03 and a cluster 05.

Here, the flow of one processing example using the partial information relation generation unit of one example will be described. In the first image and the second image constituting the moving image, first, the partial information extraction unit of one example extracts the first partial image from the first image (step 1). Next, the partial information extraction unit of the example extracts the second partial image from the second image (step 2). Next, the partial information relationship generation unit generates the relationship between the first part image and the second part image (step 3).

An example system is
An extraction unit that extracts the first part information related to the first image and the second part information related to the second image.
A generation unit that generates a relationship between the first part information and the second part information,
It may be a system including. When the system of one example can generate the relationship between the first part information and the second part information, it can be specified even if specific information such as a name related to the part information is specified by using a trained model or the like. There is an advantage that some information can be tracked without doing so.

In addition, the flow of other processing examples using the partial information relation generation unit of one example will be described. In the first image and the second image constituting the moving image, first, the partial information extraction unit of one example extracts the first partial image from the first image (step 1). Next, the partial information extraction unit of the example extracts the second partial image from the second image (step 2). Next, the partial information relationship generation unit of the example generates a relationship between the first part image related to the first part information and the second part image related to the second part information, and the relationship is generated. If it is determined that the first part image and the second part image indicate pseudo-identical objects, the relationship is stored. When it is not determined that it indicates a pseudo-identical object, the partial information relation generation unit of one example uses a cluster for the feature points related to the first part information and the feature points related to the second part information. Therefore, the relationship between the first part information and the second part information is generated, and it is determined whether or not the objects are pseudo-identical.

An example system is
An extraction unit that extracts the first part information related to the first image and the second part information related to the second image.
A first generation unit that generates a relationship between a first part image related to the first part information and a second part image related to the second part information.
After the processing of the first generation unit, the first part information and the second part are based on the feature points related to the first part information and the feature points related to the second part information. The second generator that generates the relationship with information,
It may be a system including. Judgment of pseudo-identical objects using feature points compared to comparison between images requires extraction and clustering of feature points, so the comparison between images is prioritized and processed, and the comparison between images is called pseudo-identical objects. When there is an inconvenience in the processing of images such as when the relationship cannot be determined, there is an advantage that the processing amount as a whole can be reduced by performing the processing using a cluster.

An example system is
An extraction unit that extracts the first part information related to the first image and the second part information related to the second image.
A first generation unit that generates a relationship between a first part image related to the first part information and a second part image related to the second part information.
After the processing of the first generation unit, the first part information and the second part are based on the feature points related to the first part information and the feature points related to the second part information. The second generator that generates the relationship with information,
After the processing of the second generation unit, the information generation unit that generates the information related to the first part image and the information related to the second part image by using the trained model.
The relationship between the first part information and the second part information is generated based on the generated information related to the first part image and the information related to the second part image. With the third generator
It may be a system including. If the processing is inconvenient in both the comparison between images and the processing using the cluster, the whole is generated by generating the information of each part of the information using the trained model although the amount of calculation is large. There is an advantage that the relationship can be generated while reducing the amount of processing as.

1.3. Positioning unit 13
The position specifying unit has a function of specifying a position related to some information. The position specifying unit of one example may specify the position related to the image when the partial information is an image, and may specify the position related to the feature point when the partial information is related to the feature point. Here, the position related to the feature point may include the position of the vector generated based on the feature point and the position related to the cluster generated based on the vector. The position specifying unit of one example may store a part of the image related to some information and the position related to the part of the image in association with each other. Further, the position specifying unit of one example may store the information related to the feature point related to some information and the position related to the feature point in association with each other.

The position of a part of the image may be any information as long as it is information indicating the position of a part of the image in the target image. For example, it may be the center of a polygonal image such as a square system including a part of the image, the center of gravity, or the center of the image including a part of the image, the center of gravity, or the like.

The position related to the feature point may be specified by various methods using the information related to the feature point. For example, a position generated by using one or more vectors based on the feature points may be specified as a position related to the feature points. For example, the coordinates of the center point of the coordinates related to the one or more vectors, the center of a polygon such as a quadrangle containing the coordinates related to the one or more vectors, the center of gravity, the coordinates, and the like can be mentioned, but are limited to these. Absent. Here, the one or more vectors may be one or more vectors included in the same cluster.

The former is centered when, for example, one cluster is composed of two vectors, a vector of (x11, y11) (x12, y12) and a vector of (x21, y21) (x22, y22). The x-coordinate of the point may be the average value of x11, x12, x21, x22, and the y-coordinate of the center point may be the average value of y11, y12, y21, y22. Here, the average value may be various methods, such as arithmetic mean, geometric mean, harmonic mean, and generalized mean.

The latter is, for example, a quadrangle when one cluster is composed of two vectors, a vector of (x11, y11) (x12, y12) and a vector of (x21, y21) (x22, y22). The vertices of may be set as (MINx, MINy), (MINx, MAXy), (MAXx, MINy) (MAXx, MAXy) and the like. Here, MINx is the minimum value of the x-coordinate of the four vertices of the quadrangle, MAXx is the maximum value of the x-coordinate of the four vertices of the quadrangle, and MINy is the four vertices of the quadrangle. It is the minimum value of the y-coordinate related to, and here, MAXy may be the maximum value of the y-coordinate related to the four vertices of the quadrangle.

These are just examples, and it is sufficient if the target position can be specified as reflecting the information of each cluster.

1.4. Target number estimation unit 14
The target number estimation unit has a function of estimating the target number of the target related to the partial information when the partial information is the information related to the feature point.

The target number estimation unit in one example uses a rule for determining the number of targets from the number of feature points included in each cluster related to some information (sometimes referred to as "target number estimation rule" in the documents of the present application). The number of targets may be determined.

For example, FIG. 7 may be generated as follows. That is, for one or more sample images, feature points in the image are extracted, clustering is performed for the one or more feature points, and one or more feature points included in one cluster are specified. To specify the number of feature points. Since multiple clusters can be detected, the number of feature points included is determined for each of these clusters. Further, there may be a plurality of clusters (for example, 12 clusters including 5 feature points) having the same number of feature points. In such a situation, in this figure, the horizontal axis is the number of feature points included in one cluster, and the vertical axis is the number of clusters including the number of each feature point. For example, 5 on the horizontal axis is plotted on 12 on the vertical axis, which is a plot showing a situation in which 12 feature points are included in one cluster. In this figure, the

portions

5 and 9 and 13 on the horizontal axis are peaks, and the

portions

7 and 11 and 15 on the horizontal axis are valleys. In such a case, in the sample image, for example, when the number of feature points included in one cluster is 1 to 7, the number of objects is 1 on average, and the number of feature points included in one cluster is 1. In the case of 8 to 11, the number of objects is 2 on average, and when the number of feature points included in one cluster is 12 to 15, the number of objects is 3 on average. The target number may be set for the number. To explain this situation with the function F,
F (x) = 1 1 <= x <= 7
F (x) = 28 <= x <= 11
F (x) = 3 12 <= x <= 15
It may be set as such. That is, a rule for estimating the number of targets may be set from the number of feature points included in each cluster specified by clustering.

In this example, since the number of feature points is 5 which is the peak, it is estimated that the number of feature points is 5 for one target, but when the target is 2. It is 9 instead of 10, and when the target is 3, it may be 13 instead of 15. This is because it may be a mere multiple, but for example, the number of feature points may decrease depending on the degree of overlap of the objects.

Further, the above-mentioned target number estimation rule may be associated with information related to the camera on which the image is captured. Further, the one target number estimation rule associated with the information related to one camera and the other target number estimation rule associated with the information related to another camera different from the information related to the one camera may be different. By setting the target number estimation rule according to the camera, there is an advantage that a precise rule can be set. Here, the information related to the camera on which the image is captured may be the position where the camera to be imaged is installed, the time taken by the camera, the moving method of the camera to be imaged, the specific camera ID, and the like. The position where the camera is installed may be a facility where the camera is installed, a position in the facility where the camera is installed, or the like. The position in the facility may be a position such as a ceiling or a wall. This is because it is considered that there is a certain relationship between the information related to the camera on which the image is captured, the number of feature points, and the number of objects. For example, considering the case where a camera installed in a department store's delicatessen section extracts feature points with a target person as a person, the delicatessen section is composed of multiple sections and a travel route in which shoppers move between them. , The travel route that people take is also limited to certain places. A camera installed in such an environment captures images from a specific direction at a specific tilt. Therefore, when an image of a target person is taken in such a case, the appearance of feature points is also limited. This is because the relationship between the number of objects and the number of feature points is considered to have a certain rule.

Also, the target number estimation rule may be associated with the target. The target may be a person, a vehicle, or an animal, but may be more specific. For example, it may be a person belonging to a predetermined group, a vehicle of a predetermined size or vehicle type, an animal belonging to a predetermined group, or the like. These have the advantage that the target number estimation rule can be set based on the target whose detection mode of the feature point may be different because the extraction of the feature point may be different due to the difference in the group to which the person, the vehicle, and the animal belong. There is. The person who belongs to the predetermined group may include a person in a suit, a person in a casual figure, a person in work clothes such as a factory or a construction workshop, and the like. There is an advantage that the accuracy of the target number estimation can be further improved by determining the target number estimation rule for the person who wears these specific clothes. Further, the vehicle of a predetermined size and vehicle type may be a general vehicle, a motorcycle, a taxi, an emergency vehicle, a truck, a bus, or the like. This is because they are different in size, and taxis may have different detection of feature points, such as a specific mark on the roof. By setting the target number estimation rule according to the specific vehicle size and vehicle type, there is an advantage that the accuracy of the target number estimation can be further improved. In addition, as an animal belonging to a predetermined group, the form may differ depending on the animal, such as bipedal walking, quadrupedal walking, and a difference in morphology. Therefore, the target number estimation rule is determined according to the specific aspect of the animal. There is an advantage that the accuracy of target number estimation can be further improved by setting.

Also, the target number estimation rule may be associated with the clustering method. Since the clusters obtained for the vector based on the feature points may differ depending on the clustering method, there is an advantage that the target number estimation rule can be set based on the clustering method.

In addition, the target number estimation rule may be associated with a predetermined group. The predetermined group may be the above-mentioned information relating to the camera included in the predetermined group, an object included in the predetermined group, a clustering method included in the predetermined group, and / or a combination thereof. Grouping has the advantage of reducing the burden of establishing target number estimation rules according to individual differences.

The target number estimation rule artificially determines the relationship between the number of feature points and the number of targets after a person sees a specific image, and the determined numerical value is input to this system. Therefore, it may be used as a rule for estimating the number of objects, or it may be determined mechanically. In the case of the mechanical determination method, there is an advantage that the human burden is reduced.

<Input means>
When the target number estimation rule is artificially determined, the target number estimation rule unit may include a configuration for inputting information related to the target number estimation rule and a configuration for acquiring information related to the target number estimation rule. .. The information related to the target number estimation rule is not limited in its mode as long as it is information necessary for determining the target number estimation rule. For example, the number of feature points and the number of targets are directly or indirectly associated with each other. It may be information. FIG. 8 shows an example of a screen for acquiring such information.

In this figure, in the histogram of the number of feature points included in one of the above-mentioned clusters and the number of clusters in each feature point, the bar is moved left and right as shown in "the boundary between the number of

objects

1 and 2" (01). A configuration for inputting a target number estimation rule and a configuration for acquiring a target number estimation rule may be provided as a boundary of the target number by using a GUI. This method has an advantage that the user can input the target number estimation rule while understanding the relationship between each cluster and the feature score on the screen. In this case, the target number estimation rule unit may acquire information on the boundary of the target number and the number of feature points. Although an example of a histogram is shown here, various graphs such as a bar graph, a line graph, a pie graph, a band graph, and a scatter graph may be used. Moreover, the input means may also have various modes. Further, in this figure, a numerical value may be directly input in conjunction with or instead of the above. For example, as in "Relationship between the number of targets and the number of feature points" (02), a numerical value may be input for the relationship between the number of targets and the number of feature points. In this case, the target number estimation unit may be used. Such information may be obtained. In this case, there is an advantage that a specific numerical value can be input.

In addition, as a method of mechanically determining the target number estimation rule, for example, machine learning can be mentioned. Various artificial intelligence techniques may be used for machine learning. Artificial intelligence technologies include, for example, machine learning technologies such as neural networks, genetic programming, functional logic programming, support vector machines, clustering, regression, classification, Bayesian networks, reinforcement learning, expression learning, decision trees, and k-means clustering. May be used.

As the machine learning technology using the neural network, deep learning (deep learning) technology may be used. This is a technique that enables output even for an unknown input by learning the relationship between input and output using a plurality of layers. There are supervised and unsupervised methods for training data, whichever method may be applied.

This system may use such machine learning technology to learn the relationship by inputting the number of one feature point and the number of objects, or the image including the number of feature points and the number of objects. You may learn such a relationship by inputting.

Although it partially overlaps with machine learning technology, the target number estimation rule may be specified by using the number of feature points and the number of targets by a statistical method. If the target number estimation rule is determined before the analysis of the captured moving image, it is not necessary to determine it at the time of analyzing the captured moving image. Therefore, the calculation amount and calculation time are used to determine the target number estimation rule. Even if machine learning is required, it does not affect the amount of calculation or calculation time of video analysis. Therefore, even if the processing of the target number estimation rule requires a large amount of calculation time and calculation amount due to machine learning or the like, there is an advantage that the analysis of the moving image can be executed in real time by this processing explained in the migration.

The target number estimation rule artificially or mechanically determined by the above-mentioned method may be used by various methods. For example, the target number estimation rule may be prepared by a table, a function, or may be used as a machine-learned system. The relationship between the number of feature points obtained by machine learning and the number of objects is such that the number of objects obtained by inputting the number of feature points on a trial basis is made into a table or function as described above. You may be able to.

An example system is
An extraction unit that extracts one or more first feature points as the first part information related to the first image, and
A target number estimation unit that estimates the number of targets related to the first part information by applying the target number estimation rule from the one or a plurality of first feature points.
It may be a system including.

The target number estimation unit has a function of estimating the number of targets for one cluster by using the above-mentioned target number estimation rule for the number of feature points in the cluster. In general, since feature points are associated with each object, many clustering studies have been conducted to classify feature points according to actual objects, but such a method has been extremely difficult. On the other hand, when the target number estimation unit uses the target number estimation rule, the number of feature points is determined by the target number estimation rule regardless of the clustering technique from low-precision clustering technology to high-precision clustering. Since the relationship with the number of targets is known, there is an advantage that the number of targets associated with one cluster can be reasonably specified from the number of feature points in the one cluster. In particular, even if a low-precision clustering technique is used, the above-mentioned target number estimation rule has an advantage that the number of target targets associated with one cluster can be reasonably specified from the number of feature points in the one cluster. There is.

According to the above method, according to the target number estimation rule, clustering is performed on a plurality of feature points related to one cluster having a target number of 2 based on a vector between images along the flow of a moving image. At some point, the clustering technique may result in more than one cluster. At this time, the number of targets for each cluster may be determined based on the target number estimation rule based on the number of feature points related to each cluster.

Further, as described above, the target number estimation unit may use any clustering technology, and therefore, a smartphone having a clock number of the arithmetic unit that processes the clustering technology used by the target number estimation unit or the like. A calculation unit that uses a CPU with a relatively small number of clocks, such as a lightweight notebook computer, or a smartphone, lightweight notebook computer, etc. that has a storage capacity used for processing the clustering technology used by the target number estimation unit. When the storage unit has a memory with a relatively small amount of storage such as, and / or when the clustering technique used by the target number estimation unit needs to be performed at the same time as other image processing or other calculation with a high processing load. There is an advantage that it can be applied even in such cases.

In addition, when a clustering technique with a low calculation load is used in the target number estimation unit, there is an advantage that the number of targets can be determined in real time.

In particular, when the detection of feature points is limited to moving vectors and, as described above, a clustering technique with a low amount of calculation is used, there is an advantage that the total amount of calculation can be reduced.

Further, when the system of one example is provided with the target number estimation unit of one example, there is an advantage that the number of targets can be specified even when the cluster is associated with a plurality of targets. In particular, there is an advantage that such a movement can be identified even when one of a plurality of objects starts a different movement.

An example system is
An extraction unit that extracts some information from the image captured by the image pickup device,
For the cluster related to the partial information, the target number estimation unit that determines the number of targets related to the partial information, and
It may be a system including. Such a target number estimation unit can estimate the number of targets regardless of whether or not the system of one example has an information generation unit capable of generating information by a function with a large amount of calculation such as a learned model described later. There are advantages.

1.4. Information generation unit 15
The information generation unit has a function of generating information related to some information. When the system of one example has an information generation unit, it can generate information related to some information, so that there is an advantage that the accuracy of the information related to some information is improved. The information generation unit may generate target information related to some information by various methods. For example, the information generation unit of one example may be configured so that an image related to a part of the information can be acquired and the target information can be generated by machine learning. The information related to the partial information may be the name or attribute of the object related to the partial information. For example, it may be a name, type, size, attribute of a vehicle or the like. In particular, in the case of a person, it may be information indicating a person such as gender, age, age group, height, physique, facial expression, etc. from a specific viewpoint, or information on fashion items.

The machine learning function of the information generation unit in the example may be for machine learning the relationship between the image or a part of the image and the target information. In addition, the machine-learned information generation unit may generate target information corresponding to the image or a part of the image with respect to the image or a part of the image.

Various artificial intelligence technologies may be used for machine learning. Artificial intelligence technologies include, for example, machine learning technologies such as neural networks, genetic programming, functional logic programming, support vector machines, clustering, regression, classification, Bayesian networks, reinforcement learning, expression learning, decision trees, and k-means clustering. May be used. In the following, an example using a neural network will be used, but the present invention is not necessarily limited to the neural network.

As the machine learning technology using the neural network, deep learning (deep learning) technology may be used. This is a technology that enables output even for unknown inputs by learning the relationship between inputs and outputs using a plurality of layers. There are supervised and unsupervised methods, either of which may be applied.

Machine learning in deep learning technology often requires a large number of calculations in learning the relationship between input and output. Therefore, in particular, when a part of the entire image captured by the image pickup device is applied to the learned information generation unit, there is an advantage that the amount of calculation can be suppressed. In addition, a part of the image in the target image related to the moving part in the moving image captured by the imaging device is compared with the image related to the non-moving part in the moving image, and the target (for example, a person, a pet, a vehicle, etc.) ) Since there are many areas that overlap with the captured image, when a part of the image in the target image related to the moving part of the moving image captured by the imaging device is applied to the learned information generation unit, There is an advantage that the target information can be easily generated with a small amount of calculation. In particular, there is an advantage that the target information can be generated by applying the trained model not only by the GPU suitable for the neural network but also by the CPU of a notebook computer, a smartphone, or the like.

Also, one example system is
An extraction unit that extracts some information from the target image,
A generation unit that generates target information related to the partial information based on the extracted image related to the partial information.
With
The image related to the partial information may be a system in which the amount of partial image information is smaller than the amount of image information of the portion corresponding to the partial in the target image. For example, in the image related to some information, the amount of image information is reduced in the corresponding portion as compared with the original target image. Compared with the original target image captured, the amount of image information is reduced at the corresponding portion, so that there is an advantage that the burden of application to the trained model can be reduced.

Further, in the image related to the partial information according to the system of one example, the amount of image information of a part of the background region is smaller than the amount of image information of the portion corresponding to the part in the target image. It can be a system. For example, in the image related to some information, the amount of image information is reduced in the corresponding portion in the background area as compared with the original target image. Since the amount of image information is reduced in the corresponding part of the background area where the required information seems to be less than the original target image captured, learning is performed without significantly adversely affecting the accuracy of the information. There is an advantage that the burden of application to the finished model can be reduced.

Further, the image related to the partial information related to the system of one example may be a system in which the amount of partial image information is smaller than the amount of other image information other than the partial information. For example, as for the image related to some information, there is an example in which the amount of image information is partially reduced as compared with other images other than the above part. Some have the advantage that the burden of application to the trained model can be reduced because the amount of image information is smaller than that of others.

Further, the image related to the partial information related to the system of one example may be a system in which the amount of image information of a part of the background area is smaller than the amount of image information other than the background area. For example, as for the image related to some information, there is an example in which a part of the background area has a smaller amount of image information than other than the background area. Some of the information in the background area, which seems to require less information, has less image information than others, reducing the burden of applying it to trained models without significantly adversely affecting the accuracy of the information. There are advantages that can be done.

Further, the image related to the partial information related to the system of one example may be a single image information amount per unit of at least a part of the background area. The single image information amount may be, for example, a single color. The single color may be, for example, 0, 255, etc. in the RGB model. For example, if the color of the background area is set to 0 or 255 in the RGB model, when the trained model is applied, the target information can be displayed without being confused by various background information. There is an advantage that processing is possible without excessively large calculation load. In this case, for example, the amount of image information related to the target other than the background area may have the same amount of information as the amount of image information related to the target in the original target image captured. In that case, for example, since the target color information, texture information, and the like are included in the image related to some information, the trained model can be applied to generate the target color and target type information. There are also advantages such as.

Here, the background area may be an image calculated from the difference between at least two images from a plurality of images constituting the moving image, or the background image captured by the imaging device and the background image captured by the imaging device. It may be a difference from the image. The background image may be an image captured before the difference processing is performed, and may be, for example, an image captured in advance. The background area may be, for example, an image captured by a traffic camera other than a vehicle or the like, or an image captured in a commercial facility other than a user. In the target image, the background area is a place other than the target for generating information such as vehicles. Therefore, when the trained model is applied by reducing the amount of information about the background area, the information of the background area is applied. There is an advantage that information can be processed without being confused and the amount of processing is dramatically reduced. According to the research by the inventors, the number of images used is reduced to about 1/100, so that it is possible to calculate with an arithmetic unit other than the arithmetic unit conventionally required for applying the trained model. .. When the amount of image information in the background region is small and the neural network is applied as a trained model, there is an advantage that the structure of the neural network can be simplified.

Further, the image related to the partial information related to the system of one example may have a single image information amount per unit other than the background area. Except for the background area, for example, if the image is captured by a traffic camera, a vehicle or the like is presumed, and if the image is captured in a commercial facility, the user is presumed. As described above, the single image information amount may be, for example, a single color. The single color may be, for example, 0, 255, etc. in the RGB model. For example, in the RGB model, the color of the background area is set to 0, the color other than the background area is set to 255, and the trained model is applied to the image related to the binarized partial information, thereby forming the background and the inside of the object. There is an advantage that the target information can be generated without being confused by various information and mainly using the contour of the target without excessively large calculation load.

The shadow of the target may be included as one of the background areas, may be included in a target other than the background area, or may have a single image information amount as described above. Good. For example, the image related to the partial information related to the system of one example is converted into ternary partial information such as 0 for the color of the background area, 255 for the target, 128 for the shadow of the target, and the like in the RGB model. The trained model may be applied to the image related to the image and the image related to the partially quantified information. In this case as well, there is an advantage that the target information can be generated without excessively increasing the calculation load mainly by the information such as the outline and shadow of the target. When the shadow of the target is included as one of the background areas, for example, the degree of the shadow may differ depending on the weather, but there is an advantage that such a difference does not affect the application of the trained model.

Further, the trained model may be selected and used in association with the place where the image is taken. For example, there are cases where the imaging target such as a traffic camera or a camera in a facility is clearly defined as a vehicle or a user. Therefore, a trained model in which the relationship between the vehicle or the like and the name is machine-learned may be applied to the target image captured by the traffic camera. In this case, there is an advantage that information on the vehicle or the like can be generated from the obtained information on the premise that the vehicle is installed in the traffic camera without burdening the calculation amount. Similarly, a trained model in which the relationship between the user or the like and the name is machine-learned may be applied to the target image captured by the camera in the facility. In this case, there is an advantage that information on users and the like can be generated from the obtained information on the premise that the information is installed in the facility without burdening the calculation amount. In these cases, when the amount of image information in the background area is reduced or the amount of image information other than the background area is reduced, there is an advantage that the amount of calculation can be reduced in applying the trained model. In this case, the trained model may be stored in association with the installed image pickup device, and the system of one example has a configuration in which the trained model to be applied in association with the image pickup device in which the target image is captured can be selected. May be. Further, as described above, the imaging device is divided into a traffic camera, an in-facility camera, or a group according to various uses described later, and the trained model for each group is provided or can be connected to provide information. The trained model may be configured to be selectable based on the imaging device related to the generated partial information.

Similarly, an example information generator includes a trained model for functioning a computer to generate information of interest corresponding to the image or part of the image based on the image or part of the image. Good. Such trained models may be implemented by programs that are part of artificial intelligence software.

The above-mentioned function related to machine learning may be a function provided in the information generation unit of the example, or may be inquired from the system of the example in an information processing device outside the system of the example. Further, the information generation unit of an example includes a transmission unit that transmits an image or a part of an image to an external information processing device having a function related to machine learning, and a target corresponding to the image or a part of the image. It may be provided with an acquisition unit for acquiring information. Further, the information generation unit of the example may store such an image or a part of the image in association with the acquired information of the target object.

An example system is
An extraction unit that extracts some information from the image captured by the image pickup device,
An information generation unit that generates target information related to partial information with respect to the image related to the partial information.
It may be a system including.

1.6. Boundary 16
The boundary portion has a function of managing the boundary. The boundary may be any one that divides at least a part of a certain area. What divides the area may be a line segment, an arc, or the like, or a circle, a polygon, or the like. Boundaries may generate closed areas or may not be closed, such as partial lines.

For the boundary part, the boundary may be set for the image. Boundaries may associate one or more boundaries with an image and may store them. The boundary portion may store the boundary as a function or as a graph consisting of vertices and paths.

FIG. 9 shows an example of the boundary. The boundary may be a circle (01), a polygon such as a quadrangle (02), or a boundary (03) that is a part of a closed area together with the edge of the image or the edge of the screen on which the image is displayed. It may be a simple line segment (04) that is not closed as an area, an arc (05), or other non-regular lines. Considering the boundary as the vertices and paths of the graph, the boundary may or may not be a closed path. The boundary may be stored by a so-called graph in the storage unit, and may be a static memory such as a matrix, a list, a structure, or a dynamic memory, and these may be dynamically compiled. Alternatively, it may be selected according to the means for execution processing such as static compilation. The former has the advantage that performance can be improved by using information available at runtime, and the latter has the advantage that performance can be improved within the range using known information. Further, the graph may be a directed graph or an undirected graph.

Further, the statistical processing unit of one example may associate one direction with respect to one boundary. Since the boundary has a role of detecting the intersection with the position of some information, the direction related to the boundary is information for determining from which side the partial information has moved to the other side with respect to the boundary. May be used as.

The boundary has the advantage that it may be possible to detect the movement of some information even if the road is not closed. For example, if the image is an image of the sales floor of a store, people cannot pass through the places where shelves and things are installed, so if a boundary is set in a place other than such shelves and things, it will be called such a boundary. This is because the movement of some information can be detected by the intersection with the position related to some information. However, when the boundary is not a closed road, it is not always the case that the target path is excluded in relation to the image, and the closed road is set to an arbitrary movement point of some information that the user wants to detect. Good.

In addition, the boundary part may provide a function that allows the user to input the boundary. If the user has such a function that the boundary portion is applied, the user has an advantage that the boundary portion can be input. Further, the boundary portion may have a function of displaying a screen that supports input of the boundary. Having such a function of the boundary has an advantage that the user can understand the information regarding the input of the boundary. In addition, the boundary portion may have a function of acquiring information related to the boundary input by the user. When the boundary portion has such a function, there is an advantage that the boundary portion can use the information related to the boundary input by the user.

FIG. 10 is an example of a screen display provided by the boundary portion where the user can input the boundary. The boundaries themselves may be set graphically using a pointing device such as a mouse or pointer. It may be possible to select a plurality of boundaries to set the region (1003), or it may be possible to deselect the boundaries (1007). Boundaries on the screen may be deleted individually (1008) or all may be deleted (1009). Further, the name of the image (1001) and the name of the area (1002) can be displayed. Further, the set boundary is memorable (1005), and the boundary can be set one after another so that the boundary can be set for various screens (1006).

Further, the system of one example may be a system provided with a specific unit for specifying the position of the target by using the estimated number of targets for one cluster in the image.

Further, the system of one example is a system provided with a specific unit for specifying the position of the object and a statistical processing unit using the estimated number of objects for one cluster in the image. It may be.

1.7. Statistical processing unit 17
The statistical processing unit has a function of generating statistical information. Statistical information is information generated by using some information directly or indirectly. The statistical information itself or the information necessary for generating the statistical information may be stored in a storage device (hereinafter, may be referred to as a "statistical information database"). Here, the term database only means a collection of data, and may or may not have a large-scale data storage function, a rapid data access function, and the like. .. Such a statistical information database may be provided by an example system, or may be inquired from an example system in an information processing device outside the example system. An example system may have a communication unit for inquiring information from the statistical information database and an acquisition unit for acquiring information acquired from the statistical information database.

Statistical information may be information related to the position related to some information or information related to the boundary. Further, such statistical information may be generated when the relationship based on some information is a pseudo-identical object.

The information related to the position related to the partial information may be the movement direction of the position related to the partial information, or may be the movement of the position related to the partial information.

For example, the moving direction of the position related to the partial information may generate the moving direction of the position related to the partial information. The moving direction may be specified by a numerical value such as 0 to 360 degrees or 0 to 2π, or may be specified by a value associated with a predetermined range such as north, south, east, west, up, down, left, or right. .. Further, the degree of width of the predetermined range may be various. There is an advantage that it is easier for the viewer to understand when the information is displayed even when the information is specified by the predetermined range. The moving direction of the partial information may be generated by the vector of the position related to the image related to the partial information, or may be generated by using the information of the vector of the feature points related to the partial information. Further, the moving direction of the position related to the partial information may be generated by using a plurality of partial information. In this case, an average of vectors based on positions related to a plurality of partial information may be used.

An example system may have a statistical processing unit that generates a moving direction from a position related to the first part information and a position related to the second part information.

An example system is
In the first image and the second image that make up the moving image,
A partial information extraction unit that extracts the first part information from the first image and the second part information from the second image.
A position specifying unit that specifies a position related to the first part information and a position related to the second part information.
A statistical processing unit that generates a moving direction from a position related to the first part information and a position related to the second part information.
May be equipped. When one system has such a configuration, there is an advantage that information on the moving direction can be easily generated. In such a case, the system of the example is possible without the function of estimating the number of targets and the function of the boundary in generating the moving direction of such a part of information. On the other hand, when the system of one example uses the target number estimation function or uses the number of partial information, it is displayed because it can generate information on the number of targets and / or the number of partial information in addition to the moving direction. In this case, the viewer has the advantage of being able to understand how many targets the movement is. In addition, the system of one example may use the function of the boundary to specify the timing of generation and display of the moving direction. For example, there is an advantage that the viewer can display at a specific position by being configured to generate and display the moving direction at the position where the boundary is set.

FIG. 11 is an example of displaying the moving direction of the position related to some information, the display 1101 shows the moving direction to the information, and the display 1102 shows the moving direction to the right.

Further, as for the movement of the position related to the partial information, in the plurality of images captured by the imaging device, one partial information is used for one image, and a plurality of corresponding images are used for the plurality of images. Partial information may be generated based on each position related to the plurality of partial information. Further, the movement of the position related to the partial information may be generated by using a plurality of partial information in one image. As statistical information, the statistical processing unit of one example may store information in which some information is associated with a position related to some information.

An example system may have a statistical processing unit that generates a movement of a part of information from a position related to the first part information and a position related to the second part information.

An example system is
In the first image and the second image that make up the moving image,
A partial information extraction unit that extracts the first part information from the first image and the second part information from the second image.
A position specifying unit that specifies a position related to the first part information and a position related to the second part information.
A statistical processing unit that generates motion from the position related to the first part information and the position related to the second part information.
May be equipped. When the system of one example has such a configuration, there is an advantage that information on the movement of some information can be easily generated. Further, as the movement, the movement of a part of the information for a predetermined period may be displayed, or the movement of a predetermined distance may be displayed. In this case, the viewer has the advantage of being able to understand certain past movements. In such a case, the system of the example is possible without the function of estimating the number of objects and the boundary in generating the movement of such a part of information. On the other hand, when the system of one example uses the target number estimation function or uses the number of partial information, it can generate information on the number of targets and / or the number of partial information in addition to the movement, so that it is displayed. , The viewer has the advantage of being able to understand how many objects are moving. The example system may also use the boundary function to specify the timing of motion generation and display. For example, by being configured to generate and display a movement at a position where a boundary is set, there is an advantage that the viewer can display at a specific position.

FIG. 12 is an example of displaying the movement of the position related to the partial information, the display 1201 is an example showing the movement of the curve of the partial information, and the display 1202 shows the linear movement of the partial information. This is an example.

Further, the system of one example may display the moving direction and movement of the position related to some information. In this case, the viewer has the advantage of being able to understand both past movements and current movement directions.

Further, the information related to the boundary may be information generated in relation to the boundary, for example, the number related to the position of the partial information intersecting the one boundary or the partial information intersecting the one boundary. The number of such objects may be the number used, and so on. When the system of one example has such a configuration, there is an advantage that information indicating the relationship between the position and the boundary of some information can be generated.

The number of the partial information intersecting the one boundary is, for example, the total number of the partial information intersecting the one boundary, the total number of the partial information in the predetermined period, and the predetermined period. It may be an average value of the number of the partial information in a unit, a density obtained by dividing the total number by a numerical value related to the boundary, or the like. Here, the numerical value related to the boundary may be, for example, the area associated with the boundary or the rent associated with the boundary. These may be the area or rent of the closed area if it is a closed boundary, or the virtual area or rent associated with that boundary if it is not a closed boundary. If one system has such a configuration, it has the advantage of being able to generate statistical information based on the location of some information. In particular, there is an advantage that quantitative information of some information can be generated by the total number, the average value, and the like. Further, in the case of density, there is an advantage that the utilization efficiency of such an area associated with the boundary and the consideration in relation to the rent can be generated by the area associated with the boundary and the rent.

The number used for the number of objects related to the partial information intersecting the one boundary is, for example, the total number of the objects related to the partial information intersecting the one boundary, the partial information in a predetermined period. It may be the total number of such objects, the average value of the number of objects related to the partial information in a predetermined period unit, the density obtained by dividing the total number by the numerical value related to the boundary, or the like. When one system has such a configuration, there is an advantage that statistical information including the number of objects related to some information can be generated.

In addition, when one boundary is associated with one specific place, one specific product location, one specific advertisement location (hereinafter, also referred to as "boundary location, etc."), etc. , The information related to the boundary is the total number of some information that intersects the one boundary, etc. associated with the specific boundary place, etc., the total number of the predetermined period, the average value in the predetermined period unit, and the like. You can. When an example system generates such information, it has the advantage of being able to generate statistical information that includes elements such as boundary locations. Here, when such a boundary place or the like is used, the relationship between one boundary and the boundary place or the like may be related and stored in the storage device (hereinafter, may be referred to as a “boundary place database”). Here, the term database only means a collection of data, and may or may not have a large-scale data storage function, a rapid data access function, and the like. .. Such a boundary location database may be provided by an example system, or may be inquired from an example system in an information processing device outside the example system. An example system may have a communication unit for inquiring information from the boundary location database and an acquisition unit for acquiring information acquired from the boundary location database.

In addition, the information related to the boundary may include a store as one specific place. In this case, the information related to the boundary may include the number of visits when a person who is the target of some information visits one store.

Further, the information related to the boundary may be various rankings as described later.

In addition, the statistical processing unit of one example generates statistical information by using a partial information relation generation unit, a position identification unit, a target number estimation unit, an information generation unit, a boundary unit, and / or a tracking unit described later. Good.

1.8. Tracking unit 18
An example system may have a tracking unit. The tracking unit has only the function of tracking the movement of some information in the moving image captured in one imaging device, from the moving image captured by one imaging device to the moving image captured by another imaging device. It may have only the function of tracking the movement of the object, or both of these functions. In addition, the tracking unit may track the movement of the position related to some information.

The tracking unit may have a function of determining whether or not the target related to some information is a pseudo-identical target. The tracking unit may determine whether or not it is such a pseudo-identical object by using the feature points related to the cluster related to some information, the positional relationship of the imaging range of the imaging apparatus, and / or the time information of such imaging. .. In addition, the tracking unit may determine whether or not they are pseudo-identical objects by using the above-mentioned partial information relation generation unit. The tracking unit may track the target related to the partial information when it is determined that the target related to the partial information is the same.

When the tracking unit uses the feature points related to the target, for example, the positional relationship of one or a plurality of feature points may be used to determine whether the target is a pseudo-identical target. This is because the positional relationship of one or more feature points related to one object changes between the position in the image related to the moving image by one imaging device and the position in the image related to the moving image by another imaging device. Based on estimates that there is no or the change is constant. The constantness of change can occur due to the difference between the installation angle and position of one imaging device and the installation angle and position of another imaging device, but a rule of such positional relationship is created in advance. As a result, it may be used as information for determining whether or not the object is the same pseudo-identical object. If it is difficult to create a rule of such a positional relationship, the positional relationship of the feature points may not be used.

Further, when the tracking unit uses the positional relationship of the imaging range of the imaging device, for example, the imaging range of one imaging device and the imaging range of another imaging device are adjacent to each other or partially overlap. There may be a relationship. When the imaging range of one imaging device and the imaging range of another imaging device are adjacent to each other, if the feature points move as they are, the feature points in the imaging range of one imaging device are other. Since it is estimated that the object moves to the imaging range of the imaging device of the above, it is estimated that the object within the imaging range of one imaging device has moved to the object within the imaging range of the other imaging device, thereby simulating the object. It may be determined whether they are the same object. Further, when the imaging range of one imaging device and the imaging range of another imaging device partially overlap, the feature points in the overlapping range are the imaging range of one imaging device and the other. Since it exists at the same time as the imaging range of the imaging device of the above, it may be determined whether or not the object is a pseudo-identical object by determining the identity of the feature points by using such information.

Further, when the tracking unit uses the time information of imaging, when the imaging range of one imaging device and the imaging range of another imaging device are adjacent to each other, the feature points move as they are. If so, it is presumed that the feature points in the imaging range of one imaging device move to the imaging range of another imaging device. By using the time information, such a relationship is estimated and the feature The identity of the points may be determined to determine whether the object is a pseudo-identical object.

As described above, the tracking unit of the system of one example may detect the movement of the target across each moving image by a plurality of imaging devices, thereby specifying the movement of the target.

The tracking unit may store information relating the position and time of the target based on the information on the movement of the target. FIG. 13 is an example of such a memory.

The tracking unit may have a function of identifying the position of the target in a specific period from a specific time to a specific other time by using such information, and obtains the position of the target. May have the function of

Further, the tracking unit may have a function of specifying a specific position for a specific time zone and a target related to a part of information by using the boundary location database. This has the advantage that, for example, it is possible to identify where an object was at a certain time zone. In addition, the tracking unit may have a function of specifying one or more partial information or an object related to the partial information for a specific time zone and a specific place by using the boundary location database. This has the advantage that, for example, it is possible to identify what kind of object was in a specific time zone and a specific place. In addition, the tracking unit may have a function of specifying a specific time zone for a target related to a specific place and a part of information by using the boundary place database. This has the advantage that, for example, it is possible to identify when a specific person was in a specific place.

FIG. 20 shows the processing flow of an example system using each of the above functions. Here, in the flows 2002 to 2008, the order of generating each information may be appropriately changed as long as the information necessary for generating the information is prepared. Further, in this figure, the processing of the flow 2002 to 2008 is shown after the first part information and the second part information are generated, but this is an example, and the position is determined after the generation of the first part information. Information that is a prerequisite for processing each function, such as identifying and then generating the second part information, and managing the boundary part (2006) before generating the first part information (2001), is available. When they are complete, the processing flow may be changed so that they can be processed.

2. 2. Example of Embodiment
2.1. <One system example>
An example system may include the configuration illustrated in FIG. In this figure, the system of one example may be composed of the information processing apparatus 1410. The information processing device 1401 may include an arithmetic unit 1411, a storage device 1412, a bus 1415, and a communication device 1416. The arithmetic unit 1411 may have an arithmetic function, may be a CPU, a GPU, or the like, and may have a function capable of executing various instructions. Further, it may have a storage function such as a cache. The storage device 1412 may be a storage device having a storage function, may be a primary to tertiary storage device, or may be a volatile / non-volatile memory. The storage device may store a program that can be executed by software in the documents of the present application, and information scheduled to be processed or information after processing by an instruction related to the program. Further, the memory may be a temporary memory or a permanent memory. The bus 1415 may have a function of transmitting information in the information processing device. Further, the communication device 1416 may have a function for transmitting information via a network.

In this figure, the information processing device 1401 is provided with one arithmetic unit and one storage device, but a plurality of arithmetic units may be provided, and a plurality of storage devices may also be provided. Further, the arithmetic unit may be various types of arithmetic units, and the storage device may also be various types of storage devices. Further, the information processing device 1401 itself may be composed of one information processing device, or may be composed of a plurality of information processing devices. Further, the information processing device 1401 may be on the cloud or may be a server or the like.

An example system may include an AI device 1418. Although the AI device 1418 is shown in this figure so as to be connectable to the information processing device 1410 via the network 1417, the AI device 1418 may be inside the information processing device 1410 or may be directly connected to the information processing device 1410. .. In this case, since it is not necessary to go through the network 1417, there is an advantage that information can be communicated quickly.

An example system may include one or more imaging devices 1401a to c in addition to the information processing device 1401. The imaging device will be described later.

Further, the system of one example may include a display device 1414 and / or an input device 1413. In this figure, the display device is shown as a display device in the information processing device 1410, but the display device may be an information processing device independent of the information processing device 1410. Further, there may be one or more information processing devices related to such a display device, and the information processing device may be via the network 1417. Further, the information processing device related to the display device may be a terminal, may be used by an administrator who manages the system of this example, or may be used by a user who uses the system of this example. .. Further, an input device 1413 may be provided corresponding to such a display device. The input device may be a touch panel integrated with the display device, or may be a separate device. The input device may be a keyboard, pointer, mouse, or the like. An example system may or may not include these input devices and / or display devices. Further, when the display of information is explained in the documents of the present application, the system of one example is a transmission unit capable of transmitting such information to the information processing device related to the display device so that the display can be performed. May be equipped. Further, when the input of information is explained in the documents of the present application, the system of the example includes an acquisition unit capable of acquiring the information from the information processing device related to the input so that the input can be performed by the input device. Good.

Next, an application example of an example system will be described with reference to FIG. This figure assumes, for example, one floor of a department store, but shopping malls, stores, offices, financial institutions, medical facilities, accommodation facilities, government offices, educational facilities, cultural facilities, sports facilities, factories, aircraft facilities. Needless to say, it is not limited to these, such as vehicle facilities and their affiliated facilities, and it is possible to think in the same way both inside and outside the facility as long as it is a place where one or more imaging devices are installed.

Examples of the outside of the facility include outside the above-mentioned facility, a farm, a mine, a mountain field, or an artificial object constructed in these, and an imaging device described later may be installed in these places.

Further, the imaging device described later may be installed in a place inside or outside the above-mentioned facility, but it does not have to be installed in a specific place. For example, by installing it in a drone in flight in the air, it may be possible to take an image from a viewpoint different from that of the facility.

In this figure, an imaging device (1A, 1B, 1C, 1D), an imaging range (2A, 2B, 2C, 2D) of the imaging device, and a person (3 to 5) are schematically illustrated. There is.

The image pickup device may be various types of image pickup devices. For example, it does not matter what kind of camera it is, such as a digital camera or an analog camera. The purpose of imaging may also be crime prevention, monitoring of visitors and employees, marketing, recording, and the like. The camera may be a stereo camera, a panoramic camera, a construction camera, an underwater camera, or the like. The lens may be an ordinary lens, or a wide-angle lens such as a fisheye lens, an ultra-wide-angle lens, or a wide-angle lens may be used. There is an advantage that one camera can shoot a wider range than a normal lens. In particular, when it is desired to image a wide area, there is an advantage that the number of image pickup devices can be reduced by using a wide-angle lens as compared with the number of image pickup devices using an ordinary lens.

Each imaging device may capture a moving image or a large number of still images. The partial information extraction unit of one example may extract partial information from the image based on these. The target of the partial information may be, for example, a person, a pet, a vehicle, or the like. The number in the information related to the boundary may be the number of people, the number of pets, the number of vehicles, and the like.

The statistical processing unit of one example may specify the moving direction of the partial information by using the partial information. Depending on the moving direction, there is an advantage that the moving direction of the target represented by the partial information can be arranged.

In the case where the system of one example is a cluster of people and some information related to feature points, in this figure, person 3 is one person, person 4 is two people, person 5 is three people, and so on. The number of targets may be specified for each cluster. An example statistical processing unit may store such information. For example, FIG. 16 shows an example stored by the statistical processing unit of one example. In this way, the statistical processing unit of one example may generate information indicating the direction in which an object such as a person moves. By generating the moving direction of the target, there is an advantage that the moving direction of the target can be arranged.

FIG. 17 is an example in which the movement direction generated by the statistical processing unit of one example is displayed. In this figure, the direction in which each person (3 to 5) moves is indicated by an arrow. By displaying the moving direction of the target, the viewer has an advantage that the moving direction of the target can be understood. The display mode is not limited to the arrow, and may be various modes that the viewer can understand.

Further, the statistical processing unit of one example may display the information related to the boundary in various images. For example, although the statistical processing unit is the above-mentioned example as the information relating to the boundary, the total number of some information intersecting the boundary for a predetermined period may be used. The total number of such numbers may be displayed using an image. Such an image may be of various kinds as long as it can be intuitively felt by a person. , The amount of lines, the color, or the mark indicating the number of people. The transparency of patterns and gradations indicates that the lower the transparency, the higher the number, the higher the transparency, the lower the number, and the higher the amount of lines, the higher the number, and the lower the amount of lines, the lower the number. , The more reddish the number is, the smaller the number is, and the more bluish the color is, the more the number is related to the human sensitivity. Further, the mark may be a predetermined mark. For example, a mark having a large number, a mark having a medium number, a mark having a small number, and the like may be predetermined, and such a mark may be displayed in a target area. The pattern may also be associated with a time zone and displayed by an animation showing the number in each region in each time zone. In this case, the viewer has the advantage that it is easier to understand the slight change in the number at each time.

In addition, information related to boundaries as statistical information may be generated in real time. When generated in real time, the number related to some current information may be displayed without calculation such as the total number in a predetermined period, such as 1 minute, 3 minutes, 5 minutes, 30 minutes, 1 hour, etc. The total number in a relatively short period of time may be displayed depending on the browsing situation. When some of the numbers are displayed in real time, the viewer can understand the degree of congestion. When the statistical processing department of one example displays a small image of the number of people on the terminal for the store so that the manager of the store is the viewer, it is crowded by allocating the staff of the store to the crowded place, guiding customers, etc. Can be dealt with appropriately. In addition, the statistical processing unit of the system in the example may provide information on the congested area to the visitor, for example, in the area around the entrance of the store, so that the visitor of the store is the viewer. There are advantages that can be done. Further, if the place displayed in real time is on the WEB, there is an advantage that the user can understand the congested place on the WEB using the mobile terminal.

Further, the statistical processing unit of one example may generate, as statistical information, a relationship between a position related to some information, a staying time at the position, and a product such as a product shelf in a store. As a result, it is possible to collect information on how long the user related to the partial information is browsing the product. Further, the statistical processing unit of one example may generate the degree of interest in the product by using the staying time as the statistical information. This has the advantage that the degree of interest of the user in the product can be generated.

Further, the statistical processing unit of one example may generate various rankings as information related to boundaries. Such a ranking may be a ranking of stores in the facility based on the number of visitors of a person as a target for some information. For example, store A may be ranked first, store B may be ranked second, and so on. In this case, there is an advantage that information regarding the number of visitors to the store can be generated.

Further, the statistical processing unit of one example may generate a ranking in the order of most or few stores visited before or after a specific store as a ranking. In this case, there is an advantage that information on the relationship between the user's stores can be generated depending on what kind of store the user visits before or after the specific store.

Further, the statistical processing unit of one example may generate a ranking of the stores visited by the user at the i-th (i> = 1) in the facility as a ranking. In this case, there is an advantage that the user can generate information on which store is visited in a specific order. In particular, when the statistical processing unit of one example generates the ranking of the first visited store in the facility, the information of the first visited store in the facility can be generated. Further, when the statistical processing unit of one example generates the ranking of the last visited store in the facility, there is an advantage that the information of the last visited store in the facility can be generated.

Further, the statistical processing unit of one example may generate a ranking of the number of i-stores when the user visits only the number of i-stores (a positive number of i> = 1) as a ranking. In this case, there is an advantage that a ranking can be generated in a limited number of stores, which is the number of i stores. In particular, for example, when the statistical processing unit of one example generates a ranking of stores when only one store is visited by the user, the user visits only one store, which is a clearly intended destination. It has the advantage of being able to generate rankings for stores.

Further, the statistical processing unit of one example may generate a ranking regarding the places visited before and after a specific store as a ranking. The ranking of the visited places may be the ranking of the stores and places visited before the specific store, or the ranking of the stores and places visited after the specific store. In this case, there is an advantage that it is possible to understand what kind of place is visited before and after a specific store.

Further, the statistical processing unit of one example may be a ranking of stores and places visited before a predetermined number of specific stores, or a ranking of stores and places visited after a predetermined number of specific stores. You can. For example, it may be the store visited first of the specific store, or it may be the store visited secondarily of the specific store. Similarly, for example, it may be a store visited one before a specific store, or a store visited two times before a specific store. This has the advantage of being able to understand the stores and locations visited before and after a predetermined number of specific stores, and in some cases, the route from such stores and locations to such specific stores. If the user can understand such a route, there is an advantage that the user can carry out marketing activities such as easy advertisement for a specific store and distribution of leaflets.

An example system is
A partial information extraction unit that extracts partial information from images captured by imaging devices installed inside and outside the facility,
An information generation unit that uses the partial information to generate information related to the target related to the partial information.
May be equipped. Since such a system generates information using a part of information related to a part of the image, there is an advantage that the amount of calculation is small and the information can be generated. In particular, when some information is related to a moving part in a moving image including an image, there is a high possibility that some information is related to a moving target, so information related to the target can be efficiently used by using the learned information generation function. Has the advantage of being able to generate. For example, when some images as partial information mainly focus on people, the proportion of people in the whole image is high, so it is possible to identify people, gender, age, age group, and height. It has the advantage of being able to generate information that indicates a person from a specific point of view, such as physique, facial expression, and information about fashion items.

An example system is
A partial information extraction unit that extracts multiple partial information,
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

An example system is
A partial information extraction unit that extracts multiple partial information,
A partial information relationship generation unit that generates a relationship using the plurality of partial information,
A statistical processing unit that generates statistical information using some of the above information,
May be equipped. Since such a system generates statistical information based on the relationship related to some information, there is an advantage that statistical information can be generated while easily generating the movement of the target related to some information.

The various functions described above and other application examples of an example system will be described below. In the following, only modifications in each specific example will be mainly described in order to avoid repeated explanations, but it goes without saying that the above-mentioned techniques may be applied to the application examples in the following.

2.2. Example of worker's flow line The statistical processing unit of one example uses the movement of the position related to some information based on the image acquired from the image pickup device installed in the factory to move the worker in the factory. Such information may be generated. For example, the statistical processing unit of one example may generate the movement of the worker indicated by the position related to some information. Such movement has an advantage that information indicating what kind of movement the worker is doing can be generated. In addition, such movements may be displayed. This has the advantage that the viewer can check, for example, whether or not the movement of the worker is wasteful. The statistical processing unit of one example may generate and display the movement of the worker indicated by the position related to some information as an average value for a predetermined period. In this case, there is an advantage that information on the movement of the worker can be generated as an average value for a predetermined period. In addition, the system of one example may have a tracking unit and track the movement of some information across a plurality of imaging devices.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to the worker based on the image captured by the image pickup device installed in the factory.
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to the worker based on the image captured by the image pickup device installed in the factory.
A partial information relationship generation unit that generates a relationship using the plurality of partial information,
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

2.3. The system of one example of a traffic vehicle is based on an image taken by an image pickup device such as a traffic camera installed in a place where a vehicle can be located, such as an intersection, a tollhouse on a highway, a parking lot, or a side of a road. Part information may be used.

The statistical processing unit related to the system of one example may generate the statistical information of the vehicle related to the partial information based on the partial information. In particular, traffic cameras operate in an external environment and generally do not operate under favorable conditions compared to indoors. Therefore, in image analysis, a computational load is applied when a trained model is applied. It may increase. Therefore, the system of one example has an advantage that statistical information about a vehicle or the like can be generated by using the above-mentioned partial information relation generation unit when the trained model is not applied or the number of times of application thereof is reduced.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to the vehicle based on an image captured by an image pickup device installed in a place where the vehicle can be located.
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to the vehicle based on an image captured by an image pickup device installed in a place where the vehicle can be located.
A partial information relationship generation unit that generates a relationship using the plurality of partial information,
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

2.4. The system of the construction site / mine example example may use some information based on the image captured by the image pickup device installed at the construction site or the mine. In addition, some information may be vehicles used at construction sites and mines. Image pickup devices installed at construction sites and mines generally do not operate under favorable conditions compared to image pickup devices in facilities, so the computational load is calculated when applying a trained model in image analysis. May increase. Therefore, in the case of not applying the trained model or reducing the number of times the trained model is applied, the system of one example uses the above-mentioned partial information relation generation unit and / or target number estimation unit to obtain statistical information about the vehicle or the like. There is an advantage that it can be generated. Further, when the system of one example generates the movement of the vehicle, there is an advantage that reference information can be generated in order to move the vehicle efficiently.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to the vehicle based on an image captured by an imaging device installed at a position that includes a part or all of the vehicle movement area in the field of view.
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to the vehicle based on an image captured by an imaging device installed at a position that includes a part or all of the vehicle movement area in the field of view.
A partial information relationship generation unit that generates a relationship using the plurality of partial information,
A statistical processing unit that generates statistical information using some of the above information,
May be equipped. Since such a system generates statistical information based on the relationship related to some information, there is an advantage that statistical information can be generated while easily generating the movement of the target related to some information.

2.5. Animal Target Example The system of one example may image an animal with an imaging device. The animals may be, for example, cows, pigs, sheep, birds, horses, sheep, goats, reindeer, and the like. The animal may be livestock. In particular, when these animals live in a wide area such as living outdoors, the imaging device will be installed outdoors. In this case, a wide area is imaged by a few imaging devices, and it generally does not operate under favorable conditions compared to the imaging devices in the facility. Therefore, in image analysis, the calculation load is applied when the trained model is applied. May increase. Therefore, when the trained model is not applied or the number of times the trained model is applied is reduced, the system of one example reduces the burden of calculation amount by using the above-mentioned partial information relation generation unit and / or target number estimation unit. However, it has the advantage of being able to generate statistical information about animals.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to animals based on an image captured by an image pickup device installed outdoors.
A statistical processing unit that generates statistical information using some of the above information,
May be equipped.

An example system is
A partial information extraction unit that extracts a plurality of partial information related to animals based on an image captured by an image pickup device installed outdoors.
A partial information relationship generation unit that generates a relationship using the plurality of partial information,
A statistical processing unit that generates statistical information using some of the above information,
May be equipped. Since such a system generates statistical information based on the relationship related to some information, there is an advantage that statistical information can be generated while easily generating the movement of the target related to some information.

2.7. Drone usage example The imaging device may be installed in the drone. When the drone is equipped with an image pickup device, there is an advantage that the image can be taken from a position different from that of the image pickup device installed in the facility. For example, even if there is no place to install the image pickup device, the image pickup device provided in the drone has an advantage that the image pickup place can be set more freely by changing the flight place of the drone.

In addition, when a drone equipped with an image pickup device flies at a position higher than the height of the facility, it is possible to take an image from a higher altitude than the image pickup device provided in the facility, so that an image with a wider field of view is possible. There is an advantage of becoming.

Further, when the hovering is controlled so that the drone equipped with the image pickup device flies at a substantially fixed position, the images captured by the drone are captured from substantially the same location, so that some information extraction units and The partial information relation generation unit has an advantage that information on some information can be appropriately generated. Further, even in the case of moving at a low speed, the same processing as in hovering can be applied by using the image after subtracting the expected movement amount. As for the prediction of movement, preset movement information, actual movement information acquired by a gyro, GPS, or the like, or movement information estimated from the correspondence of feature points of an image may be used.

The target of the image captured by the drone equipped with the image pickup device may be the various targets described in 2.1 to 2.6 described above, or may be other than these.

An example system is
A partial information extraction unit that extracts one or more partial information based on the image captured by the image pickup device provided in the drone.
A statistical processing unit that generates statistical information using the one or more partial information.
May be equipped.

An example system is
A partial information extraction unit that extracts one or more partial information based on the image captured by the image pickup device provided in the drone.
A partial information relationship generation unit that generates a relationship using the one or more partial information,
A statistical processing unit that generates statistical information using the one or more partial information.
May be equipped. Since such a system generates statistical information based on the relationship related to some information, there is an advantage that statistical information can be generated while easily generating the movement of the target related to some information.

2.7. Notification system example The statistical processing unit of the system of one example may have a notification unit that notifies the terminal of the system of one example when an object related to some information invades a predetermined area. The notification unit has an advantage that the user of an example system can understand that the target related to some information has invaded a predetermined area.

For example, specific areas include places where entry is prohibited, dangerous places, places where it is not desirable to enter, and the like. If a specific area is inside or outside the facility, it may be, for example, a factory site or some facility (for example, a high electric wire facility, a substation facility, a water supply facility, a hospital). If a specific area is in the house, the specific area includes, for example, a place where there is a source of fire such as a kitchen, or a water place such as a bathroom, but is not limited to these.

In addition, examples of the target related to some information include minors, elderly people, dementia patients, suspicious persons, and the like. In this case, the users related to the system include system administrators, parents of minors, relatives of the elderly, guardians of dementia patients, etc., and the terminals related to the system are the terminals used by such persons. You can.

When the statistical processing unit of one example notifies the terminal of the above notification at the timing of intrusion, the user of the system of one example can understand the intrusion in real time. In this case, there is an advantage that the user related to the notified terminal can quickly respond according to the mode of intrusion such as the intruder, the intrusion time, and the intrusion location.

The notification unit may be provided by the system in each of the above-mentioned 2.1 to 2.6 examples. In that case, the specific place may be set as a prohibited place, a dangerous place, a place where it is not preferable to enter, etc. according to each application example, and the target related to some information is a person, It may be a vehicle, livestock, etc.

It goes without saying that the invention examples described in the examples of the documents of the present application are not limited to those described in the documents of the present application, and can be applied to various examples within the scope of the technical idea. Further, the external information processing device may be a cloud or a server that uses software such as SAAS, PAAS, or IAAS.

Further, the processes and procedures described in the documents of the present application may be feasible not only by those explicitly described in the embodiments but also by software, hardware or a combination thereof. Further, the processes and procedures described in the documents of the present application may be able to be implemented by various computers by implementing the processes and procedures as a computer program. Further, these computer programs may be stored in a storage medium. Also, these programs may be stored on a non-transient or temporary storage medium.

1401a to 1401c Image pickup device 1410 Information processing device 1411 Computing device 1412 Storage device 1413 Input device 1414 Display device 1415 Bus 1416 Communication device 1417 Network 1418 AI device

Claims

An extraction unit that extracts a part of information in the target image related to a moving part in a moving image including the target image from the target image.
A generation unit that generates target information related to the partial information based on the extracted partial information.
System with.
An extraction unit that extracts some information from the target image,
A generation unit that generates target information related to the partial information based on the extracted image related to the partial information.
With
In the image related to the partial information, the amount of image information of a part of the background region is smaller than the amount of image information of the portion corresponding to the part in the target image.
system.
The extraction unit extracts a part of the information by using the movement of the target in the moving image.
The system according to claim 1.
The extraction unit extracts a part of the information by using the difference between one image and the other image among the plurality of images constituting the moving image.
The system according to claim 1.
The generation unit generates information on the target related to the image related to the partial information by applying the image related to the partial information to the machine learning unit in which the relationship between the image and the target is machine-learned.
The system according to any one of claims 1 to 4.
The generator
A transmitter that transmits an image related to the partial information,
An acquisition unit that acquires target information corresponding to an image related to the transmitted partial information, and an acquisition unit.
The system according to any one of claims 1 to 5.
The generation unit includes a machine learning unit that machine-learns the relationship between an image and an object.
By applying the image related to the partial information to the machine learning, the target information related to the image related to the partial information is generated.
The system according to any one of claims 1 to 6.
The partial information is a partial image of the target image.
The system according to any one of claims 1 to 7.
The partial information is information relating to feature points of a partial image of the target image.
The system according to any one of claims 1 to 8.
The partial information is information related to background subtraction of the target image.
The system according to any one of claims 1 to 9.
The partial information is an image obtained by binarizing or binarizing the target image.
The system according to any one of claims 1 to 10.
The system
An estimation unit that estimates the number of targets based on predetermined rules for the cluster related to the partial information.
To prepare
The system according to any one of claims 1 to 11.
The system further includes a statistical processing unit that generates statistical information.
The system according to any one of claims 1 to 12.
The computer
A step of extracting a part of a moving part of a moving image including the target image from the target image, and
A step of generating target information related to the part of the image based on the extracted part of the image, and
How to do.
A program for operating a computer as the system according to any one of claims 1 to 13.