JP2021039642A - Object identification method for determining object identification position using encoding parameter, device and program - Google Patents

Abstract

To provide an object identification method capable of reducing a processing load of object identification processing on a side where image data to be processed is received.SOLUTION: An object identification method that is an object identification method in a computer for identifying a predetermined object from image data including the object in an image includes the steps of; determining at least one candidate image region related to a predetermined object in the image data on the basis of information on a position in the image of a unit image region where an encoding parameter determined when video data including the image data is encoded satisfies the predetermined condition; and defining at least one determined candidate image region as input, using a learned identifier for outputting information on a class of the predetermined object, and identifying the predetermined object from the image data. Encoding of the video data including the image data may be encoding by MPEG, and the unit image region can be a macro block.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for identifying a predetermined object from image data that can include the object in the image.

Currently, for the purpose of monitoring and marketing, and also as a "visual system" for autonomous vehicles and autonomous robots, there is active development of technology that analyzes image data generated by cameras and identifies captured objects. It is being advanced.

Here, in many cases, this object identification process is performed using an object detector that has been trained for object detection. As an example of using such an object detector, Non-Patent Document 1 discloses a technique for estimating the existence position and its type in an object included in an image by using a convolutional neural network (CNN). ing.

Further, Patent Document 1 discloses an object detection device that uses CNN to speed up the detection process of a specific object. Specifically, this device is equipped with one or more layers of neural networks that perform convolution calculations, and has a feature extraction unit that outputs a feature map, and a sliding window process that executes sliding window processing on this feature map to output multiple feature map windows. It includes a sliding window unit and an identification unit that determines whether or not a specific object is included in these feature map windows.

Further, in this object detection device, for example, when object detection is performed on an image having an image size of 1280 × 448, this image is used as a plurality of images having different image quality, for example, 320 × 112, 640 × 224, and 1280 × 448. The images are converted to the above images, and object detection is performed using these images.

Japanese Unexamined Patent Publication No. 2018-005520

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, “SSD: single shot multibox detector”, European Conference on Computer Vision, Computer Vision-ECCV 2016, 2016, 21 ~ Page 37

In the situation where the image identification technology as described above is implemented, in many cases, the captured and generated image data is an image identification device in the form of a video stream compressed and encoded from an in-vehicle device, a surveillance camera device, or the like. Will be sent to.

Here, in the prior art as described in Patent Document 1 and Non-Patent Document 1, in order to perform the estimation processing of the image area related to the target, which is the premise of the target identification processing, this stream data is used as the image data for the time being. It becomes necessary to convert it into a form and input it to the machine learning model.

Therefore, the processing load on the target identification device that receives the stream data becomes heavy, and this large processing load has become a serious problem especially in a situation where real-time performance of the target identification processing is required.

Therefore, an object of the present invention is to provide an object identification method, an apparatus, and a program capable of reducing the processing load of the object identification process on the side receiving the image data to be processed.

According to the present invention, it is an object identification method in a computer that identifies an object from image data that can include a predetermined object in the image.
Based on the information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies the predetermined condition, in the image data, the target is A step of determining at least one candidate image region, and
A target identification method having a step of identifying the target from the image data by using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class. Provided.

In the step of determining the candidate image area of the target identification method according to the present invention, as the candidate image area, the first candidate image area including the position of the unit image area in the image and the nth (n is 2). N th candidate image regions (each integer from to N (≧ 2)) including the entire or nth candidate image region including a predetermined or more portion of the (n−1) candidate image region. It is also preferable to determine the candidate image region.

Further, specifically, as one embodiment, in the nth candidate image area, the (n-1) th candidate image area is arranged in the center of the nth candidate image area, and the entire (n-1) candidate image area is covered. It is also preferred that it be determined to include. As yet another embodiment, it is also preferable that the position of the first candidate image area in the image of the unit image area is determined to be the lower end or the lower end of the unit image area.

Further, in the above-described embodiment, the first candidate image area may be determined to have a larger area as the distance from the lower end of the image at the position in the image of the unit image area is smaller. preferable. Further, it is also preferable that the first candidate image region is determined to have a larger area as the distance from the vanishing point at the position in the image of the unit image area increases.

Further, in the object identification method according to the present invention, specifically, the coding of the video data is coding by MPEG (Moving Picture Experts Group), and the coding parameter satisfying a predetermined condition is
(A) A motion vector for forward prediction having a size equal to or larger than a predetermined value and a direction having an angle equal to or larger than a predetermined direction with respect to a reference direction.
The unit image area is a macroblock, which is at least one of (b) an in-screen prediction code amount having a size equal to or larger than a predetermined value and (c) a quantization step size having a size equal to or smaller than a predetermined size. It is also preferable to have.

Further, the classifier in the object identification method according to the present invention is
At least one convolutional layer unit that takes each of the at least one candidate image area as an input and outputs the feature information related to the feature of the candidate image area, respectively.
It is also preferable that the feature information output from at least one convolution layer portion is collected and input, and the fully-connected layer portion (Fully-Connected Layers) that outputs the information related to the target class is included.

According to the present invention, it is also a device that identifies a predetermined object from image data that can include the predetermined object in the image.
Based on the information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies the predetermined condition, in the image data, the target is A candidate area determining means for determining at least one candidate image area, and
Target identification having a target identification means for identifying the target from the image data by using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class. Equipment is provided.

According to the present invention, it is a server generated by a client that acquires image data that can include a predetermined object in an image and identifies the object.
Information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies a predetermined condition, and is acquired from the client together with the image data. A candidate area determining means for determining at least one candidate image area related to the target in the image data based on the information related to the position, and
Target identification having a target identification means for identifying the target from the image data by using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class. A server is provided.

According to the present invention, further, it is a program that functions a computer that identifies a predetermined object from image data that can include the object in the image.
Based on the information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies the predetermined condition, in the image data, the target is A candidate area determining means for determining at least one candidate image area, and
Using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class, the computer functions as a target identification means for identifying the target from the image data. A target identification program is provided.

According to the object identification method, the apparatus, and the program of the present invention, it is possible to reduce the processing load of the object identification process on the side receiving the image data to be processed.

It is a schematic diagram and the functional block diagram for demonstrating one Embodiment of the object identification system provided with the object identification apparatus (server) and the client by this invention. It is a schematic diagram for demonstrating one Example of the candidate image area determination process in the candidate area determination part which concerns on this invention. It is a schematic diagram for demonstrating various embodiments about the candidate image area determination process in the candidate area determination part which concerns on this invention. It is a schematic diagram for demonstrating one Example of the object identification process in the object identification part which concerns on this invention. It is a schematic diagram for demonstrating the Example of the macroblock sorting process in the macroblock sorting section which concerns on this invention. It is a schematic diagram for demonstrating one Embodiment which concerns on the generation of the frame which constitutes the significant video stream in the significant video stream generation part which concerns on this invention. It is a schematic diagram for demonstrating one Embodiment of the significant video stream generation processing in the significant video stream generation part which concerns on this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Target identification system]
FIG. 1 is a schematic diagram and a functional block diagram for explaining an embodiment of an object identification system including an object identification device (server) and a client according to the present invention.

The target identification system of the present embodiment shown in FIG. 1 is
(A) At least one terminal 20 that is a mobile client, and
(B) Since the terminal 20 has a cloud server 1 which is a server capable of acquiring image data capable of including a predetermined target in the image, and the cloud server 1 identifies the predetermined target from the image data. is there.

Further, the terminal 20 of the above (a) is a drive recorder having a communication function in the present embodiment and is installed in the automobile 2. Here, the installation location can be arbitrarily set, for example, a position where the front of the vehicle can be photographed through the front glass of the automobile 2 (for example, the upper part of the dashboard). Of course, the terminal 20 may be installed at a position where the side or the rear of the vehicle can be photographed, and the terminal 20 may be installed at each of a plurality of different positions.

Further, in the present embodiment, the terminal 20 (drive recorder) may, for example, capture the situation of the traveling direction of the automobile 2 with a camera, generate image (video) data, and store the image (video) data in a memory or storage provided in the terminal 20 (drive recorder). it can. Further, the terminal 20 can be wirelessly connected to the cloud server 1 via, for example, a mobile phone communication network or the Internet, and a part or all of the stored image (video) data can be appropriately or requested. It can also be sent to the cloud server 1.

Here, when transmitting image (video) data from the terminal 20 to the cloud server 1, the terminal 20 usually performs compression coding processing on the image (video) data to generate a compressed video stream with a small transmission load. Will be sent. In the present embodiment, as this compression coding process, H.M. 264 and H. Processing is performed in a standard format such as 265, but it is also possible to use a non-standard format. Of course, the spatial resolution, frame rate, and bit rate can be set arbitrarily.

Further, as will be described in detail later with reference to FIGS. 5 to 7, the terminal 20 provides a “significant video stream” (FIG. 7) composed of coded frames synchronized with the “coded parameter map” frame described later. It is also preferable to transmit to the cloud server 1.

Here, the "significant video stream" is composed of frames that will be required for the target identification process in the cloud server 1 (which will be significant in the process), and is actually photographed by the camera 202. Frames that capture objects that suddenly appear or disappear in the shooting range, or whose position or shape changes at a rapid rate equal to or greater than a predetermined value, are selected and included in the surrounding conditions of the automobile 2. .. As a result, the "significant video stream" is video data having a smaller transmission capacity than the encoded original video stream.

On the other hand, the cloud server 1 of the above (b) has a specific configuration thereof.
(A) Information related to the position in the image of the "unit image area" in which the "coding parameter" determined when the video data including the image data to be identified is encoded satisfies a predetermined condition. In the image data, the candidate area determination unit 112 that determines at least one "candidate image area" related to a predetermined target, and
(B) Target identification for identifying a predetermined target from the image data by using a learned classifier that inputs at least one determined "candidate image area" and outputs information related to a predetermined target class. It is characterized by having a portion 113.

Here, the image data for determining the "candidate image area" in the above (A) can be decompressed (decoded) the received compressed encoded image (video) data, or a "significant video stream". It may be the image data constituting the above.

Further, the "coding parameter" of (A) above can be at least one of the motion vector of forward prediction, the code amount of in-screen prediction, and the quantization step size in MPEG in the present embodiment. Furthermore, the "unit image area" in which the "coding parameter" satisfies the predetermined condition is
(A) The motion vector of forward prediction has a size equal to or larger than a predetermined value and a direction forming an angle equal to or larger than a predetermined direction with respect to a reference direction (for example, a direction toward a vanishing point (convergence point on the horizon in the image)). ing,
One, two, or one of three conditions that (b) the code amount of the in-screen prediction has a magnitude equal to or greater than a predetermined value, and (c) the quantization step size has a magnitude equal to or less than a predetermined value. It is also preferable that it is a "macroblock" that is completely filled.

The information of the "macro block (unit image area)" satisfying such a condition may be acquired by the "encoding parameter map" transmitted from the terminal 20 together with the "significant video stream". The “coded parameter map” will be described in detail later, but the higher the degree to which the motion vector of forward prediction, the code amount of in-screen prediction, and / or the quantization step size satisfy a predetermined condition, the more the corresponding macroblock It is the map data in which the pixel value of the part of is larger.

For example, in the "encoded parameter map", the larger the size of the motion vector of the forward prediction and the larger the angle formed by the direction with respect to the reference direction, the higher the gradation of the hue in the macroblock part. It may be (darkened). Further, the larger the code amount of the in-screen prediction, the higher (darker) the gradation of the hue in the macroblock portion can be made. Further, the smaller the quantization step size, the higher (darker) the gradation of the hue in the macroblock portion may be.

In any case, by using such a "coded parameter map", information relating to the position in the image of the "unit image area" in which the "coded parameter" satisfies a predetermined condition is acquired / determined. However, this makes it possible to determine the "candidate image area".

By the way, the terminal 20 notifies the cloud server 1 of the position information (for example, the address of the macro block and the position coordinate value in the image) in the image of the "macro block" satisfying a predetermined condition instead of the "encoding parameter map". The cloud server 1 can also determine the "candidate image area" based on this position information.

Further, the "encoded parameter map" and the above macroblock position information may be generated in the cloud server 1 instead of being acquired from the terminal 20. That is, the cloud server 1 simply decompresses (decodes) the received compressed coded video stream, interprets (parses) the bit stream, and extracts the coded parameters, thereby forming a "coded parameter map" or the like. The above macroblock position information may be generated.

In any case, the cloud server 1 sets at least one "candidate image area" based on the information relating to the position in the image of the "unit image area" in which the "coding parameter" satisfies a predetermined condition. Can be decided. That is, when carrying out the target identification process, it is not necessary to first perform a process of detecting a predetermined target in the image using a detector and then perform a heavy-duty process of determining the candidate position in advance. ..

In other words, it can be considered that the processing result corresponding to the target detection processing is generated at the stage of compressing and coding the transmitted video data on the terminal 20 side which is the client. As a result, it is possible to reduce the processing load of the target identification process on the side receiving the image data to be processed (that is, the cloud server 1). Further, depending on the embodiment, the cloud server 1 does not need to accumulate the data originally required for executing the target detection process, and as a result, the amount of data accumulated on the server side can be reduced. is there.

Naturally, the terminal 20 is not limited to the in-vehicle device (drive recorder) installed in the automobile 2, and is installed in, for example, a bicycle, a railroad vehicle, or another moving body such as a robot or a drone. It may be a boarded device. Further, the terminal 20 may be a wearable terminal such as an HMD (Head Mounted Display) or a glass type terminal. In this case, for example, the image data generated by the user walking while walking is uploaded to the cloud server 1. Further, the client of the image data generation / transmission source may be a non-movable terminal unlike the terminal 20, and may be, for example, a fixed camera device having a communication function.

[Functional configuration of target identification server]
According to the functional block diagram shown in FIG. 1, the cloud server 1 has a communication interface 101 and a processor memory. Here, this processor memory stores one embodiment of the target identification program according to the present invention, has a computer function, and executes the target identification process by executing the target identification program. To do.

From this, as the target identification server according to the present invention, for example, a non-cloud server device, a personal computer (PC), a notebook type or a tablet type, which is equipped with a target identification program according to the present invention instead of the cloud server 1. It is also possible to adopt a computer, a smartphone, or the like.

Further, for example, the target identification program according to the present invention can be mounted on the terminal 20, and the terminal 20 can be used as the target identification server according to the present invention. Further, an embodiment in which the target identification server according to the present invention is installed in the automobile 2 together with the terminal 20 is also possible.

The processor memory also includes an image acquisition unit 111, a candidate area determination unit 112, an object identification unit 113, a learning data generation unit 114, a learning model generation unit 115, and a transmission / reception control unit 116. Note that these functional components can be regarded as the functions of the target identification program stored in the processor memory. Further, the processing flow shown by connecting the functional components of the cloud server 1 in FIG. 1 with arrows is also understood as an embodiment of the target identification method according to the present invention.

Similarly, in the functional block diagram of FIG. 1, the image acquisition unit 111 acquires and manages a compressed coded video stream received from the terminal 20 via the communication interface 101 and the transmission / reception control unit 116. Here, it is also preferable that the image acquisition unit 111 expands (decodes) the video stream and manages it as an image data group constituting the original video stream. Further, as described above, as one modification, it is also possible to generate a "coded parameter map" by interpreting (parse) the bit stream and extracting the coding parameters.

The candidate area determination unit 112 obtains at least one candidate image area related to a predetermined target in the image data captured from the image acquisition unit 111 based on the “macroblock position information” received from the terminal 20 together with the video stream. decide. Here, "macroblock position information" is
(A) Position information in the image of the macroblock in which the coding parameter satisfies a predetermined condition, for example, the macroblock address, the position coordinate value in the image of the macroblock, or (b) the coding parameter map. it can.

Specifically, the "macroblock position information" in the present embodiment is described as described above.
(A) The motion vector of the forward prediction has a size equal to or larger than a predetermined value and a direction forming an angle equal to or larger than a predetermined direction with respect to the reference direction toward the vanishing point.
One, two, or one of three conditions that (b) the code amount of the in-screen prediction has a magnitude equal to or greater than a predetermined value, and (c) the quantization step size has a magnitude equal to or less than a predetermined value. It is the position information in the image of the macro block that satisfies all (which one is set in advance).

The candidate area determination unit 112 determines the candidate image area to be used for the target identification process with reference to the position in the image designated by such “macroblock position information”.

FIG. 2 is a schematic diagram for explaining an embodiment of the candidate image area determination process in the candidate area determination unit 112.

According to FIG. 2, the candidate region determination unit 112 acquires the image data to be identified and the coded parameter map corresponding to the image data, and from the coded parameter map, the macro satisfying the above-mentioned predetermined conditions. The position coordinates in the image of the block are extracted to determine the reference target position (black circle in FIG. 2) which is a reference when determining the candidate image area.

Here, when macroblocks satisfying a predetermined condition are distributed as a plurality of blocks, for example, the coordinates of the center of gravity of each block can be determined at the reference target position. In this embodiment as well, a plurality of reference target positions are actually determined, but FIG. 2 shows one of them.

Next, the candidate region determination unit 112 determines three candidate image regions based on the determined reference target position in this embodiment. Specifically, as shown in FIG.
(A) A first candidate image area including the reference target position (position in the image of the macro block satisfying a predetermined condition) and
(B) A second candidate image area including the entire first candidate image area (or a portion equal to or larger than a predetermined value) and
(C) The third candidate image area including the entire second candidate image area (or a portion equal to or larger than a predetermined value) is determined.

Here, of course, the candidate area determination unit 112 is not limited to the three candidate image areas, and can determine N (integer of 2 or more) candidate image areas set in advance. In this case, the first candidate image area including the reference target position (the position in the image of the macroblock satisfying a predetermined condition) and the nth (n is each integer from 2 to N) candidate image area. It is also preferable to determine N candidate image regions including the entire or nth candidate image region including a predetermined or more portion of the candidate image region of (n−1).

FIG. 3 is a schematic diagram for explaining various embodiments of the candidate image region determination process in the candidate region determination unit 112.

First, as shown in FIG. 3A, it is assumed that the candidate region determination unit 112 determines one reference target position in the image data to be identified. Here, when a car is included in the image as shown in the figure, the reference target position is often at the feet in contact with the flat ground (road surface) of the car, that is, near the tires.

Then, as one embodiment, the candidate region determination unit 112, as shown in FIG. 3 (B),
(A) The first candidate image area including the determined reference target position and
(B) A second candidate image area that arranges the first candidate image area in the center of itself and includes the entire first candidate image area, and a second candidate image area.
(C) The second candidate image area can be arranged in the center of the second candidate image area, and a third candidate image area including the entire second candidate image area can be determined.

Here, of course, the candidate area determination unit 112 is not limited to the three candidate image areas, and may determine N (integer of 2 or more) candidate image areas set in advance. In this case, the nth (n is each integer from 2 to N) candidate image region has the (n−1) th candidate image region in its center and the (n−1) th candidate image. It is determined to cover the entire area.

By the processing as described above, the reference target position where a predetermined target may exist can be surely included, and the entire target can be included (possibly included) (at least in the Nth candidate image region which is the maximum). It is possible to determine a plurality of candidate image regions (highly likely). Further, after that, by using such a candidate image area, it becomes possible to more reliably identify the target.

Further, as another embodiment, the candidate area determination unit 112 sets the first candidate image area so that the reference target position is the lower part or the lower end of the first candidate image area, as shown in FIG. 3C. It is also preferable to determine. In this case, as also shown in FIG. 3 (C),
(A) The reference target position is its own lower part or lower end, and the reference target position is its own lower part or lower end as well as the second candidate image area including the entire first candidate image area. It may be determined that the third candidate image area includes the entire second candidate image area, or
(B) The first candidate image area is arranged in the center of the user, and the second candidate image area including the entire first candidate image area and the second candidate image area are arranged in the center of the user. Moreover, it is also possible to determine a third candidate image area that includes the entire second candidate image area.

Here, of course, the candidate area determination unit 112 is not limited to the three candidate image areas, and may determine N (integer of 2 or more) candidate image areas set in advance in the same manner as described above.

In any case, by the processing as described above, the reference target position where the predetermined target may exist is surely included, and the entire target is covered (at least in the Nth candidate image region which is the maximum). It is possible to determine a plurality of candidate image regions that can be included (and are likely to be included). In particular, as described above, when a moving body such as an automobile is a predetermined target, the reference target position is usually the foot in contact with the flat ground (road surface) of the moving body. Therefore, in this case, the target is more likely to be included in the candidate image area developed above the reference target position. Further, after that, by using such a candidate image area, it becomes possible to more reliably identify the target.

Further, in any of the embodiments shown in FIGS. 3 (B) and 3 (C), the first candidate image area is (a) the lower end of the image at the reference target position (the position in the image of the macro block satisfying a predetermined condition). It is also preferable that the smaller the distance a from is, the larger the area is determined. In this case, the area (number of pixels) S1 of the first candidate image area is calculated by the following equation (1) S1 = f _s (a) _{, where f s is a monotonous decrease function of the distance a.}
Can be represented by.

Similarly, in any of the embodiments of FIGS. 3 (B) and 3 (C), the first candidate image area is (b) a vanishing point (a position in the image of a macroblock satisfying a predetermined condition) at a reference target position (b). It is also preferable that the larger the distance from FIG. 3 (A)), the larger the area is determined.

By determining the area of the first candidate image area as described above, for example, when the vehicle is identified as a predetermined target from the image data obtained by photographing the vehicle traveling in front, the position of the vehicle is determined. The closer is, the larger the area of the first candidate image area, and thus the area of the remaining candidate image area, can be set, and as a result, the candidate that can (and is likely to include) the entire vehicle is included. The image area can be easily determined.

On the other hand, the farther the position of the vehicle is, the smaller the area of the first candidate image area and, by extension, the area of the remaining candidate image area can be set. It becomes easy to determine a candidate image area that does not include the image area of the above as much as possible.

As a more preferred embodiment, the Nth candidate image region also has a larger area as the distance a from the lower end of the image at (a) the reference target position (the position in the image of the macroblock satisfying a predetermined condition) is smaller. Further, the larger the distance from the vanishing point (FIG. 3 (A)) at the reference target position (the position in the image of the macroblock satisfying a predetermined condition), the larger the area. It is also preferred to be determined to have.

In any case, by setting the area in this way, the size of the Nth candidate image area, which becomes the largest, matches the assumed size of the target (for example, an automobile) related to the reference target position, and the target is concerned. It becomes possible to adjust so that the image area according to the above can be included more reliably.

By the way, when there are m candidate image regions between the first and Nth candidate image regions, the areas of these candidate image regions are the area value of the first candidate image region and the Nth candidate. It can be set to the area value of each equally divided position when the area value of the image area is divided into (m + 1) equal parts.

Returning to the functional block diagram of FIG. 1, the target identification unit 113 uses a trained classifier that inputs at least one determined candidate image area and outputs information related to a predetermined target class to obtain an image. Identify the target from the data.

Here, the classifier that performs the target identification process includes a deep neural network (DNN, Deep Neural Network) widely used for image recognition, an SVM (Support Vector machine), a random forest (Random Forest), and the like. , At least one candidate image area can be input and the identification result can be output by various types of machine learning algorithms. Hereinafter, an example of a classifier capable of accepting a plurality of candidate image areas and performing target identification will be described.

FIG. 4 is a schematic diagram for explaining an embodiment of the target identification process in the target identification unit 113.

In the embodiment shown in FIG. 4, the target identification unit 113 uses the three candidate image areas determined in the embodiment of FIG. 2 as inputs, and inputs the "automobile" as a predetermined target class and its certainty (score). ) And is output, and the identification process is performed.

Here, this classifier is
(A) A plurality of convolutional layers that input each of a plurality of (three in this embodiment) candidate image regions and output feature information related to the features of the candidate image regions.
(B) Information related to the class of a predetermined target (automobile in this embodiment) (for example, the class "automobile" and its score) by collecting and inputting the feature information output from the plurality of convolution layer portions of the above (a). It is configured to include Fully-Connected Layers that output.

Here, the convolution layer portion of the above (a) executes a convolution process for generating a feature map by sliding the kernel (weighting matrix filter) with respect to the image data. By this convolution process, it is possible to extract basic features such as edges and gradients and obtain information on local correlation patterns while gradually reducing the resolution of the image. For example, as the convolution layer portion, it is possible to use a known AlexNet using a plurality of convolution layers.

In this AlexNet, each convolution layer is paired with a pooling layer, and the convolution process and the pooling process are repeated. Here, the pooling process is to summarize the feature map (reaction of the convolution filter within a certain area) output from the convolution layer by the maximum value, average value, etc., reduce the adjustment parameters, and secure the local translation invariance. It is a process to do.

As yet another embodiment, the target identification unit 113 has a configuration in which a support vector machine (SVM) provided for each class to be discriminated is connected to the output side of the plurality of convolution layer units of the above (a). It is also possible to use the discriminator of the above and to perform the target identification process after learning from the discriminator.

In any case, the target identification result (information related to the target class, for example, the class "automobile" and its score) generated by the target identification unit 113 is externally processed via the transmission / reception control unit 116 and the communication interface 101. It is also preferable that the information is transmitted to the device, for example, the terminal 20. Further, it may be used by a predetermined application program in the cloud server 1.

Returning to the functional block diagram of FIG. 1, the learning data generation unit 114 generates and manages learning data for constructing a learning model constituting the classifier of the target identification unit 113. Here, the learning data is specifically generated by assigning a correct answer label (for example, "automobile") for the original image data to the candidate image area determined by the candidate area determination unit 112. Next, the learning model generation unit 115 generates a learning model for the object identification process using the (sufficient amount) of the learning data generated and managed by the learning data generation unit 114.

Here, of course, in order to correspond to a plurality of objects (for example, automobiles, humans, etc.), the learning data generation unit 114 generates learning data for each object, and the learning model generation unit 115 uses these learning data. It is also preferable to generate a learning model for each object. In this case, the object identification unit 113 can acquire a plurality of classifiers capable of identifying each of the plurality of objects, and can output the identification results for each object in parallel.

[Functional configuration of image data providing client]
According to the functional block diagram also shown in FIG. 1, the terminal 20 has a communication interface 201, a camera 202, a display (DP) 203, and a processor memory. Here, this processor memory stores one embodiment of the image data providing program according to the present invention, and also has a computer function, and by executing this image data providing program, image data Carry out the provision process.

For this reason, as the image data providing client according to the present invention, instead of the terminal 20 which is a drive recorder, another in-vehicle information processing device equipped with the image data providing program according to the present invention, and further, a camera is provided. It is also possible to adopt a smartphone, a notebook type or tablet type computer, a personal computer (PC), or the like. Further, a terminal connected to the drive recorder via Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like, for example, a smartphone may be used as the image data providing client.

Further, the processor memory includes a video generation unit 211, a coding parameter extraction unit 212, a macroblock selection unit 213, a significant video stream generation unit 214, a presentation information generation unit 215, and a transmission / reception control unit 216. .. It should be noted that these functional components can be regarded as the functions of the image data providing program stored in the processor memory. Further, the processing flow shown by connecting the functional components of the terminal 20 in FIG. 1 with arrows is also understood as an embodiment of the image data providing method according to the present invention.

In the present embodiment, as will be described later, the terminal 20 transmits a significant video stream (FIG. 7) composed of coded frames synchronized with the coded parameter map frame to the cloud server 1. Of course, a normal compressed coded video stream may be transmitted.

Similarly, in the functional block diagram of FIG. 1, the image generation unit 211 generates image (video) data based on the shooting data output from the camera 202. In the present embodiment, the terminal 20 is a drive recorder, and as a default setting, the image generation unit 211 acquires image (video) from the camera 202 to capture the shooting data of the situation outside the vehicle at least when the vehicle 2 is running. Data is generated and saved.

Further, in the present embodiment, the video generation unit 211 is the H.A. 264 and H. The generated image (video) data is subjected to compression coding processing in a standard format such as 265 to generate a compressed coded video stream (original image stream).

The coding parameter extraction unit 212 simply decompresses (decodes) the generated compressed coded video stream, interprets (parses) the bit stream, and extracts the coding parameters in each macroblock. Here, the coding parameter can be at least one of (a) a motion vector for forward prediction, (b) a code amount for in-screen prediction, and (c) a quantization step size. ..

The macroblock selection unit 213 is based on the coding parameters extracted from each macroblock.
(A) The motion vector of the forward prediction has a size equal to or larger than a predetermined value and a direction forming an angle equal to or larger than a predetermined direction with respect to the reference direction toward the vanishing point.
One, two, or one of three conditions that (b) the code amount of the in-screen prediction has a magnitude equal to or greater than a predetermined value, and (c) the quantization step size has a magnitude equal to or less than a predetermined value. Select macroblocks that satisfy all (which one is preset). By the way, the coding parameters satisfying such conditions generally indicate that the temporal fluctuation of the image is larger than a predetermined value.

Here, with respect to the above condition (a), the motion vector may be calculated by referring to one frame two or more frames before, may be calculated by referring to one frame immediately before, or may be calculated by referring to a plurality of frames. It may be calculated by referring to the frame. In any case, it is also preferable that a reference (threshold value) having a size equal to or larger than a predetermined value is appropriately adjusted according to such a calculation method.

Further, with respect to the above condition (b), the code amount of the in-screen prediction is usually large when the edge (contour of the target) is present in the image, and is small when the image is flat. Therefore, for example, when the predetermined target is an automobile, the code amount in the vicinity of the tire position corresponding to the boundary between the automobile and the flat road surface becomes large, and the macroblock around that portion becomes easy to be selected.

Further, regarding the above condition (c), the quantization step size is set to a small value in the image region where there is little change in order to increase the compression efficiency under the adaptive quantization method, while the change is made. In a large area, it is set to a large value according to the insensitivity of human vision. Therefore, for example, when a predetermined object is an automobile, the quantization step size of the automobile that rapidly changes its position in the image is set to be smaller, and the macroblock around that is easily selected.

The macroblock selection unit 213 then uses the "macroblock position information" as "macroblock position information".
(A) Position information in the image of the macroblock in which the coding parameter satisfies a predetermined condition, for example, the macroblock address, the position coordinate value in the image of the macroblock, or (b) the coding parameter map is generated. This "macroblock position information" is notified to the cloud server 1.

Here, the macroblock position information generated in this way simply indicates an image area that has changed with a rapidness of a predetermined value or more, and should be identified and monitored, for example, existing around the automobile 2. There is a high possibility that the information is related to the position in the image of the target (for example, another car or pedestrian).

FIG. 5 is a schematic diagram for explaining an embodiment of the macroblock selection process in the macroblock selection unit 213.

First, the original video frame (camera image data) shown in FIG. 5 (A) (generated by the camera 202) includes an object that is changing at a substantially constant speed. Further, the motion vector of each macroblock in this original video frame tends to increase as the distance from the vanishing point increases, but in any case, the motion vector tends toward the vanishing point. Therefore, in this case, the macroblock that satisfies the predetermined condition for the motion vector is not selected.

Next, the original video frame (camera image data) (generated by the camera 202) shown in FIG. 5 (B) includes an object whose position is rapidly changed in the lower right corner of the image. There is. Further, the motion vector of the macroblock around the lower right of this image has a size of a predetermined value or more, and further has a direction forming an angle of a predetermined value or more with respect to the direction toward the vanishing point. As a result, with respect to the motion vector, the macroblock group near the lower right of this image is selected as satisfying a predetermined condition.

In addition, at the position of the macroblock group selected as described above, for example, a sudden lane change or a sudden deceleration is performed while traveling in front (of the vehicle 2 equipped with the camera 202). It is expected that a car (moving body) is shown.

Finally, the original video frame (camera image data) (generated by the camera 202) shown in FIG. 5 (C) includes an object whose position is rapidly changed in the lower right corner of the image. There is. Further, the code amount of the in-screen prediction around the lower right of this image has a size equal to or larger than a predetermined value. As a result, the macroblock group in the lower right corner of this image is selected as satisfying the predetermined condition with respect to the code amount of the in-screen prediction.

In the embodiment described below, a frame (frame of the coding parameter map) to which the coding parameters satisfying the predetermined conditions as described above are mapped is generated in units of GOP (Group Of Pictures) (the frame of the coding parameter map). See FIG. 7, which will be described later).

Returning to the functional block diagram of FIG. 1, the significant video stream generation unit 214 generates a significant video stream in which the frames of the original video stream synchronized with the frames of the coding parameter map are combined in chronological order. Here, in the present embodiment, the generated significant video stream is transmitted to the cloud server 1 via the transmission / reception control unit 216 and the communication interface 201 in association with the generated corresponding “macroblock position information”. become.

FIG. 6 is a schematic diagram for explaining an embodiment relating to the generation of frames constituting the significant video stream in the significant video stream generation unit 214.

The distribution range of macroblocks satisfying a predetermined condition in the original video frame shown in FIG. 6 (A) is shown in FIG. 6 (B). In the present embodiment, the significant video stream generation unit 214 generates an original video frame (FIG. 6 (C)) in which only the portion corresponding to the distribution range of the macroblock shown in FIG. 6 (B) is extracted. It is a frame that constitutes a significant video stream. This also makes it possible to significantly reduce the amount of data in the finally generated significant video stream.

FIG. 7 is a schematic diagram for explaining an embodiment of the significant video stream generation process in the significant video stream generation unit 214.

According to FIG. 7, the significant video stream generation unit 214 extracts frames synchronized with the frames of the coded parameter map generated for each GOP from the coded original video frame group, and extracts these frames. Generate a significant video stream by combining them in chronological order.

Further, as also shown in FIG. 7, the significant video stream generation unit 214 stops the automobile 2 related to itself when the code amount of the I (intra) frame becomes equal to or more than a predetermined code amount threshold value (for example, 1 megabit). It is determined that the value is inside, and the frame of this time interval is excluded from the constituent frames of the significant video stream. This makes it possible to further reduce the amount of data (number of frames) of the significant video stream. Here, it is also preferable that the predetermined code amount threshold value as a determination standard is set to a different value depending on the case of coding by CBR (constant bit rate) and the case of coding by VBR (variable bit rate).

Incidentally, the reason why the running / stopping of the automobile 2 can be determined by the code amount of the I frame is as follows. That is, for example, in the original video stream, when the object in the image changes at a constant speed and there is no abrupt change, there is a constant movement in the image, so that the macroblock of forward prediction increases. On the other hand, the macroblocks of the in-screen prediction decrease, and as a result, the code amount of the entire frame tends to decrease. For example, H.I., which performs coding processing at a fixed bit rate of about 6 megabits per second. In 264, the code amount of the I frame in the running original video stream usually changes in 0.5 to 0.8 megabits.

On the other hand, when there is a steep change in the image in the original video stream, the macroblocks of the in-screen prediction increase, while the macroblocks of the forward prediction decrease, and eventually the code amount of the entire frame becomes large. It becomes a tendency.

Furthermore, when there is no movement in the image in the original video stream, that is, when the vehicle 2 is stopped, the macroblocks of forward prediction decrease, while the macroblocks of in-screen prediction increase, resulting in In addition, the code amount of the entire frame tends to increase. For example, in an original video stream that is stopped and does not change, the code amount of the I frame exceeds 1 megabit. By observing the code amount of the I frame in this way, it is possible to determine whether the vehicle is running or stopped.

Returning to the functional block diagram of FIG. 1, the presentation information generation unit 215 acquires the target identification result distributed from the cloud server 1 via the communication interface 201 and the transmission / reception control unit 216, and the terminal 20 presents it to the user. It may be used to generate service information.

For example, the presentation information generation unit 215 incorporates the target identification result into the on-board driving support program, and in the real-time image of the traveling direction by the camera 202 displayed on the display 203, sudden lane change, width adjustment, etc. It is also possible to highlight the image portion of a vehicle that is making a sudden movement such as overtaking, and also sound an alarm to warn the user.

As described in detail above, according to the present invention, at least one candidate is based on information about the position of a unit image area (eg, macroblock) in an image where the coding parameters satisfy a predetermined condition. The image area can be determined. That is, when carrying out the target identification process, it is not necessary to first perform a process of detecting a predetermined target in the image using a detector and then perform a heavy-duty process of determining the candidate position in advance. .. As a result, it is possible to reduce the processing load of the target identification processing on the side receiving the image data to be processed.

By the way, the configuration and method of the present invention uses 5G (5th generation mobile communication system) capable of transmitting a huge amount of video data, and uploads a compressed and encoded video stream from a large number of clients to a server. In the situation, it is considered that it will greatly contribute to the solution of expected important issues such as improving the efficiency of image analysis / target identification processing on the server and reducing the burden. For example, depending on the embodiment of the present invention, high-resolution images taken by autonomous vehicles, drones, and various robots may be collected by 5G, and the collected images may be efficiently and reliably identified to create and provide new services. It is also possible to connect.

Various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely an example and is not intended to be a constraint. The present invention is limited only by the claims and their equivalents.

1 Cloud server (target identification device)
101, 201 Communication interface 111 Image acquisition unit 112 Candidate area determination unit 113 Target identification unit 114 Learning data generation unit 115 Learning model generation unit 116 Transmission / reception control unit 2 Automobile 20 Terminal (client)
202 camera 203 display (DP)
211 Video generation unit 212 Coding parameter extraction unit 213 Macroblock selection unit 214 Significant video stream generation unit 215 Presentation information generation unit 216 Transmission / reception control unit

Claims (11)

A method for identifying an object in a computer that identifies the object from image data that can include a predetermined object in the image.
Based on the information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies the predetermined condition, in the image data, the target is A step of determining at least one candidate image region, and
It is characterized by having a step of identifying the target from the image data by using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class. Target identification method to be performed. In the step of determining the candidate image region, as the candidate image region, the first candidate image region including the position of the unit image area in the image and the nth (n is from 2 to N (≧ 2)). It is determined that N candidate image regions including the entire (n−1) candidate image region or the nth candidate image region including a predetermined or more portion of the candidate image region (each integer) are determined. The target identification method according to claim 1, which is characterized. The nth candidate image region is characterized in that the (n-1) th candidate image region is arranged in the center of itself and is determined to include the entire (n−1) candidate image region. The target identification method according to claim 2. The object identification method according to claim 2, wherein the first candidate image area is determined so that the position of the unit image area in the image is determined to be the lower end or the lower end of the unit image area. Any of claims 2 to 4, wherein the first candidate image area is determined to have a larger area as the distance from the lower end of the image at the position in the image of the unit image area is smaller. The target identification method according to item 1. Any of claims 2 to 5, wherein the first candidate image region is determined to have a larger area as the distance from the vanishing point at a position in the image of the unit image area increases. The target identification method according to item 1. The video data is coded by MPEG (Moving Picture Experts Group), and the coding parameters that satisfy the predetermined conditions are
(A) A motion vector for forward prediction having a size equal to or larger than a predetermined value and a direction having an angle equal to or larger than a predetermined direction with respect to a reference direction.
The unit image area is a macroblock, which is at least one of (b) an in-screen prediction code amount having a size equal to or larger than a predetermined value and (c) a quantization step size having a size equal to or less than a predetermined size. The object identification method according to any one of claims 1 to 6, wherein the object is identified. The classifier is
At least one convolutional layer unit that takes each of the at least one candidate image area as an input and outputs the feature information related to the feature of the candidate image area, respectively.
Claim 1 is characterized in that feature information output from at least one convolution layer portion is collectively input and includes a fully connected layer portion (Fully-Connected Layers) that outputs information related to the target class. The target identification method according to any one of 7 to 7. A device that identifies a predetermined object from image data that can include the object in the image.
Based on the information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies the predetermined condition, in the image data, the target is A candidate area determining means for determining at least one candidate image area, and
Having a target identification means for identifying the target from the image data by using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class. A featured object identification device. A server generated by a client that acquires image data that can include a predetermined target in an image and identifies the target.
Information relating to the position of the unit image area in the image in which the coding parameter determined when the video data including the image data is encoded satisfies a predetermined condition, together with the image data from the client. A candidate area determining means for determining at least one candidate image area related to the target in the image data based on the acquired position information, and
Having a target identification means for identifying the target from the image data by using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class. Characteristic target identification server. A program that functions a computer that identifies a predetermined object from image data that can include the object in the image.
Based on the information related to the position in the image of the unit image area where the coding parameter determined when the video data including the image data is encoded satisfies the predetermined condition, in the image data, the target is A candidate area determining means for determining at least one candidate image area, and
Using a learned classifier that inputs the determined at least one candidate image area and outputs information related to the target class, the computer functions as a target identification means for identifying the target from the image data. A target identification program characterized by the fact that.