JP7448721B2 - Imaging device and video processing system - Google Patents

Description

The present invention relates to an imaging device and a video processing system, and more particularly to an imaging device and a video processing system that can perform inference processing using machine learning and have a video processing function for protecting privacy.

In recent years, there has been an increase in demand for cameras such as surveillance cameras that can photograph a large number of people. These cameras have the advantage of being connected to a LAN (Local Area Network) and allowing remote video monitoring. On the other hand, if the security is breached, the information captured may be leaked, which may pose a problem in terms of privacy protection.

Therefore, Patent Document 1 discloses a method of protecting privacy by performing processing such as reversible mosaic processing and mask processing on captured images. The processed image can be restored to its original image by performing corresponding restoration processing.

Japanese Patent Application Publication No. 2009-33738

In Patent Document 1, if restoration information including restoration information for performing restoration processing is leaked to the outside, a malicious third party can perform restoration processing and obtain the original image. To prevent this, it is necessary to distribute irreversible images over a LAN, but in that case, the original images cannot be restored. For this reason, it becomes impossible to perform face recognition or action recognition using image recognition technology or the like.

SUMMARY OF THE INVENTION In view of the above-mentioned problems, an object of the present invention is to provide an imaging device and a video processing system that can transmit predetermined information regarding an image while protecting the image information at a higher level.

In order to achieve the above object, one of the typical imaging devices of the present invention captures a video to obtain an image, detects a predetermined area from within the image, resizes the detected detection area, and then detects the image. The present invention is characterized in that the feature amount of the region is extracted, and an image arranged in the detection region of the acquired image is output as a mask image in which the extracted feature amount is arranged two-dimensionally.

Furthermore, one of the video processing systems of the present invention includes an imaging device and a video processing device, and the imaging device captures a video to obtain an image, detects a predetermined area from within the image, and detects a predetermined area from within the image. resizing the detected detection area to extract feature quantities of the detection area, outputting an image arranged in the detection area of the acquired image as a mask image in which the extracted feature quantities are arranged in two dimensions, and the video processing device , the method is characterized in that an image output by the imaging device is input, a feature amount is acquired from the mask image, and inference processing is performed based on this feature amount.

According to the present invention, in an imaging device and a video processing system, predetermined information regarding an image can be transmitted while providing higher protection of the image information.
Problems, configurations, and effects other than those described above will be clarified by the following embodiments.

FIG. 1 is a block diagram showing an embodiment of the video processing system of the present invention. FIG. 2 is a block diagram showing an example of the processing system section of FIG. FIG. 3 is a diagram illustrating an example of a process for calculating feature amounts applied in the video processing system of the present invention. FIG. 4 is a diagram showing an example of the processing of the imaging device in the video processing system of the present invention. FIG. 5 is a diagram showing an example of processing of the video processing device in the video processing system of the present invention.

A mode for carrying out the present invention will be described.

FIG. 1 is a block diagram showing an embodiment of the video processing system of the present invention. The video processing system in FIG. 1 includes an imaging device 1 and a video processing device 5. The imaging device 1 includes an imaging section 2 and a processing system section 3. The video processing device 5 also includes a processing system section 6 and a display output section 7. Note that the display output section 7 may not be included in the video processing device 5 and may be configured separately from the video processing device 5. The video processing device 5 can be a personal computer, a tablet computer, a server, or the like.

The imaging device 1 includes a configuration of one or more cameras, and can be placed in various locations. For example, it may be placed as a surveillance camera at a monitoring location.

The imaging unit 2 is a camera configured to obtain information by forming an image of incident light on an image sensor through a lens or an aperture. Examples of the image sensor here include a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the like. The obtained information is sent to the processing system unit 3. Further, the imaging unit 2 can perform imaging processing using a video processing IC (Integrated Circuit) such as an FPGA (Field Programmable Gate Array). On the other hand, this video processing IC may be integrated with the processing system section 3.

The processing system section 3 acquires information captured by the imaging section 2 and performs the processing shown in FIG. 4, which will be described later. A specific configuration example will be described later with reference to FIG. 2, and specific processing contents will be described later with reference to FIG. The processed information is sent to the processing system unit 6.

The processing system unit 6 acquires information from the processing system unit 3 and performs the processing shown in FIG. 5, which will be described later. A specific configuration example will be described later with reference to FIG. 2, and specific processing details will be described later with reference to FIG.

The display output unit 7 is a device that can display the content processed by the processing system unit 6. For example, the information is displayed using a liquid crystal display (LCD), an organic EL (OEL) display, a touch panel, or the like.

Information can be exchanged between the imaging device 1 and the video processing device 5 via the Internet or the like. For example, connect to a LAN or the like. In addition, information may be exchanged via a dedicated communication line. That is, the processing content of the imaging device 1 located at a remote location can be confirmed by the video processing device 5. Further, the imaging device 1 and the video processing device 5 do not have to be in a one-to-one relationship, and one imaging device 1 may have a plurality of video processing devices 5, and a plurality of imaging devices 1 may have a one-to-one correspondence. The video processing device 5 may also handle this. Further, the video processing device 5 may be configured to be capable of setting and operating the imaging device 1.

FIG. 2 is a block diagram showing an example of the processing system section of FIG. A specific example of the processing system units 3 and 6 will be described as a computer system 300 in FIG.

The main components of computer system 300 include one or more processors 302 , memory 304 , terminal interface 312 , storage interface 314 , I/O (input/output) device interface 316 , and network interface 318 . These components may be interconnected via memory bus 306, I/O bus 308, bus interface 309, and I/O bus interface 310.

Computer system 300 may include one or more processing devices 302A and 302B, collectively referred to as processor 302. Each processor 302 executes instructions stored in memory 304 and may include onboard cache. As the processing device, a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), etc. can be applied.

Memory 304 may include random access semiconductor memory, storage devices, or storage media (either volatile or nonvolatile) for storing data and programs. Memory 304 also represents the entire virtual memory of computer system 300 and may include virtual memory of other computer systems connected to computer system 300 via a network. Although memory 304 may be conceptually considered a single entity, it may be a more complex arrangement, such as a hierarchy of caches and other memory devices.

Memory 304 may store all or some of the programs, modules, and data structures that implement the functions described in this embodiment. For example, memory 304 may store application 350. Application 350 may include instructions or writings that perform functions described below on processor 302, or may include instructions or writings that are interpreted by other instructions or writings. Applications 350 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to processor-based systems. may be done. Application 350 may include data other than instructions or descriptions. Other data input devices, such as cameras and sensors, may also be provided to communicate directly with bus interface 309, processor 302, or other hardware of computer system 300.

Computer system 300 may include a bus interface 309 that provides communication between processor 302 , memory 304 , display system 324 , and I/O bus interface 310 . I/O bus interface 310 may couple with I/O bus 308 for transferring data to and from various I/O units. I/O bus interface 310 connects multiple I/O interfaces 312, 314, 316, and 318, also known as I/O processors (IOPs) or I/O adapters (IOAs), via I/O bus 308. You may communicate with Display system 324 may include a display controller, display memory, or both. A display controller may provide video, audio, or both data to display device 326. Computer system 300 may also include devices, such as one or more sensors, configured to collect data and provide the data to processor 302. Display system 324 may be connected to a display device 326, such as a standalone display screen, a television, a tablet, or a handheld device. Display device 326 may include speakers for rendering audio. Alternatively, a speaker for rendering audio may be connected to the I/O interface. Alternatively, the functionality provided by display system 324 may be implemented by an integrated circuit that includes processor 302. Similarly, the functionality provided by bus interface 309 may be implemented by an integrated circuit that includes processor 302.

The I/O interface provides the ability to communicate with various storage or I/O devices. For example, terminal interface 312 may include a user output device such as a video display device, a speaker television, or a user input device such as a keyboard, mouse, keypad, touch pad, trackball, buttons, light pen, or other pointing device. User I/O devices 320 can be attached. Using the user interface, a user operates a user input device to input input data and instructions to user I/O device 320 and computer system 300, and to receive output data from computer system 300. Good too. The user interface may be displayed on a display device or played through a speaker via user I/O device 320, for example.

Storage interface 314 allows attachment of one or more disk drives or direct access storage devices 322 . Storage device 322 may be implemented as any secondary storage device. The contents of memory 304 may be stored in storage device 322 and read from storage device 322 as needed. I/O device interface 316 may provide an interface to other I/O devices. Network interface 318 may provide a communication pathway so that computer system 300 and other devices can communicate with each other. This communication path may be, for example, network 330.

Although computer system 300 includes a bus structure that provides a direct communication path between processor 302, memory 304, bus interface 309, display system 324, and I/O bus interface 310, computer system 300 has a hierarchical configuration. , star, or web configurations, multiple hierarchical buses, and parallel or redundant communication paths. Further, although I/O bus interface 310 and I/O bus 308 are shown as a single unit, in reality, computer system 300 may include multiple I/O bus interfaces 310 or multiple I/O buses 308. may be provided. Also, although multiple I/O interfaces are shown to separate I/O bus 308 from various communication paths leading to various I/O devices, some or all of the I/O devices may It may also be connected directly to the system I/O bus.

Computer system 300 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer.

When the computer system 300 in FIG. 2 is applied to the processing system unit 3 in FIG. 1, the display device 326 has an arbitrary configuration and may or may not be included. Furthermore, the imaging unit 2 can be applied as a user I/O device 320. Furthermore, when the computer system 300 in FIG. 2 is applied as the processing system unit 6 in FIG. 1, the display device 326 can be applied as the display output unit 7. Further, the network 330 can be applied as a network interposed between the processing system section 3 and the processing system section 6.

FIG. 3 is a diagram illustrating an example of a process for calculating feature amounts applied in the video processing system of the present invention. FIG. 3 shows a configuration example of machine learning using CNN (Convolution Neural Networks) for estimating a person from a face image. The numbers listed above each layer are the numbers of neurons in that layer, but these are examples.

A part of a specific image is input from the input layer 11, and is transmitted to the first convolution layer 12 and pooling layer 13, and is connected to the convolution layer 12 and pooling layer 13, which are the subsequent layers. After these processes, there is a fully connected layer, including an input layer 16, an intermediate layer 17, and an output layer 18. The number of neurons in the output layer 18 is equivalent to the number of classes. When performing face recognition, this is approximately equivalent to the number of people that can be identified. Note that when a part of a specific image is input from the input layer 11, for example, a 200×200 image is input after being resized to 64×64.

The input layer 11 acquires image information of a specific size (64×64 pixels in FIG. 3). The example in FIG. 3 is an image of a human face captured by face detection.

Next, the convolution layer 12 performs convolution processing. A filter is applied to the image acquired by the input layer 11. By applying the filter, the size becomes smaller (60×60 in FIG. 3). Then, as many filters as the number of prepared filters (eight in FIG. 3) are output.

Next, the pooling layer 13 performs pooling processing. Compression is applied to the information output by the convolution layer 12. As a result, the size is halved (30×30 in FIG. 3).

Next, the convolution layer 14 performs convolution processing. The information compressed by the pooling layer 13 is further filtered to reduce the size (26×26 in FIG. 3). Then, as many filters as the number of prepared filters (16 in FIG. 3) are output.

Next, the pooling layer 15 performs a pooling process. Compression is applied to the information output by the convolution layer 14. As a result, the size is halved (13×13 in FIG. 3).

Next, in the input layer 16 of the fully connected layer, the three-dimensional information (13×13×16) is rearranged into one-dimensional information (2704) in the pooling layer 15. The information here indicates the feature amount. Although FIG. 3 shows that the convolutional layer and the pooling layer are repeated twice (two layers), the present invention is not limited to this, and may be repeated even more times.

From the input layer 16 of the fully connected layer, a mask image can be formed. A mask image here means an image in which the original image cannot be identified (if it is a face, someone cannot be identified from the image alone). This process is an irreversible video processing process, and once a mask image is formed, the original image cannot be restored.

Specifically, as shown in FIG. 3, one-dimensional information 16-1 (2704 in FIG. 3), which is the information of the input layer 16 of the fully connected layer, is converted into two-dimensional image information 16-2 (52× in FIG. 3). 52). The information at this time can be held as image information, such as color density information for a monochrome image, and color type and color density information for a color image. For example, in the case of a black and white image, one pixel can be converted as 8-bit information, and in the case of an RGB color image, one pixel can be converted as 24-bit information. The 52×52 pixel image information is expanded into a 200×200 pixel mask image 16-3. This is a conversion process to match the size of the originally captured face image.

The created mask image 16-3 is then returned to the original one-dimensional information for inference processing. Specifically, the mask image 16-3 (200×200 in FIG. 3) is resized to the two-dimensional image information 16-4 (52×52 in FIG. 3) before being stretched, and then the one-dimensional image The information is rearranged as information 16-1 (2704 in FIG. 3). This makes it possible to once convert the information in the input layer 16 of the fully connected layer into the mask image 16-3 and place it on the image.

Next, in the intermediate layer 17 of the fully connected layer, the number of neurons is 1000 in FIG. 3. This is just an example, and an appropriate number can be applied as needed. Further, the number of intermediate layers 17 may be increased to constitute a plurality of layers.

In the output layer 18 of the next fully connected layer, the number of neurons is 100. Here, this number of neurons is the number of classes, and corresponds to the number that can be classified. For example, in the case of face recognition, person A, person B, person C, etc., and the person is estimated based on the neuron that fires the most. Through such inference processing, it is possible to classify 100 people. Alternatively, it is also possible to classify the remaining 99 people into "other".

FIG. 4 is a diagram showing an example of the processing of the imaging device in the video processing system of the present invention. The processing here is performed on the imaging device 1 side, and unless otherwise specified, is performed in the processing system section 3 of the imaging device 1. Here, irreversible video processing is performed.

The imaging device 1 first performs video shooting 21. This is performed by the imaging unit 2, and can be realized using an image sensor and a video processing IC such as an FPGA. The shoot will be filmed on video. For example, shooting may be performed at 30 frames per second (30 fps) or more. The images photographed by the imaging section 2 are sent to the processing system section 3 for each frame of image, and can be processed individually.

Next, the processing system unit 3 performs face detection 22 on this input video. Face detection 22 is a process of identifying the shape of a human face and detecting a range that includes the face. This is done automatically using existing techniques. If it is identified as a human face, that area is detected. Further, since the processing described later is performed, the processing can be performed to detect when the range identified as a face has a certain number of pixels or more. If the number of bits handled by one neuron of the input layer 16 is the same as the number of bits of one pixel, in the example of FIG. 4, the minimum range is set to 52×52 pixels.

Next, the detection area is resized 23 by the detection area resizing section. This resizes the area detected by face detection 22 to a predetermined size. In this resizing, since the area detected by the face detection 22 is not constant, the area is converted to a predetermined size suitable for calculating the next feature amount. In the example of FIG. 4, processing is performed to convert 200x200 pixels to 64x64 pixels.

Next, the feature amount calculation section performs feature amount calculation 24 of the detection area. Here, feature amounts necessary for face recognition are obtained using CNN or the like. The calculation of this feature amount is similar to the processing from the input layer 11 to the input layer 16 of the fully connected layer explained with reference to FIG.

Next, the feature values are rearranged/resized 25. Here, processing is performed to convert the data into a format of a size that can be applied to the area where the face has been detected. The number of feature neurons calculated in the input layer 16 of the fully connected layer is 2704, and when this is converted into a two-dimensional area, it becomes an area of 52×52. On the other hand, the area detected by face detection 22 is 200×200. In order to apply the data of a two-dimensional area of 52×52 calculated from the number of neurons of the feature amount to the area of 200×200 of the face detection 22, the data of one neuron is expanded to approximately 4 pixels and allocated. As a result, data in an area of 52×52 is converted to data in an area of 200×200. Note that the process of rearranging/resizing the feature amounts 25 here is the same as the process from the one-dimensional information 16-1 to the mask image 16-3 explained with reference to FIG.

Here, the larger the above-mentioned enlargement ratio is, the smaller the change between pixels or between frames in the mask area becomes. This alleviates sudden changes between pixels and between frames, making it easier to perform processing using a lossy codec. Further, this feature amount needs to fit within the data area of the minimum image size in which face detection is performed, but depending on this minimum size, it is also possible to treat, for example, the output of a pooling layer in the middle of the CNN as the feature amount.

Next, mask processing 26 is performed on the rearranged feature amounts to the original image detected by face detection 22. This is placed on the original image by applying the rearranged feature amounts (200×200) to the area detected by the face detection 22 as a mask image 16-3. Since the mask image 16-3 is an image whose color type and density are based on feature amounts, it is different from the original image of the area detected by the face detection 22 and has information different from that of a human face.

Next, mask processing metadata addition 27 is performed on the image that has been subjected to mask processing 26 . Here, the index number of the image on which the masking process has been performed, the coordinates of the starting point on the image, the length of one side, etc. are given. As a result, information for specifying the area on which the masking process has been performed and information for specifying the image on which the masking process has been performed is provided.

Next, it is output to the outside 28. Here, when outputting to the outside, processing is performed using a codec in order to compress the transmission capacity. In the case of video, a lossy codec is generally used, but depending on the application, only intermittent transmission of images is required, in which case a lossless codec may be used. The externally outputted information here is sent to the video processing device 5 via the Internet network or the like.

FIG. 5 is a diagram showing an example of processing of the video processing device in the video processing system of the present invention. The processing here is performed on the video processing device 5 side, and unless otherwise specified, it is performed in the processing system unit 6 of the video processing device 5. Here, inference processing using machine learning is performed to identify the person.

First, video data having an image outputted from the imaging device 1 at the external output 28 in FIG.

Next, a feature amount extraction/resizing/rearranging unit performs feature amount extraction, resizing, and rearranging processing 32 from the metadata of the video data. In this process, first, a mask image 16-3 is extracted from the video data. This range can be specified from the attached metadata. Next, the information is returned to two-dimensional image information 16-4 (52×52 in FIG. 5), and further rearranged into one-dimensional information 16-5 (2704 in FIG. 5). This is similar to FIG. 3. This provides the value of the feature amount. Note that this value is processed by resizing, codec, etc. during the process, so the data value may shift slightly and may not match completely. However, this deviation does not affect the process of obtaining an inference result from the next feature amount, and a value that is the same as or close to the original feature amount (one-dimensional information 16-1) can be obtained.

Next, an inference result is acquired 33 from the feature amount. This is similar to the processing of fully connected layers 16-18 in FIG. Here, the class is identified from the feature amount by the inference result acquisition unit. In the case of the example shown in FIG. 5, an individual can be identified from the face through inference processing.

Note that the above processing can be performed by storing information regarding an individual's face in the video processing device 5. For example, when outputting classes for 100 people, information for 100 people is held, and individuals can be identified from their feature amounts. Furthermore, if the person does not correspond to the person recorded in advance, it is also possible to prepare one class that outputs that the person is another person.

Furthermore, arrangements for parameters for extracting feature quantities, such as the data structure of feature quantities and neural network parameters, are shared between the imaging device 1 and the video processing device 5 in advance. With this, when the mask image 16-3 is sent to the video processing device 5, it is possible to return it to one-dimensional information 16-5 and output the class from the feature amount. Regarding the setting of these parameters, a function may be provided that allows the video processing device 5 to also set the imaging device 1.

Although the above embodiment has been described as an example of a process for identifying a person by face detection, it is also possible to identify a person's behavior. For example, the imaging device 1 has a person detection function, detects the entire person, and masks the entire person using a two-dimensional image that includes feature amounts. Then, the video processing device 5 infers the behavior of the masked person from the feature amount. In this case, the class is output for each type of human behavior.

(effect)
In the embodiments described above, irreversible masking of a human region (face or entire person) where privacy protection is important can be realized. At the same time, the destination can receive the data necessary to identify people and actions, and perform post-processing inferences as necessary. This makes it possible to determine who the person is and what they are doing even in the masked area.

When using conventional reversible mask processing, decoding the masked portion restores the original image of the person, for example, and if that image is leaked, all personal information contained in the image will be leaked. On the other hand, with the method according to this embodiment, even if information is leaked and decrypted by a malicious third party, only the label information such as the name associated with it in the case of face recognition, and the label information associated with it in the case of behavioral recognition. The information can be kept to a minimum, consisting only of behavior label information.

Furthermore, when inferring the results of person recognition or action recognition on the imaging device side, if the data is transmitted and the communication is intercepted, the label information will be leaked. On the other hand, in this embodiment, inference is made from the received feature amount on the video processing device 5 side. Therefore, even if data from the imaging device 1 is leaked, inference cannot be made unless the data structure of the feature values, the structure of the parameters of the neural network, etc. are known. Therefore, the information from the imaging device 1 is doubly protected in addition to communication encryption, and can be data that is even more difficult to decode. Further, by embedding the feature amount in the mask image 16-3, the transmission capacity can be reduced.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to add, delete, or replace some of the configurations of each embodiment with other configurations.

For example, in the above embodiment, processing is performed to embed feature amounts in the mask image 16-3 in order to reduce transmission capacity. However, it is also possible to apply a configuration in which appropriate mask processing (for example, masking with the same color and density) is performed without embedding the feature information in the image, and the feature information and the image are transmitted separately.

Further, in the above embodiment, an example using CNN was shown, but the present invention can also be applied using a DNN (Deep Neural Networks) method as machine learning.

1... Imaging device, 2... Imaging unit, 3... Processing system unit, 5... Video processing device, 6... Processing system unit, 7... Display output unit, 11... Input layer, 12... Convolution layer, 13... Pooling layer, 14... Convolution layer, 15... Pooling layer, 16... Input layer of fully connected layer, 17... Middle layer of fully connected layer, 18... Output layer of fully connected layer, 21... Video shooting, 22... Face detection, 23... Resize detection area, 24... Calculate feature amount of detection area, 25... Rearrange/resize feature amount, 26... Mask processing to original image, 27... Add mask processing metadata, 28... External output, 31... Video input , 32... Extracting/resizing/rearranging feature quantities, 33... Obtaining inference results from feature quantities, 300... Computer system, 302... Processor, 302A, 302B... Processing device, 304... Memory, 306... Memory bus, 308... I/O bus, 309... Bus interface, 310... I/O bus interface, 312... Terminal interface, 314... Storage interface, 316... I/O device interface, 318... Network interface, 320... User I/O device, 322 ...Storage device, 324...Display system, 326...Display device, 330...Network, 350...Application

Claims (7)

A mask in which an image is obtained by shooting a video, a predetermined region is detected from the image, a feature amount of the detected region is extracted by resizing the detected detection region, and the extracted feature amount is arranged two-dimensionally. outputting an image placed in the detection area of the acquired image as an image;
An imaging device characterized in that the extraction of the feature amount is performed using a CNN (Convolution Neural Networks) or DNN (Deep Neural Networks) method . The imaging device according to claim 1,
An imaging device characterized in that the predetermined area is a human face area . A mask in which an image is obtained by shooting a video, a predetermined region is detected from the image, a feature amount of the detected region is extracted by resizing the detected detection region, and the extracted feature amount is arranged two-dimensionally. outputting an image placed in the detection area of the acquired image as an image;
An imaging device characterized in that the image to be output is provided with information specifying a range of the mask image within the image . Equipped with an imaging device and a video processing device,
The imaging device captures an image to obtain an image, detects a predetermined region from within the image, resizes the detected detection region to extract a feature amount of the detection region, and doubles the extracted feature amount. outputting an image placed in the detection area of the acquired image as a dimensionally arranged mask image;
The video processing system is characterized in that the video processing device receives an image output from the imaging device, acquires a feature amount from the mask image, and performs inference processing based on the feature amount . The video processing system according to claim 4,
A video processing system characterized in that the feature amount extraction and the inference processing are performed using a CNN (Convolution Neural Networks) or DNN (Deep Neural Networks) method . The video processing system according to claim 4 ,
The video processing system is characterized in that the predetermined area is a human face area, and the inference process is a process for identifying a person . The video processing system according to claim 4 ,
A video processing system characterized by having a function of setting parameters used for extracting feature amounts in the imaging device from the video processing device .

Images