JP2021177597A - Imaging device and imaging system - Google Patents

Abstract

To allow a proper protection of personal information of an image taken by an imaging device equipped to such a mobile body as a drone or a vehicle.SOLUTION: The imaging device, which is equipped on or equipped in a mobile body, includes: a camera for taking an image around the mobile body; an image processing unit for processing the image taken by the camera; and a post-processing unit for transferring or recording the image processed by the image processing unit. The image processing unit detects personal information in an image taken by the camera, and performs image processing including mosaic processing to make the personal information unrecognizable.SELECTED DRAWING: Figure 5

Description

The present invention relates to an imaging device and an imaging system.

In recent years, a camera has been mounted on an air vehicle called a drone to shoot from the sky. Drone flight is currently subject to some restrictions in many countries. For example, in Japan, the Aviation Law prohibits flying in densely populated areas. However, if the safety of drone flight is ensured in the future, it is highly likely that regulations will be relaxed and drone flight will be permitted in densely populated areas.

By the way, since the camera mounted on the drone takes an image from the sky, there is a possibility that the photographed image contains personal information that cannot be captured when the image is taken from the ground. Therefore, in the case of an image taken by a camera mounted on a drone, a personal information protection function different from that of a general camera is required. For example, the veranda of a house hidden from the road side by a fence, windows on the second floor and above, etc. are originally invisible from the ground, and it is not preferable to photograph these parts.

In addition to flying objects such as drones, there is a drive recorder mounted on a car traveling on the road as an example in which the privacy of captured images becomes a problem. Even in the case of drive recorders, the protection of personal information contained in captured images has become a problem. That is, there are many cases where an image taken by a drive recorder contains personal information such as a person's face or a nameplate of a building, which may cause a problem when the taken image is released.

Conventionally, when publishing an image taken with a drone or drive recorder on the net or broadcasting, edit work such as applying a mosaic to the part containing personal information in the image before publishing it. Was there.

Patent Document 1 describes a technique for wirelessly transmitting a surveillance image taken by an airship from an airship to a center device and converting the image processing device installed in the center device into a low-resolution image or the like so as to maintain privacy. Has been done.

Japanese Unexamined Patent Publication No. 2016-119628

As described in Patent Document 1, it has been conventionally known to process an image taken by an air vehicle for privacy protection. However, even if this technology is applied to the above-mentioned drone as it is, there is a possibility that personal information cannot be properly protected. That is, normally, the image taken by the camera mounted on the drone is wirelessly transmitted to the controller that controls the drone and recorded in a memory or the like in the controller.
Here, by incorporating the image processing device described in Patent Document 1 into the controller, a recorded image with privacy protection should be obtained.

However, when the image signal wirelessly transmitted from the drone is illegally received by another device, the image signal is not subjected to image processing for privacy protection. Therefore, as a result, there is a possibility that an image in which personal information is not protected may be leaked.
Of course, during wireless transmission from the drone to the controller, it is possible to encrypt the image signal to prevent unauthorized reception, but it is not uncommon for the encryption to be broken.

In addition, the image taken by the drone needs to be monitored by the operator in real time on the controller side in order to control the flight of the drone. Therefore, it is necessary that the image taken by the drone does not require much time for encryption and decryption, and it is not preferable to perform very strong encryption.

Even in the case of a drive recorder for automobiles, when using the image recorded by the drive recorder, it is common to perform image processing for privacy protection, so usually when recording the image on the drive recorder. Is not considered for privacy protection.

In view of these points, an object of the present invention is to provide an imaging device and an imaging system that can appropriately protect personal information of captured images.

The image pickup device of the present invention is an image pickup device mounted on or built in a moving body, and is processed by a camera that photographs the surroundings of the moving body, an image processing unit that processes an image taken by the camera, and an image processing unit. It is provided with a post-processing unit for transmitting or recording an image, and the image processing unit detects personal information included in an image taken by a camera and performs image processing that makes the personal information indistinguishable.

Further, the image pickup system of the present invention is an image pickup system including a moving body having a built-in image pickup device and a controller for receiving an image signal transmitted from the image pickup device.
Here, the image pickup apparatus includes a camera that photographs the surroundings of the moving object, an image processing unit that detects personal information contained in the image captured by the camera and performs image processing that makes the personal information indistinguishable, and an image processing unit. It is provided with a transmission processing unit that transmits the image processed by the camera to the controller.
The controller includes a display unit that receives and displays an image transmitted by the transmission processing unit, and an operation unit that operates the movement of the moving body.

According to the present invention, an image taken by a camera built in or mounted on a moving body is transmitted or recorded after being subjected to image processing that makes personal information indistinguishable in the image pickup apparatus. Therefore, personal information is transmitted or recorded. Can be properly protected.
For example, since the recorded image signal has already been subjected to image processing so that the personal information cannot be identified, even if the recorded image signal is released, the personal information contained in the image signal is appropriately protected. Will be done. Further, on the side receiving the image signal taken by the image pickup device, the received image signal has already been subjected to image processing so that the personal information cannot be discriminated, so that the personal information can be appropriately protected. ..

It is a figure which shows the schematic structure example of the image pickup system by one Embodiment of this invention. It is a block diagram which shows the structural example of the image pickup system by one Embodiment of this invention. It is a figure which shows the list of priority at the time of image processing by one Embodiment of this invention. It is a flowchart which shows the flow which the processing is performed in order in a plurality of image recognition processing units according to the example of one Embodiment of this invention. It is a flowchart which shows the flow of the image processing in the 1st image recognition processing unit by one Embodiment of this invention. It is a flowchart which shows the flow of the image processing in the 2nd image recognition processing part by the Example of 1 Embodiment of this invention.

Hereinafter, an example of an embodiment of the present invention (hereinafter, referred to as “this example”) will be described with reference to the drawings.

[System configuration]
FIG. 1 shows a configuration example of the imaging system of this example.
The imaging system of this example includes a drone 100 and a controller 200.
The drone 100 is an air vehicle that flies according to instructions from the controller 200. The drone 100 has a built-in image pickup device composed of a camera 101 (FIG. 2) and the like, and can photograph the surroundings during flight.
The controller 200 is a terminal that wirelessly communicates with the drone 100, and can instruct the drone 100 in the direction and altitude of flight. Further, the controller 200 can receive the image signal captured by the drone 100 and display it.

[Internal configuration of device]
FIG. 2 is a block diagram showing an example of the internal configuration of the drone 100 and the controller 200.
The drone 100 includes a camera 101, a propeller 102, a wireless LAN module 103, a battery 104, and a sensor 105.
Further, the drone 100 includes a photographing control module 111, an image processing control module 112, a flight control module 113, a data transmission / reception module 114, a power supply control module 115, a first image recognition processing unit 121, and a second image recognition processing unit 122. ..
However, although the image recognition processing units 121 and 122 show two examples in FIG. 2, the number of the image recognition processing units is only one example and is not limited to two.

The camera 101 photographs the surroundings of the drone 100 at a constant frame cycle. The shooting with the camera 101 is performed based on the instruction from the shooting control module 111.
The propeller 102 can rotate based on the instruction from the flight control module 113 to fly the drone 100 in the instructed altitude and direction.

The wireless LAN module 103 wirelessly communicates with the controller 200 based on the control of the data transmission / reception module 114. The image signal captured by the camera 101 by wireless communication by the wireless LAN module 103 is wirelessly transmitted to the controller 200.
Further, the wireless LAN module 103 receives an instruction regarding flight such as the altitude and direction of flight from the controller 200, and supplies the received instruction regarding flight to the flight control module 113.
When the wireless LAN module 103 receives an instruction for shooting from the controller 200, the wireless LAN module 103 supplies the received instruction for shooting to the shooting control module 111.

The battery 104 supplies the power required to operate each part of the drone 100. The power supply by the battery 104 and the management of the remaining battery level are performed by the power control module 115.
The sensor 105 detects the flight state of the drone 100 and supplies the detected flight state data to the flight control module 113. For example, the sensor 105 has a function of detecting the flight altitude of the drone 100 from the ground. Further, the sensor 105 may be provided with a positioning unit using GPS (Global Positioning System) or the like, and may have a function of detecting the flight position of the drone 100. When the positioning unit is provided, the sensor 105 may use the altitude (elevation) obtained by the positioning as the flight altitude.

The photographing control module 111 causes the camera 101 to perform photographing based on the instruction received from the controller 200 by wireless transmission. For example, when receiving instructions such as a shooting angle and a zoom magnification, the shooting control module 111 also controls shooting with the camera 101 so that the shooting state is based on those instructions.
On the controller 200 side, while the drone 100 is in flight, it is necessary to monitor the image taken by the camera 101 and instruct the flight direction and the like. Therefore, the camera 101 constantly captures an image while the drone 100 is in flight.

The image processing control module 112 is an image processing unit that executes processing (image processing) on an image signal captured and acquired by the camera 101.
When executing image processing, the image processing control module 112 uses the first image recognition processing unit 121 and the second image recognition processing unit 122 to recognize an object or a person included in each area in the image. .. Then, the image processing control module 112 executes image processing based on the recognized result so that the personal information of a part of the area in the image cannot be determined.
When recognizing a person or an object, the image recognition processing units 121 and 122 perform recognition by, for example, machine learning processing. Further, the image recognition processing units 121 and 122 acquire the flight position (absolute position on the map), the flight altitude, and the direction of the camera from the flight control module 113 and the camera 101, and recognize by referring to these data. You may do so. The details of the image processing performed by using the image recognition processing units 121 and 122 will be described later.

Then, the image signal for which image processing has been executed by the image processing control module 112 is transmitted to the controller 200 side via the wireless LAN module 103 under the control of the data transmission / reception module 114. The data transmission / reception module 114 and the wireless LAN module 103 are post-processing units for transmitting an image signal that has undergone image processing.

In the case of this example, the image signal wirelessly transmitted from the wireless LAN module 103 is an image signal processed by the image processing control module 112. However, if the first image recognition processing unit 121 or the second image recognition processing unit 122 cannot recognize the personal information from the captured image, that is, the image that does not include the personal information, the personal information by the image processing control module 112 is used. It is wirelessly transmitted via the wireless LAN module 103 as it is without performing image processing that makes it impossible to distinguish.

The controller 200 includes a wireless LAN module 201, a battery 202, a display module 203, an operation module 204, a power supply control module 205, and a recording module 206.
The wireless LAN module 201 wirelessly transmits commands such as a flight state and a shooting state to the drone 100, and receives an image signal wirelessly transmitted from the drone 100.
The battery 202 supplies power to operate each part of the controller 200 under the control of the power control module 205.

The display module 203 includes a display unit for displaying an image, displays an image transmitted from the drone 100, and displays information necessary for operating the drone 100.
The operation module 204 is an operation unit that accepts user operations for controlling the flight of the drone 100. The operation module 204 may be configured with a touch panel incorporated in the display unit so that operations for flight can be performed by touch operations on the screen.
The recording module 206 records the image transmitted from the drone 100 in a built-in memory or the like.

The controller 200 may be configured as a dedicated device for operating the flight of the drone 100, etc., but for example, an application program for functioning as a controller may be mounted on an information processing terminal such as a smartphone or a tablet terminal. It may be a controller for drone control.

[Processing of captured images]
The image processing control module 112 is an image that prevents personal information from being discriminated by the first image recognition processing unit 121 and the second image recognition processing unit 122 based on an object recognized from the image captured by the camera 101. Execute the process.
Next, when the connected image recognition processing units 121 and 122 determine the personal information, the image processing control module 112 displays an image so that the personal information cannot be determined from the captured image based on predetermined conditions. Perform processing.
Here, in the image processing performed by the image processing control module 112, for example, a mosaic process is performed in which the area including the corresponding personal information is converted into an image whose color and brightness change in a mosaic pattern at regular pixel intervals, and the personal information is personal information. There is a process that makes it impossible to determine. However, the image processing control module 112 performs mosaic processing as an example, and personal information may not be discriminated by other processing.

The first image recognition processing unit 121 and the second image recognition processing unit 122 discriminate each set object (specific object or person) as personal information.
Here, the first image recognition processing unit 121 discriminates the face of a person included in the captured image as personal information, and performs mosaic processing on the discriminated face so that the personal information cannot be discriminated.
In addition, the second image recognition processing unit 122 determines a specific object included in the captured image as personal information.

FIG. 3 shows a list of the attributes of the object that the second image recognition processing unit 122 determines as personal information, and an example of setting the priority order for each attribute of the object.
In the list of examples in FIG. 3, windows on the second floor and above of the building are set as objects having the highest priority. Below, in the list of FIG. 3, the entire building on the second and higher floors, the garden, the windows on the first floor, the entire building on the first floor, the entrance, the walls, and the roof are set with their respective priorities.

The priority of the list shown in FIG. 3 is used to limit the one-frame image taken by the camera 101 to the maximum range in which the mosaic processing is applied when the object is subjected to the mosaic processing.
That is, a value such as 50% is set in the image processing control module 112 as the maximum range for performing mosaic processing.

Then, when performing the mosaic processing, the image recognition processing units 121 and 122 perform the mosaic processing within one frame up to the set maximum value area, and even if there is an object in the range beyond that, the mosaic processing is performed. Do not apply. This is to prevent the flight position and the like from being indistinguishable from the mosaic-processed image when the captured image is transmitted to the controller 200 side and the drone 100 is operated with the image.

In limiting the range of applying the mosaic processing, which object is subjected to the mosaic processing is determined by the priority given to the list of FIG. That is, the image recognition processing units 121 and 122 perform mosaic processing in order from the object having the highest priority, and when the set value (50%, etc.) is exceeded, the frame is not further mosaic processed. do.

FIG. 4 is a flowchart showing a flow in which processing is sequentially performed in the image recognition processing units 121 and 122.
First, the image processing control module 112 determines whether or not the captured image of one frame is generated by the camera 101 (step S1). When it is determined in step S1 that the captured image of one frame has been generated (YES in step S1), the image processing control module 112 acquires the image of the corresponding one frame (step S2).

Then, the image processing control module 112 passes the acquired one-frame image to the first image recognition processing unit 121 (step S3). The first image recognition processing unit 121 performs mosaic processing on the target portion by the recognition processing of the face of a person, and sends the image of one frame to which the mosaic processing is performed to the image processing control module 112 (step S4).

After that, the image processing control module 112 passes the one-frame image acquired from the first image recognition processing unit 121 to the second image recognition processing unit 122 (step S5). When the second image recognition processing unit 122 detects a predetermined object, the corresponding portion is subjected to mosaic processing, and the mosaic-processed one-frame image is sent to the image processing control module 112 (step S6). ..

Mosaic processing is performed on the image of one frame in this way, and the image processing control module 112 returns to step S1 and waits until the image of the next one frame is supplied.
Further, when it is determined in step S1 that the captured image of the next one frame is not generated by the camera 101 (NO in step S1), the image processing control module 112 ends the image processing.

FIG. 5 is a flowchart showing a processing example when the first image recognition processing unit 121 discriminates a human face included in a captured image as personal information.
First, the first image recognition processing unit 121 detects a face included in the one-frame image taken by the camera 101 by face recognition processing, and the area of the object (face) in the detected one-frame image is 100 pixels. It is determined whether or not it is × 100 pixels or more (step S11). In step S11, when the area of the object is 100 pixels × 100 pixels or more (YES in step S11), the first image recognition processing unit 121 performs image processing for replacing the face portion of the object with a mosaic. (Step S12).

Further, even when the first image recognition processing unit 121 detects a face, if the area of the object (face) is less than 100 pixels × 100 pixels (NO in step S11), the image processing is replaced with a mosaic. Do not execute.
The first image recognition processing unit 121 executes the image processing shown in the flowchart of FIG. 4 for all the frames captured by the camera 101.

FIG. 6 is a flowchart showing a processing example when the second image recognition processing unit 122 discriminates an object included in the captured image as personal information.
First, the first image recognition processing unit 121 detects an object included in the one-frame image taken by the camera 101 in the image analysis process. The objects to be detected here are those shown in the list of FIG.

Then, the first image recognition processing unit 121 determines whether or not the area of the object in the detected one frame image is 100 pixels × 100 pixels or more (step S21). In step S21, when the area in the image of the object is 100 pixels × 100 pixels or more (Yes in step S21), the first image recognition processing unit 121 starts the flight altitude of the current drone 100 from the flight control module 113. Data is acquired and it is determined whether or not the altitude is 2 m or more (step S22).

When it is determined in step S22 that the flight altitude is 2 m or more (YES in step S22), the first image recognition processing unit 121 determines whether or not the current position of the drone 100 is on the road (step S23). .. The flight control module 113 determines whether or not the current position of the drone 100 is on the road, for example, from the map data of the drone 100 and the data of the current flight position.

When it is determined in step S23 that the current position of the drone 100 is not on the road (NO in step S23), and when it is determined in step S22 that the flight altitude is less than 2 m (NO in step S22), the second image recognition The processing unit 122 determines whether or not the three-dimensional distance (straight line distance) from the object is 5 m or more (step S24). When it is determined in step S24 that the distance to the object is not 5 m or more (NO in step S24), the second image recognition processing unit 122 performs mosaic processing on the area of the object (step S25).

Further, in step S21, when it is determined that the area in the image of the object is not 100 pixels × 100 pixels or more (NO in step S21), and in step S23, when it is determined that the current position of the drone 100 is on the road. (YES in step S23), and when it is determined in step S24 that the distance to the object is 5 m or more (YES in step S24), the second image recognition processing unit 122 performs mosaic processing on the object. No processing is performed, and the process ends.

When the area in one frame subjected to mosaic processing by the image recognition processing units 121 and 122 exceeds the above-mentioned set value (50% or the like), the image recognition processing units 121 and 122 are at that time. The frame should not be further mosaicked.
Since the mosaic processing in the image recognition processing units 121 and 122 is executed in the order described with reference to FIG. 4, for example, when the mosaic processing in the first image recognition processing unit 121 has already performed the mosaic processing up to the set value. Is not executed by the second image recognition processing unit 122.

As described above, the image taken by the camera 101 in the drone 100 is subjected to mosaic processing so that the personal information cannot be discriminated by the image processing control module 112, and then transmitted to the controller 200 to display the controller 200. Displayed on module 203. Further, the image recorded by the recording module 206 of the controller 200 is also an image subjected to mosaic processing.

The operator of the drone 100 controls the flight direction of the drone 100 and the like while looking at the image displayed on the display module 203 of the controller 200. At this time, since the display image includes a portion containing personal information such as a person's face or a window of a building, the personal information can be appropriately protected. Further, the recorded image is the same as the image displayed on the controller 200, and it is possible to appropriately protect personal information.

Further, even if the image signal wirelessly transmitted from the drone 100 is illegally received by another device, the personal information of the image signal cannot be identified by the mosaic processing, so that the human face cannot be identified. Images containing personal information such as will not be leaked.

Further, when performing the mosaic processing, the processing area in one frame is limited to the set value, so that the operator of the drone 100 sees the image displayed on the display module 203 of the controller 200 and at least It is possible to discriminate the limited image content, and it is possible to prevent the occurrence of a situation in which maneuvering becomes impossible due to image processing.

[Modification example]
The present invention is not limited to the above-described embodiment, and can be modified or modified without departing from the gist of the present invention.
For example, the object and the priority shown in FIG. 3 are examples, and the image processing control module 112 may be an object for mosaic processing of other objects.
For example, the image processing control module 112 mosaic-processes things in a private space such as a house nameplate, a car license plate, a veranda, a window of a room that cannot be seen from the road (ground), laundry, and the private space itself. It may be an object.

Further, in the above-described embodiment, when the processing area in one frame reaches the set value, the mosaic processing is not performed any more. On the other hand, when the area to be subjected to the mosaic processing in one frame is increased, the image recognition processing units 121 and 122 may be a low-resolution image in which the size of one mosaic at the time of performing the mosaic processing is reduced. .. However, even in that case, the operator of the drone 100 needs to make an image capable of maneuvering the flight as it is.

Further, the image processing control module 112 may cancel the mosaic processing of a specific portion in the image according to the instruction from the controller 200. For example, when a specific part in the image displayed by the display module 203 of the controller 200 is specified by the touch operation of the operator, the image processing control module 112 cancels the mosaic processing of the corresponding part and transmits it to the controller 200. However, it is possible not to apply the mosaic processing that interferes with the maneuvering.

Further, the drone 100 wirelessly transmits the image processed image signal to the controller 200, but a recording module is provided in the drone 100, and the recording module records the image processed image signal. You may.

Further, the drone 100 or the controller 200 may not perform the mosaic processing when receiving some emergency signal from the outside.
Further, the image recognition processing units 121 and 122 may exclude the person or the object registered in advance from the object to be subjected to the mosaic processing.

Further, in the above-described embodiment, the example in which the two image recognition processing units 121 and 122 are installed has been described, but a larger number of image recognition processing units are installed in each image recognition processing unit. The object may be detected in more detail and mosaic processing may be performed. In this case, the image processing control module 112 may change the algorithm for detecting an object from an image in a plurality of image recognition processing units. For example, one specific image recognition processing unit may detect an object from an image by deep learning.

Further, in the above-described embodiment example, an example in which a mosaic process is performed on the relevant portion has been described as a process for making the personal information indistinguishable. On the other hand, other processing may be performed as a processing for making the personal information indistinguishable. For example, the image processing control module 112 includes a process of filling an area containing personal information with a single color or the like, a process of forming a wire frame of an area containing personal information into an image shown by an outline, or an area containing personal information. May be processed to make an animation image. Alternatively, these processes may be combined.

Further, in the above-described embodiment, the camera 101 is built in the drone 100. On the other hand, even if the camera is externally attached to the drone 100, the same processing may be performed.
Further, as an application example of the image pickup device or the image pickup system of the present invention, the point where the drone is applied is also an example, and the image pickup device or the image pickup system as another moving body may be applied. For example, the present invention may be applied to an image pickup device mounted on an automobile called a drive recorder.

100 ... Drone, 101 ... Camera, 102 ... Propeller, 103 ... Wireless LAN module, 104 ... Battery, 105 ... Sensor, 111 ... Shooting control module, 112 ... Image processing control module, 113 ... Flight control module, 114 ... Data transmission / reception module , 115 ... power control module, 121 ... first image recognition processing unit, 122 ... second image recognition processing unit, 200 ... controller, 201 ... wireless LAN module, 202 ... battery, 203 ... display module, 204 ... operation module, 205 … Power control module, 206… Recording module

Claims (8)

An imaging device mounted on or built into a moving body.
A camera that captures the surroundings of the moving object,
An image processing unit that processes images taken by the camera, and
A post-processing unit for transmitting or recording an image processed by the image processing unit is provided.
The image processing unit is an imaging device that detects personal information contained in an image taken by the camera and performs image processing that makes the personal information indistinguishable. The imaging device according to claim 1, wherein the image processing that makes the area containing personal information indistinguishable is any one of a mosaic processing, a single color filling process, a wireframe processing, and an animation image processing. .. The image processing unit acquires the attribute of the current position of the moving body, and based on the acquired attribute, determines that a predetermined area in the captured image is an area containing personal information according to claim 1 or 2. The imaging device described. The image processing unit acquires the current height information of the moving body, and based on the acquired height information, determines a predetermined area in the captured image as an area containing personal information. Or the image pickup apparatus according to 2. The image processing unit detects the distance to the object in the captured image, and determines that a predetermined area in the captured image is a region containing personal information based on the detected distance. 2. The image pickup apparatus according to 2. The image processing unit limits the area where the image processing is performed so that the ratio of the image processed area to the entire screen of the image captured by the camera is equal to or less than a preset predetermined value. Item 2. The image pickup apparatus according to any one of Items 1 to 5. When limiting the area where the image processing is performed so that the ratio of the area where the image processing is performed is equal to or less than a preset predetermined value, the limitation is made based on the priority of the preset object. The imaging apparatus according to claim 6, wherein the region to be processed is selected. It is an imaging system including a moving body having a built-in imaging device and a controller for receiving an image signal transmitted from the imaging device.
The image pickup device
A camera that captures the surroundings of the moving object,
An image processing unit that detects personal information contained in an image taken by the camera and performs image processing that makes the personal information indistinguishable.
A transmission processing unit that transmits an image processed by the image processing unit to the controller is provided.
The controller includes a display unit that receives and displays an image transmitted by the transmission processing unit.
An imaging system including an operation unit for operating the movement of the moving body.

Images