TWI673682B - Method of modifying image including photographing restricted element, and device and system for performing the method - Google Patents

Abstract

The present invention provides a user device including: a memory configured to store data; and a controller configured to identify included in an image captured by the user device or received by the user device from an external source Based on the data stored in the storage in response to the degree of intimacy between the identified individual and the user of the user device being lower than a predetermined level, changing a portion of the image corresponding to the identified individual to Unrecognizable.

Description

Image correction method containing photography restriction element, device and system for executing method 【priority】

This application claims the priority of each patent application filed with the Korean Intellectual Property Office: Korean Patent Application No. 10-2014-0098652 filed on July 31, 2014 and Korean Patent filed on April 10, 2014 Application No. 10-2015-0051133, the disclosure of which is incorporated herein by reference in its entirety.

The apparatus and method consistent with the exemplary embodiments are related to a method for correcting an image (where a photo or moving image captured by a user device includes a photographic restriction element, the photo or moving image is corrected), and a device and method for performing System of methods.

When capturing photos or moving images by using a photographic device such as a video camera or camcorder, strangers can be included in the captured photos or captured movements In the image. At this time, the stranger may be exposed to the captured photo or captured moving image against his / her will, and sometimes, the privacy of the stranger may be violated. However, it is difficult for a photographer to notice a stranger whenever he / she is photographing.

When categorizing photography-restricted individuals and performing image processing from people included in the captured images, making photography-restricted individuals indistinguishable, photographers can protect the privacy of the photographer while conveniently using common methods Capture the image.

In detail, in recent years, since a portable device such as a smart phone or a personal computer (PC) generally supports a photographing function, an individual may be photographed by a stranger, and therefore preparation is necessary.

Meanwhile, the photography restriction element may include not only individuals but also objects or locations. Therefore, if the image captured by the personal device includes a photography-restricted object or a photography-restricted position, image processing can be performed so that the photography-restricted object or the photography-restricted position is indistinguishable.

If the image captured by the personal device includes a photography restriction element, one or more exemplary embodiments provide a method of correcting the image.

According to an exemplary embodiment aspect, a user device is provided, including: a memory configured to store data; and a controller configured to identify a device included in a device captured by the user device or An external source receives the individual in the image and responds to the identified individual's degree of intimacy with the user's device user below a predetermined level based on the data stored in the storage, corresponding to the identified individual Part of the personal image changed to be unrecognizable.

The user device may further include a camera configured to capture an image.

The controller may be further configured to determine whether the position of the captured image is a photographing-restricted position, and when the position is the photographing-restricted position, change a portion corresponding to the identified personal image to be unrecognizable.

The data stored in the memory may be an address book, and the controller may be further configured to determine that the degree of intimacy is lower than a predetermined level in response to the identified individual not being included in the address book.

The data stored in the storage may be at least one photo, and the controller may determine that the degree of intimacy between the user and the identified individual is lower than a predetermined level in response to the identified individual not being included in the at least one photo. .

The data stored in the storage can be a user's social networking service (SNS) usage record, and the controller can determine the degree of intimacy by analyzing the SNS usage record.

The controller can determine whether the position of the captured image is a photography-restricted position by accessing an external server database, and correct the image when the position is determined as the photography-restricted position.

The controller may be further configured to recognize the photography-restricted object from the image, and may determine the position of the captured image as the photography-restricted position in response to identifying the photography-restricted object from the image.

The controller may perform at least one of a mosaic process, a blur process, a background image layout process, and an alternative image layout process on a portion of the image to change the portion of the image to be unrecognizable.

According to another aspect of the exemplary embodiment, there is provided a system for correcting an image, providing a server configured to execute an image processing server; and configured to request image processing while transmitting an image to a server user device, The server is further configured to identify the individuals included in the image received from the user device and respond to the identified individuals and the user device user ’s degree of intimacy below a predetermined level to correspond to the identified A part of the personal image changed to unrecognizable.

The server can determine whether the position of the captured image is a photography-restricted position by using the pre-stored data, and change the portion corresponding to the recognized personal image to be unrecognizable in response to the position being the photography-restricted position.

The server can receive the time and location of the captured image from the user device, and determine whether the identified personal photography is restricted at the received time and the received location by using pre-stored data.

The user device may encode the image by using the encryption key received from the server and transmit the encoded image to the server, and the server may decode the encoded image and modify the portion corresponding to the identified individual decoded image.

According to another aspect of the exemplary embodiment, there is provided a user device including: a camera configured to capture an image; and a communication interface configured to transmit a photography notification to a device of a photographer captured in the image And receiving a response to the photography notification from the photographer's device; and a controller configured to modify the image based on whether the captured image is permitted to respond.

According to another aspect of the exemplary embodiment, a method for correcting an image by a user device is provided, which includes: identifying an individual included in the image; and determining the degree of intimacy between the identified individual and the user of the user device Whether it is below the predetermined level And in response to the degree of intimacy being lower than a predetermined level, changing a portion corresponding to the identified personal image to be unrecognizable.

The determination may include determining whether the position where the image is captured is a photography-restricted position, and the change may include a portion that corrects the image when the position is a photography-restricted position.

The determination may include determining that the degree of intimacy is below a predetermined level in response to the identified individual not being included in an address book stored in the user device.

The determination may include determining that the degree of intimacy is below a predetermined level in response to the identified individual not being included in at least one photo stored in the user device.

The determining may include determining whether the degree of intimacy is lower than a predetermined level by analyzing a user social network service (SNS) usage record based on data stored in the user device.

The change may include performing at least one of a mosaic process, a blur process, a background image layout process, and an alternative image layout process on a portion of the image to change the portion of the image to be unrecognizable.

According to another aspect of the exemplary embodiment, a method for correcting an image by a server is provided, including: receiving an image from a user device and an image processing request; identifying an object included in the image in response to the image processing request; Determining whether the object is a photography-restricted element by at least one of a position where the image was captured, a time when the image was captured, and the degree of intimacy between the user and the recognized object; and executing the image in response to the object being determined as the photography-restricted element Processes to correct objects contained in the image.

The image processing request can indicate the type of image processing to be performed on the object, and the image processing type can correspond to the mosaic processing, blur processing, and background image layout. To manage or replace image editing processing.

The image processing request may include at least one of a position where the image is captured, a time when the image is captured, and a degree of intimacy between the user and the identified object.

Determining whether the object is a photography restriction element may include analyzing the received image to extract features indicating location information, comparing the extracted features with data stored in a server to find a location; and in response to the location being designated as a photography prohibited area, The object is determined to be a photographing restriction element.

1‧‧‧ users

2‧‧‧ users

3‧‧‧ users

3-1‧‧‧Edge

20‧‧‧ Statue

20-1‧‧‧Edge

30‧‧‧No photography sign

30-1‧‧‧ symbols

30-2‧‧‧ phrase

100‧‧‧ Smart Glasses

100-1‧‧‧Smart Watch

100-2‧‧‧Smart Phone

100-2a‧‧‧Smart Phone

100-2b‧‧‧Smartphone

100-3‧‧‧Camera

100-4‧‧‧Camcorder

100-5‧‧‧ Tablet PC

200‧‧‧ Smart Phone

201‧‧‧ Operation

202‧‧‧Operation

203‧‧‧Operation

204‧‧‧Operation

205‧‧‧Operation

300‧‧‧Server

400a‧‧‧ original image

400b‧‧‧corrected image

500a‧‧‧ original image

500b‧‧‧corrected image

600a‧‧‧ original image

600b‧‧‧corrected image

700a‧‧‧ original image

700b‧‧‧corrected image

800a‧‧‧ original image

800b‧‧‧ corrected image

801‧‧‧ pixels

802‧‧‧ pixels

900a‧‧‧ original image

900b‧‧‧corrected image

1000‧‧‧user device

1001‧‧‧ Alternative image

1000a‧‧‧ original image

1000b‧‧‧ corrected image

1100‧‧‧user interface

1100a‧‧‧original image

1100b‧‧‧corrected image

1110‧‧‧ Alternative Images

1200‧‧‧Camera

1200a‧‧‧original image

1200b‧‧‧corrected image

1210‧‧‧ Alternative Images

1300‧‧‧Storage

1301‧‧‧ Operation

1302‧‧‧ Operation

1303‧‧‧ Operation

1304‧‧‧ Operation

1305‧‧‧operation

1400‧‧‧ communication interface

1401‧‧‧ Operation

1402‧‧‧ Operation

1403‧‧‧ Operation

1404‧‧‧operation

1500‧‧‧controller

1501‧‧‧ Operation

1502‧‧‧ Operation

1503‧‧‧ Operation

1504‧‧‧ Operation

1510‧‧‧Image Analyzer

1520‧‧‧Image Corrector

1600a‧‧‧original image

1600b‧‧‧corrected image

1601‧‧‧ Operation

1602‧‧‧ Operation

1603‧‧‧ Operation

1604‧‧‧operation

1605‧‧‧operation

1800a‧‧‧original image

1800b‧‧‧corrected image

1901‧‧‧operation

1902‧‧‧operation

1903‧‧‧Operation

1904‧‧‧operation

1905‧‧‧Operation

2301‧‧‧Operation

2302‧‧‧ Operation

2303‧‧‧ Operation

2304‧‧‧ Operation

2305‧‧‧ Operation

2306‧‧‧Operation

2307‧‧‧ Operation

2401‧‧‧ Operation

2402‧‧‧ Operation

2403‧‧‧ Operation

2404‧‧‧Operation

2405‧‧‧ Operation

2406‧‧‧ Operation

2600a‧‧‧original image

2600b‧‧‧corrected image

2701‧‧‧Operation

2702‧‧‧ Operation

2703‧‧‧ Operation

2704‧‧‧Operation

2705‧‧‧Operation

2706‧‧‧Operation

2707‧‧‧Operation

3201‧‧‧ Operation

3202‧‧‧ Operation

3203‧‧‧ Operation

3204‧‧‧ Operation

3205‧‧‧ Operation

3206‧‧‧ Operation

3207‧‧‧Operation

3400a‧‧‧original image

3400b‧‧‧corrected image

3500a‧‧‧original image

3500b‧‧‧corrected image

3600a‧‧‧original image

3600b‧‧‧corrected image

3601‧‧‧pixels

3602‧‧‧pixels

3700a‧‧‧original image

3700b‧‧‧corrected image

3710‧‧‧ Alternative Images

3801‧‧‧ Operation

3802‧‧‧ Operation

3803‧‧‧ Operation

3804‧‧‧ Operation

3805‧‧‧operation

3806‧‧‧ Operation

4000a‧‧‧ original image

4000b‧‧‧corrected image

4100a‧‧‧Original image

4100b‧‧‧corrected image

4201‧‧‧ Operation

4202‧‧‧operation

4203‧‧‧ Operation

4204‧‧‧ Operation

4400a‧‧‧original image

4400b‧‧‧corrected image

4500a‧‧‧ original image

4500b‧‧‧corrected image

4601‧‧‧operation

4602‧‧‧ Operation

4603‧‧‧ operation

4604‧‧‧operation

4710‧‧‧Privacy Information

5001‧‧‧ Operation

5002‧‧‧ Operation

5003‧‧‧ operation

5004‧‧‧ Operation

5005‧‧‧ Operation

5006‧‧‧ operation

5007‧‧‧ operation

5210‧‧‧Image

5211‧‧‧Area

5220‧‧‧Enlarge image

5221‧‧‧area

5301‧‧‧Operation

5302‧‧‧ Operation

5303‧‧‧ Operation

5304‧‧‧ Operation

5305‧‧‧ Operation

5306‧‧‧Operation

5410‧‧‧Image

5411‧‧‧area

5420‧‧‧Enlarge image

5421‧‧‧area

5510‧‧‧Image

5520‧‧‧Image

5521‧‧‧area

5530‧‧‧ "Send" button

5601‧‧‧operation

5602‧‧‧Operation

5603‧‧‧ Operation

5604‧‧‧Operation

5605‧‧‧operation

5606‧‧‧operation

5607‧‧‧Operation

5608‧‧‧Operation

5710‧‧‧ Video Call Screen

5711‧‧‧Facial area

5712‧‧‧Facial area

5801‧‧‧ Operation

5802‧‧‧ Operation

5803‧‧‧ Operation

5804‧‧‧Operation

5805‧‧‧operation

5806‧‧‧Operation

5910a‧‧‧Image

5910b‧‧‧Image

5911‧‧‧persons

5920a‧‧‧Image

5920b‧‧‧Image

6201‧‧‧ Operation

6202‧‧‧Operation

6203‧‧‧ Operation

6204‧‧‧ Operation

6205‧‧‧operation

6400a‧‧‧original image

6400b‧‧‧corrected image

6500a‧‧‧original image

6500b‧‧‧corrected image

6600a‧‧‧Original image

6600b‧‧‧corrected image

6610‧‧‧ Pop-up screen

6701‧‧‧Operation

6702‧‧‧Operation

6703‧‧‧Operation

6704‧‧‧Operation

6705‧‧‧Operation

6801‧‧‧ Operation

6802‧‧‧ Operation

6803‧‧‧ Operation

6804‧‧‧ Operation

6805‧‧‧operation

6806‧‧‧Operation

6807‧‧‧Operation

A‧‧‧User

B‧‧‧User

C‧‧‧User

The above and / or other aspects will be more apparent by describing certain exemplary embodiments with reference to the accompanying drawings, wherein:

FIG. 1 is a diagram for describing an environment for performing a method of correcting an image including a photography restriction element according to an exemplary embodiment.

FIG. 2 is a flowchart of a method of correcting an image including a photographing restriction element according to an exemplary embodiment.

3 to 6 are diagrams for describing a method of correcting an image including a photography restriction element according to an exemplary embodiment.

7 to 12 are diagrams for describing a method of correcting a personal image including a photography restriction according to an exemplary embodiment.

13 to 16 are flowcharts of a method of correcting a personal image including a photography restriction according to an exemplary embodiment.

17 and 18 are diagrams for describing a method of correcting an image including a photographing restriction element by using at least two user devices according to an exemplary embodiment.

19 is a flowchart of a method of correcting an image including a photographing restriction element by using at least two user devices according to an exemplary embodiment.

20 to 22 are diagrams for describing a method of correcting an image including a photography restriction element via communication with a subject's device according to an exemplary embodiment.

23 and 24 are flowcharts of a method of correcting an image including a photographing restriction element via communication with a device of a subject according to an exemplary embodiment.

25 and 26 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server according to an exemplary embodiment.

FIG. 27 is a flowchart of a method of correcting an image including a photographing restriction element by using a server according to an exemplary embodiment.

28 to 31 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server according to other exemplary embodiments.

FIG. 32 is a flowchart of a method of correcting an image including a photographing restriction element by using a server according to another exemplary embodiment.

FIG. 33 to FIG. 37 are diagrams for describing a method of correcting an image of an object including a photography restriction according to an exemplary embodiment.

FIG. 38 is a flowchart of a method of correcting an image including a photographic restricted object according to an exemplary embodiment.

39 to 41 are diagrams for describing a method of correcting capturing an image at a photographing restriction position according to an exemplary embodiment.

FIG. 42 is a flowchart of a method of correcting capturing an image at a photographing restricted position according to an exemplary embodiment.

FIG. 43 to FIG. 45 are diagrams for describing correction in photography according to other exemplary embodiments. A diagram of a method for capturing images at a restricted location.

FIG. 46 is a flowchart of a method of correcting capturing an image at a photographing restricted position according to another exemplary embodiment.

47 to 49 are diagrams for describing an example of restricting personal photography according to privacy protection information according to an exemplary embodiment.

FIG. 50 is a flowchart of a method for restricting personal photography according to privacy protection information according to an exemplary embodiment.

FIG. 51 is a block diagram of a user device according to an exemplary embodiment.

FIG. 52 is a diagram for describing an example of modifying a captured image before storing the captured image in a user device according to an exemplary embodiment, with respect to a method of modifying the captured image.

FIG. 53 is a flowchart describing an example of modifying a captured image before storing the captured image in a user device according to a method for modifying the captured image according to an exemplary embodiment.

54 and 55 are diagrams describing an example of a method for correcting a captured image describing an example of correcting an image stored in a user device while transmitting the image externally, according to an exemplary embodiment.

FIG. 56 is a flowchart describing an example of modifying a captured image while externally transmitting the image with respect to a method for modifying a captured image according to an exemplary embodiment.

FIG. 57 is a description of a method for modifying a captured image according to an exemplary embodiment. The captured image is immediately corrected without storing the captured image in a user device and the modified captured image is transmitted externally. Instance of image Illustration.

FIG. 58 is a description of a method for modifying a captured image according to an exemplary embodiment. The captured image is immediately corrected without storing the captured image in a user device and the modified captured image is transmitted externally. Flow chart of an example of images.

59 to 61 are diagrams for describing an example of correcting an image captured at a photographing restriction position according to an exemplary embodiment.

FIG. 62 is a flowchart illustrating a method of correcting an image captured at a photographing restriction position according to an exemplary embodiment.

FIG. 63, FIG. 64A, FIG. 64B, FIG. 65A, FIG. 65B, FIG. 66A, and FIG. 66B are diagrams for describing examples of correcting a captured image according to a user selection according to an exemplary embodiment.

67 and 68 are flowcharts illustrating a method of correcting a captured image according to a user selection according to an exemplary embodiment.

Exemplary embodiments are described in more detail below with reference to the accompanying drawings.

In the following description, similar drawing reference numerals are used for similar elements even in different drawings. The matters defined in the description, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, it is apparent that the exemplary embodiments can be practiced without their specifically defined matters. Also, well-known functions or constructions are not described in detail because well-known functions or constructions will confuse the description with unnecessary details.

When preceded by a component list, an expression such as "at least one of" Decorates the entire list of components, not individual components of the list.

FIG. 1 is a diagram for describing an environment for performing a method of correcting an image including a photographing restriction element according to an exemplary embodiment.

Referring to FIG. 1, the user A can capture an image including a person or an object, or an image of a specific location by using the smart glasses 100. Here, the captured image may be not only a still image such as a photo, but also a moving image.

As shown in FIG. 1, the smart glasses 100 may capture an image including a person (such as the user B), an object (such as the statue 20), or a photography prohibition sign 30. The smart glasses 100 can recognize at least one of an individual included in an image, an object included in the image, and a position where the image is captured, and can determine whether the image includes an element that restricts photography (ie, photography) based on the recognition result. Limiting element).

Here, the photography restriction element is a person, an object, or a place that is restricted from being photographed due to privacy, copyright, or security. This photography restriction element can be applied consistently to all people or differently depending on the photographer. For example, photography in military security areas is restricted to all people, but photography of specific individuals may be limited to individuals who do not know the specific individuals. If the photography-restricted person or object is contained in the image or the image is captured at the photography-restricted position, it is determined that the image includes a photography-restricted element.

Meanwhile, when the smart glasses 100 captures an image and transmits the image to a device or server connected to the smart glasses 100, the device or server can recognize the position of a person, an object, or the image and determine whether the image includes a photography restriction element.

If it is determined that the image includes a photography restriction element, the smart glasses 100 or a device or server connected to the smart glasses 100 corrects the image. In detail, the image is modified so that the photographic restriction element is indistinguishable or unrecognizable. Will be detailed later Describe the method of correcting the image.

Meanwhile, the user A may capture an image by using any of various devices other than the smart glasses 100. For example, User A may use a wearable device (such as smart watch 100-1), a mobile device (such as smart phone 100-2 or tablet personal computer (PC) 100-5), or a general photography device (such as Video camera 100-3 or video camera 100-4). However, any of various devices capable of capturing images or moving images is usable and will be apparent to those having ordinary knowledge in the art.

The device typically carried by most users is the smart phone 100-2, and information about the user (need to perform one or more exemplary embodiments) is likely to be stored in the smart phone 100-2. In addition, since the smart phone 100-2 can correct the image, it is assumed that the device described below by the user is the smart phone 100-2.

FIG. 2 is a flowchart of a method of correcting an image including a photographing restriction element according to an exemplary embodiment.

Referring to FIG. 2, in operation 201, an image is captured by using a user device. Here, the image can be obtained by using any of a general photography device such as a video camera or a video camera, a wearable device such as smart glasses or a smart watch, and a mobile device (smart phone or tablet PC) Or by using any of a variety of other devices capable of taking pictures.

In operation 202, the user device identifies at least one of a person included in the image, an object included in the image, and a location where the image is captured. Here, the user device may identify an individual included in the image by using, for example, a facial recognition method. However, the exemplary embodiment is not limited thereto, and the user device can be used for Any of various methods for identifying individuals to identify individuals contained in the image. The facial recognition method includes detecting a face from an image and extracting features from the detected face.

At the same time, the user device can identify the objects contained in the image by detecting the objects contained in the image and extracting features such as the shape and color of the detected objects.

At the same time, the user device can identify the position of the captured image by using a method of identifying an absolute position, a method of determining whether the position is a specific place, and a method of determining whether the position is a photographing restricted position.

First of all, the method of identifying absolute position can be captured by identifying in a global positioning system (GPS) or a Wi-Fi-based positioning system (WPS) embedded in a user device. The absolute position (such as latitude and longitude) at the point in time of the image is performed.

Second, the method of determining whether the location is a specific place may be performed by determining whether the user device is located at a specific place (such as a museum, restaurant, or church) at the time point of capturing the image. Alternatively, the absolute position of the user device may be identified by using the GPS of the user device, and the absolute position of the user device may be compared with a specific location, which is pre-stored in the database.

Again, the method of determining whether the position is a photographing-restricted position may be performed by determining whether the user device is located at the photographing-restricted position at the time point of capturing the image. For example, the user device may determine whether the location is a photographing prohibited area by analyzing a flag (such as a photographing prohibited area flag) included in the image. Alternatively, the absolute position of the user device may be identified by using the GPS of the user device, and the absolute position of the user device may be compared with the absolute position of the prohibited photographing area. The shooting area is pre-stored in the database.

In operation 203, after identifying at least one of a person, an object, and a location, the user device is based on the relationship between the user of the user device and the identified person, the identified object, and the identified location. At least one is used to determine whether the image includes a photographing restriction element. In other words, the user device determines whether the image needs to be corrected. If the image includes a photography restriction element, the user device determines that the image needs to be corrected; and if the image does not include a photography restriction element, the user device determines that the image does not need to be corrected.

Here, the relationship between the user of the user device and the identified individual is the relationship between the photographer and the photographer, and it is determined whether the photographer and the photographer know each other, and whether the photographer allows the photographer to treat him. / She is photographed or whether the subject is a celebrity and is not restricted to be photographed by the photographer. When it is determined based on the relationship that photography of the subject is restricted, it is determined that the image includes a photography restriction element.

At the same time, if the identified object is a photography-restricted object, the user device determines that the image includes a photography-restricted element. The photography-restricted object may be a copyright-protected object, such as a piece of art, or any object that may be restricted from photography by ordinary individuals due to various reasons.

It is determined whether the recognized object is a photography-restricted object by using any of various methods. For example, the memory of the user device may store a database of photography-restricted objects, and the user device may compare the identified objects with the stored database. Alternatively, the external server may store a database of photographing restricted objects, and the user device may communicate with the external server to access the database and compare the identified objects with the stored database. Alternatively, the user device may place the identified object The image of the file is transmitted to an external server, and then the external server can compare the transmitted image with the image stored in its database and transmit the comparison result to the user device. Alternatively, when the user enters a specific place (such as a museum), the user device may freely receive a database of photography-restricted objects in the specific place from a server managed by the specific place, and may compare the identified objects with the database . The determination of whether the identified object is a photography-restricted object may be performed by any of various other methods.

At the same time, if the recognized position is a photographing restriction position, the user device determines that the image includes a photographing restriction element. Photography restricted locations may be locations that are photographed due to security restrictions (such as military security areas) or locations that are photographed due to copyright restrictions (such as concert venues). In addition, for various reasons, locations can be restricted from being photographed by unauthorized individuals.

It is determined whether the recognized position is a photography restriction by using any of various methods. For example, the memory of the user device may store a database of photography-restricted locations, and the user device may compare the identified location with the stored database. Alternatively, the external server may store a database of photographed restricted locations, and the user device may communicate with the external server to access the database and compare the identified location with the stored database. Alternatively, the user device may determine whether the identified position is a photographing prohibited area by identifying and analyzing a mark (such as a photographing prohibited area mark) included in the image.

If it is determined in operation 204 that the image includes a photography restriction element, the user device may correct the image in operation 205. In detail, the user device can modify the image so that the photography restriction element included in the image is indistinguishable or unrecognizable. For example, a user device may perform mosaic processing or blur processing on photographic restricted individuals or objects. Alternatively, the user device may arrange a photograph-restricted person or object by excluding a screen of the photograph-restricted image captured by the individual or object, or by presetting an alternative image. If the image is captured at a photographing restricted position, the user device may perform mosaic processing or blur processing, so that the image is indistinguishable or unrecognizable, or the image may be arranged by presetting an alternative image. In addition, the user device may modify the image so that the photography restriction element included in the image is indistinguishable or unrecognizable. A method of correcting an image will be described in detail later.

If it is determined in operation 204 that the image does not include a photography restriction element, the method ends without modifying the image.

Operations 202 through 205 of FIG. 2 are described as being performed by a user device, but instead, some or all of operations 202 through 205 may be performed by a device or server connected to the user device, as described in detail below.

3 to 6 are diagrams for describing a method of correcting an image including a photography restriction element according to an exemplary embodiment.

Referring to FIG. 3, the smartphone 100-2 of the user A captures an image including the user B and the user C. Subsequently, the smart phone 100-2 determines whether the image includes a photography-restricted individual, and if the image includes a photography-restricted individual, corrects the image.

In FIGS. 4 to 6, the smart phone 100-2 determines whether the image includes a photographing restricted individual by using different methods, and adjusts the image.

Referring to FIG. 4, the smartphone 100-2 recognizes the faces of the user B and the user C by performing facial recognition on the original image 400a. Smart phone 100-2 can determine the intimacy between user A and user B and user A and user C Whether the degree of sexuality is below a predetermined level. For this purpose, the smartphone 100-2 compares the recognized face with a photo contained in the address book stored in the smartphone 100-2. The address book stored in the smartphone 100-2 can store people's photos together with people's names and phone numbers.

In FIG. 4, as a result of comparing the recognized face with a photo included in the address book, the smart phone 100-2 determines that the photo of the user B is in the address book. The smart phone 100-2 determines that the degree of intimacy between the user A and the user B is equal to or higher than a predetermined level and the user B is an individual allowed for photography.

Meanwhile, when the smart phone 100-2 cannot find the photo of the user C in the address book, the smart phone 100-2 determines that the degree of intimacy between the user A and the user C is lower than a predetermined level and the user C is for photography restricted individuals. In other words, the smart phone 100-2 can determine that the user A and the user C are not related to each other. Therefore, the smart phone 100-2 determines that the original image 400a includes a photography restriction element and corrects the original image 400a. For example, the smart phone 100-2 may modify a part of the image corresponding to the user C to be unrecognizable or unrecognizable. In detail, as shown in FIG. 4, the smartphone 100-2 performs a mosaic process on a face area of the user C who is an individual whose photography is restricted and outputs a corrected image 400 b.

In an exemplary embodiment, the mosaic process involves a lower resolution output of the face area of the user C. The lower resolution will be at a level, so that the facial area will be unrecognizable or unrecognizable.

The photographing restriction element is determined by comparing the recognized face with a photo contained in the address book, because when the individual is registered in the address book, the individual is likely to know the user A of the smart phone 100-2. In other words, the smart phone 100-2 borrows The relationship between the user A and the subject is determined by using the address book, and whether or not the photography restriction element is included in the original image 400a is determined based on the relationship.

Referring to FIG. 5, the smartphone 100-2 recognizes the faces of the user B and the user C by performing facial recognition on the original image 500a. The smart phone 100-2 can determine whether the degree of intimacy between the user A and the user B and the user A and the user C is lower than a predetermined level. For this purpose, the smartphone 100-2 compares the recognized face with an album stored in the smartphone 100-2.

In FIG. 5, the smartphone 100-2 determines that the user B is included in at least one of the photos by comparing the recognized face with the photo. The smart phone 100-2 determines that the degree of intimacy between the user A and the user B is equal to or higher than a predetermined level and the user B is an individual allowed for photography.

Meanwhile, when the smart phone 100-2 cannot find the photo of the user C in the address book, the smart phone 100-2 determines that the degree of intimacy between the user A and the user C is lower than a predetermined level and the user C is for photography restricted individuals. In other words, the smart phone 100-2 can determine that the user A and the user C are not related to each other. Therefore, the smart phone 100-2 determines that the original image 500a includes a photography restriction element and corrects the original image 500a. For example, the smart phone 100-2 may modify a part of the image corresponding to the user C so that the user C is indistinguishable or unrecognizable. In detail, as shown in FIG. 5, the smartphone 100-2 performs a mosaic process on the face area of the user C who is an individual whose photography is restricted and outputs a corrected image 500 b.

The photographic restriction element is determined by comparing the recognized face with the photo contained in the album, because when the individual is registered in the album, the individual is likely to know User A of the smartphone 100-2. In other words, the smart phone 100-2 determines the relationship between the user A and the subject by using the photo of the album, and determines whether the photography restriction element is included in the original image 500a based on the relationship.

Referring to FIG. 6, the smartphone 100-2 recognizes the faces of the user B and the user C by performing facial recognition on the original image 600a. The smart phone 100-2 can determine whether the degree of intimacy between the user A and the user B and the user A and the user C is lower than a predetermined level. For this purpose, the smart phone 100-2 determines the relationship between the user A, the user B, and the user C by analyzing the social network service (SNS) usage record of the user A. The SNS in the exemplary embodiment may be a web-based or Internet-based online SNS, and may be a platform constituting a social network or a social relationship.

In FIG. 6, if the degree of intimacy between the user A and the user B is determined to be equal to or higher than a predetermined level by analyzing the SNS usage record, the smart phone 100-2 determines that the user B is a photography-allowed individual . For example, the degree of intimacy can be determined by the following factors: whether user A and user B are friends on SNS, whether one of user A and user B has written a post about the other Comments, etc. Based on user settings or by default, each of the factors may have the same weight or different weights. For example, a weight of "3" may indicate the fact that user A and user B are friends on SNS, and a weight of "0.2" may indicate that each user A or user B posted on another person ’s post A comment. When the pre-set level is set to "10", if user A and user B are friends (ie, weight "3") and they make 100 comments on the other's post (ie, weight "20" "), The degree of intimacy (i.e.," 23 ") is determined to be higher than a predetermined level (i.e.," 10 ").

At the same time, if the smart phone 100-2 determines that the degree of intimacy between the user A and the user C is lower than a predetermined level by analyzing the SNS usage record, the smart phone 100-2 determines that the user C is a photography restriction personal. In other words, the smart phone 100-2 can determine that the user A and the user C are not related to each other. Therefore, the smart phone 100-2 determines that the original image 600a includes a photography restriction element and corrects the original image 600a. For example, the smart phone 100-2 may modify a part of the image corresponding to the user C so that the user C is indistinguishable or unrecognizable. In detail, as shown in FIG. 6, the smartphone 100-2 performs a mosaic process on a face region of the user C who is an individual whose photography is restricted and outputs a corrected image 600 b.

The smart phone 100-2 can simultaneously analyze the address book, album, and SNS usage of the user A and the user B and between the user A and the user C. For example, the smart phone 100-2 may consider the following factors to determine the degree of intimacy between the user A and the user B: whether the user B is registered in the address book of the user A, The images are stored in the album of user A, whether user A and user B are friends on SNS and whether one of user A and user B has written a comment about the other's post. The exemplary embodiments of the present invention are not limited thereto, and other factors that determine the degree of intimacy between the photographer and the subject may be considered.

7 to 12 are diagrams for describing a method of correcting an image including a photography-restricted individual according to an exemplary embodiment.

In FIG. 7, the blurring process is performed on the photographing restriction element. Referring to FIG. 7, the original image 700 a includes a user B and a user C. Here, it is assumed that the user C is a photographing restriction element. Therefore, a region where the original image 700a of the user C is displayed Is vague. In the corrected image 700b, the user C is blurred. Meanwhile, since the method of performing blur processing in a specific area of an image is widely known, details thereof are not provided here.

In FIG. 8, the substitute image of the photography restriction element is arranged by the background image. Referring to FIG. 8, the original image 800 a includes a user B and a user C. Here, it is assumed that the user C is a photographing restriction element. Therefore, the user C is arranged by the background image.

The user C uses the background image to arrange by deleting the user C from the original image 800a and replacing the deleted user C area with an image similar to the background image surrounding the user C.

Referring to the corrected image 800b of FIG. 8, the area inside the edge 3-1 of the user C is filled with the same or similar pixel values as the pixels outside the edge 3-1. For example, pixels 802 inside and adjacent to edge 3-1 are set to have the same or similar colors, saturation, and brightness as pixels 801 outside and adjacent to edge 3-1. In addition, the pixels (from among the pixels inside the edge 3-1) of the neighboring pixels 802 are set to have colors, saturation, and brightness similar to those of the pixels 802. Meanwhile, since any one of various techniques commonly used in the prior art can be applied to a method of arranging a specific area of an image by a background image, details thereof are not provided here.

In FIG. 9, the substitute image of the photography restriction element is arranged by a pre-stored background image. Referring to FIG. 9, the original image 900 a includes a user B, a user C, and a structure 901. Here, it is assumed that the user C is a photographing restriction element. In the original image 900a, the user C and the structure 902 partially overlap each other. Therefore, the area of the user C deleted by the user C from the original image 900 a and the deleted user C is filled with the pre-stored image of the structure 902. Referring to the corrected image 900b, the user C's overlapping the structure 902 The area is replaced by a pre-stored image of structure 902. Here, all the images of the structure 902 may be pre-stored in a user device or a server of the modified original image 900a.

In FIG. 10, the replacement of the photography restriction element is shown in FIG. 10, and the replacement image of the photography restriction element is arranged by pre-storing the replacement image 1001. Referring to FIG. 10, the original image 1000 a includes a user 2 and a user 3. Here, it is assumed that the user 3 is a photographing restriction element. The corrected image 1000b is generated by replacing the user 3 with a pre-stored replacement image 1001. In FIG. 10, a pre-stored replacement image 1001 is selected to replace the face of the user C. However, instead, the pre-stored replacement image 1001 may be an image for replacing the user C.

When a substitute image of an individual's face (that is, a substitute image of a photography restriction element) is generated from the substitute image, the substitute image can be edited or corrected based on the direction and angle of the face in the original image. Figure 12 describes this in detail.

Referring to FIG. 11, the face of the individual in the original image 1100 a is facing forward. Therefore, as shown in the modified image 1100b, the individual's face is arranged by facing the positive alternative image 1110 according to the direction and angle of the individual's face in the original image 1100a.

Referring to FIG. 12, the face of the individual in the original image 1200a is facing sideways. Therefore, as shown in the modified image 1200b, the individual's face is arranged by facing the lateral alternative image 1210 according to the direction and angle of the individual's face in the original image 1200a.

Similarly, an individual's face can be arranged by an alternative image according to the direction and angle of the individual's face, and thus a natural image can be obtained.

13 to 16 are flowcharts of a method of correcting an image including a photography-restricted individual according to an exemplary embodiment.

Referring to FIG. 13, in operation 1301, a user device captures an image including a person. Here, the user device may be any of a general photography device (such as a video camera or camcorder), a wearable device (such as smart glasses or a smart watch), and a mobile device (such as a smart phone or tablet PC). One, or any other various device capable of photographing.

In operation 1302, the user device performs facial recognition on the person in the image. In detail, facial recognition can be performed by detecting a facial area from an image and extracting features from the detected facial area.

In operation 1303, the user device determines the relationship between the photographer (ie, the user of the user device) and the subject (ie, the individual in the image) by using the data stored in the user device. . In other words, the user device determines whether the photographer and the subject know each other.

In operation 1304, the user device determines whether the image contains individuals that are not related to the photographer. At this time, a method of determining a relationship between a photographer and a subject and determining whether or not an individual who is not related to the photographer is included in the image, and some exemplary embodiments will be described below with reference to FIGS. 14 to 16.

If it is determined in operation 1304 that an individual irrelevant to the photographer is included in the image, the user device performs protection processing on an individual irrelevant to the photographer in operation 1305. In other words, the user device corrects the image so that individuals unrelated to the photographer are indistinguishable. The method of correcting an image has been described above with reference to FIGS. 7 to 12.

Referring to FIG. 14, in operation 1401, the smart phone captures an image including a person.

In operation 1402, the smartphone performs facial recognition on the person included in the image. In detail, facial recognition can be performed by detecting a facial area from an image and extracting features from the detected facial area.

In operation 1403, the smart phone determines whether the individual is in an address book of the smart phone. In detail, the smartphone determines whether the individual matches at least one of the photos included in the address book. If it is determined that the individual is not in the address book, then operation 1404 is performed.

In operation 1404, the smartphone performs a mosaic process on the individual. Alternatively, the smartphone may perform blur processing or background image orchestration processing on an individual.

Referring to FIG. 15, in operation 1501, the smart phone captures an image including a person.

In operation 1502, the smartphone performs facial recognition on the person included in the image. In detail, facial recognition can be performed by detecting a facial area from an image and extracting features from the detected facial area.

In operation 1503, the smart phone determines whether the individual is included in a photo stored in an album of the smart phone. In detail, the smartphone determines whether an individual is included in at least one of the photos included in the album. If it is determined that the individual is not included in the at least one photo, operation 1504 is performed.

In operation 1504, the smart phone performs a mosaic process on the individual. Alternatively, the smartphone can perform blurring or background image editing on the individual Management.

Referring to FIG. 16, in operation 1401, the smart phone captures an image including a person.

In operation 1602, the smartphone performs facial recognition on the person included in the image. In detail, facial recognition can be performed by detecting a facial area from an image and extracting features from the detected facial area.

In operation 1603, the smart phone analyzes the SNS usage record of the photographer (ie, the user of the smart phone). The smart phone can store various records of users using the smart phone, and can analyze the SNS usage records.

In operation 1604, the smart phone determines whether the individual is registered as a friend of the photographer on the SNS. If it is determined that the individual is not registered as a friend of the photographer on the SNS, operation 1605 is performed.

In operation 1605, the smart phone performs a mosaic process on the individual. Alternatively, the smartphone may perform blur processing or background image orchestration processing on an individual.

Meanwhile, the smart phone is used as a user device in the method of FIGS. 14 to 16, but any one of various devices capable of capturing an image and storing an address book may be used instead.

17 and 18 are diagrams for describing a method of correcting an image including a photographing restriction element by using at least two user devices according to an exemplary embodiment. In detail, in the methods of FIG. 17 and FIG. 18, photography and image processing are performed by different user devices.

Referring to FIG. 17, the user A can use the wisdom of being a wearable device by The type glasses 100 captures an image including an individual. Here, the image includes user B and user C. The smart glasses 100 transmit the image to the smart phone 100-2, which is also the device of the user A. After receiving the image, the smart phone 100-2 determines whether the image includes a photography restriction element. If it is determined that the photography restriction element is included in the image, the smartphone 100-2 corrects the image.

Referring to FIG. 18, the smartphone 100-2 performs facial recognition on people included in the original image 1800 a received from the smart glasses 100. After recognizing the faces of the user B and the user C included in the original image 1800a, the smartphone 100-2 compares the photos of the recognized faces in the address book stored in the smartphone 100-2. The address book stored in the smartphone 100-2 can store people's photos and people's names and phone numbers.

In FIG. 18, if it is determined that the photo of the user B is in the address book by comparing the recognized face with the photo in the address book, the smart phone 100-2 determines that the user B is a photography-allowed individual. At the same time, when a photo of the user C is not found in the address book, the smart phone 100-2 determines that the user C is a photographing restricted individual. Therefore, the smartphone 100-2 determines that the image includes a photographing restriction element and corrects the image. For example, the smartphone 100-2 may use the background image to output a corrected image 1800b by the user C who is arranged as a photography-restricted individual. The method of arranging the user C from the background image has been described in detail above with reference to FIG. 8.

Alternatively, the smartphone 100-2 may compare a face recognized from the original image 1800a with a photo stored in an album stored in the smartphone 100-2 or analyze an SNS usage record of the user A To determine photography restricted individuals. The method has been described above with reference to FIGS. 5 and 6.

In addition, in FIG. 18, the smart phone 100-2 arranges the user C with a background image, but alternatively, it can perform mosaic processing or blurring processing on the user C (3) or by using an alternative image arrangement User C corrects the original image 1800a, as described above with reference to FIGS. 7 to 12.

19 is a flowchart of a method of correcting an image including a photographing restriction element by using at least two user devices according to an exemplary embodiment.

Referring to FIG. 19, in operation 1901, a wearable device captures an image including a person. Here, the wearable device (smart glasses or smart watch) may have a photographing function.

In operation 1902, the wearable device transmits an image to the smartphone. Here, the image is transmitted to a smartphone owned by the user of the wearable device, and a device such as a tablet PC can be used instead of the smartphone.

In operation 1903, the smartphone performs facial recognition on the person included in the image. In detail, facial recognition is performed by detecting a facial area from an image and extracting features from the detected facial area.

In operation 1904, the smart phone determines whether the individual is in an address book of the smart phone. In other words, the smartphone compares the individual with a photo stored in the address book to determine whether the individual is included in at least one of the photos.

If it is determined that the individual is not in the address book, the substitute image of the individual is arranged by the background image in operation 1905.

20 to 22 are diagrams for describing a method of correcting an image including a photography restriction element via communication with a subject's device according to an exemplary embodiment. In detail, according to the method of FIG. 20 to FIG. 22, whether the device of the photographer will allow the camera The video response is transmitted to the photographer's device.

Referring to FIG. 20, the smartphone 100-2 of the user A captures an image including the user B. When capturing an image, the smartphone 100-2 transmits a photography notification to the smartphone 200 of the user B who is the subject. Here, the photography notification may include identification (ID) information identifying the user A.

After receiving the photography notification, the smart phone 200 determines that the photographer is the user A by using the ID information included in the photography notification, and determines whether the photographer (user A) is allowed to capture the image of the user B. A method of identifying a photographer by using ID information included in a photography notification and determining whether photography is permitted will be described in detail below with reference to FIG. 21.

After determining whether photography is permitted, the smart phone 200 transmits a response to the smart phone 100-2. The smartphone 100-2 corrects the image based on the response or stores the image without correction.

In FIG. 21, the smartphone 100-2 of the user A transmits a photography notification to the smartphone 200 of the user B, and the smartphone 200 determines whether photography is permitted and transmits a response to the smartphone 100-2.

Referring to FIG. 21, the smart phone 100-2 transmits a photographing notification including ID information of the user A to the smart phone 200 while photographing the user B. Here, the ID information included in the photography notification may include the name and photo of the user A.

The smartphone 200 determines whether or not the user A is allowed to take pictures based on the ID information of the user A included in the photographing notification received from the smart phone 100-2. In detail, the smart phone 200 compares the photo of the user A included in the ID information with the photo stored in the address book, or the user included in the ID information A's name is compared with the name stored in the address book. If it is determined that the photo or name of the user A is not in the address book, the smart phone 200 does not allow photography. On the other hand, if the photo or name of the user A is in the address book, the smart phone 200 allows photography.

Alternatively, the smart phone 200 may compare the photo of the user A included in the ID information with the photo stored in the album of the smart phone 200 or analyze the SNS usage record of the user B Determine whether photography is allowed.

In FIG. 21, if the photo of the user A is not in the contact list of the smart phone 200, the smart phone 200 determines that photography is not allowed and transmits a photography prohibition message to the smart phone 100-2. At this time, the smart phone 200 transmits the photo of the user B together with the photography prohibition message to the smart phone 100-2.

Subsequently, referring to FIG. 22, the smartphone 100-2 performs facial recognition on the original image 1600a. After performing face recognition, the smartphone 100-2 compares the photo of the user B received from the smartphone 200 with the photographing prohibition message with the face recognized in the original image 1600a. If the recognized face matches the photo of the user B, the smartphone 100-2 performs a mosaic process on the face area of the user B in the original image 1600a and outputs a corrected image 1600b. Alternatively, the smartphone 100-2 may be corrected by performing blur processing on the user B, by orchestrating the substitute image and background image of the user B, or by orchestrating the substitute image and the substitute image of user B Original image 1600a.

23 and 24 are flowcharts of a method of correcting an image including a photographing restriction element via communication with a device of a subject according to an exemplary embodiment.

Referring to FIG. 23, in operation 2301, a photographer's device will contain a photographer The ID notification of the shooting notification is transmitted to the device of the photographer. For example, the photographer's device can search for devices located within a certain range during the photography, and transmit photography notifications to the found devices. The device can be searched for and transmitted through the use of near field communication (such as Wi-Fidirect or Bluetooth), and may require approval of the photographer's device for transmitting the notification of the photography.

In operation 2302, the photographer's device determines whether photography is permitted based on the photographer's ID information included in the photography notification.

If it is determined in operation 2303 that photography is permitted, operation 2307 is performed. In operation 2307, the photographer's device transmits a photography permission message to the photographer's device. Subsequently, the photographer's device determines that the captured image does not include a photography restriction element and stores the captured image without correction.

If it is determined in operation 2303 that photography is not allowed, then operation 2304 is performed. In operation 2304, the photographer's device transmits a photographing prohibition message to the photographer's device along with the photograph of the photographer.

In operation 2305, the photographer's device performs facial recognition on people included in the captured image.

In operation 2306, the photographer's device compares the face identified from the captured image with the photograph of the photographer, and performs protection processing on the face of the photograph that matches the photographer. In other words, a mosaic process, a blur process, an arrangement process by a background image, or an arrangement process by an alternative image is performed on a face matching a photograph of a photographer.

FIG. 24 is a flowchart of a method for determining whether or not photography is permitted by using an address book of a subject's device.

Referring to FIG. 24, in operation 2401, the photographer's smart phone transmits a photography notification including the photographer's ID information to the photographer's smart phone.

In operation 2402, the photographer's smart phone determines whether the photographer is in the address book by using the photographer's ID information. In detail, the name or photo of the photographer included in the ID information is compared with the name or photo stored in the address book. Alternatively, the photographer's smartphone can be determined by comparing the photographer's ID information with an album stored in the photographer's smartphone, or by analyzing the photographer's SNS usage records Whether photography is allowed.

If it is determined in operation 2402 that the photographer is in the address book, it is determined in operation 2406 that photography is permitted and the photographer's smart phone transmits a photography permission message to the photographer's smart phone.

If it is determined in operation 2402 that the photographer is not in the address book, it is determined that photography is not allowed and operation 2403 is performed.

In operation 2403, the photographer's smart phone transmits the photography prohibition message to the photographer's smart phone along with the photograph of the photographer.

In operation 2404, the photographer's smartphone performs facial recognition on people included in the captured image.

In operation 2405, the photographer's smart phone compares the recognized face with the photograph of the photographer and performs a protection process on the face matching the photograph of the photographer. In other words, a mosaic process, a blur process, an arrangement process by a background image, or an arrangement process by an alternative image is performed on a face matching a photograph of a photographer.

FIG. 25 and FIG. 26 are diagrams for describing by using the exemplary embodiment A diagram of a method in which a server corrects an image including a photographing restriction element.

Referring to FIG. 25, the smart phone 100-2 of the user A captures the original image including the user B and the user C, and transmits the original image to the server 300 for image processing. The server 300 determines whether the photography restriction element is included in the original image, and if it is determined that the photography restriction element is included, the original image is corrected and the corrected image is transmitted to the smartphone 100-2 of the user A. A method of determining whether the photographing restriction element is included in the image by the server 300 will now be described in detail with reference to FIG. 26.

Referring to FIG. 26, the server 300 performs face recognition on the original image 2600 a received from the smart phone 100-2 of the user A to recognize the faces of the user B and the user C. The server 300 determines whether the user B and the user C have allowed the user A to take an image of themselves by referring to the privacy protection information 2610 pre-stored in the server 300.

Here, the privacy protection information 2610 is about at least one user that defines another user who is allowed to shoot. Referring to FIG. 26, another user who is allowed to take a picture is mapped according to the user in the privacy protection information 2610. For example, user A does not allow anyone to photograph himself / herself. In addition, the user B only allows the user A to photograph himself / herself, and the user C allows the user B and the user D to photograph himself / herself.

The server 300 compares the user B and the user C identified from the original image 2600a with the privacy protection information 2610 to determine whether photography is permitted. According to the privacy protection information 2610, since the user B allows the user A to take an image of himself / herself and the user C does not allow it, the server 300 determines that the user C is a photography restriction element.

Meanwhile, in order to determine whether photography is allowed, the server 300 needs to check whether the photographer of the original image 2600a is the user A. Therefore, the smart phone 100-2 of the user A can transmit the ID information of the user A to the server 300 along with the original image 2600a.

After determining that the user C is a photographing restriction element, the server 300 generates a corrected image 2600b by performing mosaic processing on the user C in the original image 2600a, and transmits the corrected image 2600b to the wisdom of the user A Phone 100-2. Alternatively, the server 300 may modify the original image 2600a by performing a blurring process, a layout process by a background image, or a layout process by a substitute image.

FIG. 27 is a flowchart of a method of correcting an image including a photographing restriction element by using a server according to an exemplary embodiment.

Referring to FIG. 27, in operation 2701, a user device captures an image.

In operation 2702, the user device transmits an image to a server and requests the server to perform image processing.

In operation 2703, the server performs facial recognition on an individual included in the image received from the user device.

In operation 2704, the server determines whether the identified individual is allowed to be photographed by referring to the pre-stored privacy protection information. In other words, the server determines whether the identified individual is allowed to be photographed by the photographer of the video. The method of determining whether photography is permitted by using privacy protection information has been described above with reference to FIG. 26.

If it is determined in operation 2705 that the identified individual is not allowed to be photographed, the server performs a protection process on the identified individual in operation 2706. In operation 2707 During transmission, the server transmits the corrected image to the user device.

If it is determined in operation 2705 that the identified individual is allowed to be photographed, the server transmits the image to the user device without correction in operation 2707.

28 to 31 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server according to other exemplary embodiments. In detail, according to the method of FIG. 28 to FIG. 31, before transmitting the original image to the server, the user device that performs photography encodes the original image by using an encryption key managed by the server, and therefore the photographer cannot Access the raw image processed by the server.

Referring to FIG. 28, the smart phone 100-2 of the user A receives the encryption key from the server 300. Here, the encryption key is managed by the server 300. The smart phone 100-2 of the user A captures an image including the user B and the user C, and encodes the image by using an encryption key received from the server 300. Here, since the smartphone 100-2 of the user A encodes the image immediately after capturing the image, the user A cannot access the image.

As shown in FIG. 29, the smartphone 100-2 of the user A transmits the encoded image to the server 300.

Referring to FIG. 30, the server 300 decodes the encoded image. Since the encryption key used to encode the encoded image is managed by the server 300, the server 300 can use the encryption key to decode the encoded image. The server 300 recognizes the faces of the user B and the user C by performing facial recognition on the decoded image. After performing facial recognition, the server 300 determines whether the user B and the user C are allowed to be photographed by using the privacy protection information 3010 pre-stored in the server 300. The method for determining whether photography is permitted by using privacy protection information has been described above with reference to FIG. 26 law.

If it is determined based on the privacy protection information 3010 that the user C is not allowed to be photographed, the server 300 may correct the decoded image by performing a mosaic process on the user C.

Subsequently, the server 300 transmits the corrected image to the smart phone 100-2 of the user A, as shown in FIG. 31 in the figure.

FIG. 32 is a flowchart of a method of correcting an image including a photographing restriction element by using a server according to another exemplary embodiment.

Referring to FIG. 32, in operation 3201, a user device captures an image including a person.

In operation 3202, the user device encodes the image by using an encryption key received from the server. Here, the encryption key is managed by the server.

In operation 3203, the user device transmits the encoded image to the server.

In operation 3204, the server decodes the encoded image received from the user device and performs facial recognition to identify individuals included in the decoded image.

In operation 3205, the server determines whether the identified individual is allowed to be photographed by using the privacy protection information pre-stored in the server. The method of determining photography by referring to privacy protection information has been described above with reference to FIG. 26.

In operation 3206, the server performs a protection process on the individual who is not allowed to be photographed. For example, the server may perform mosaic processing, blur processing, orchestration processing by background images, or by orchestration processing of substitute images on individuals who are not allowed to be photographed.

In operation 3207, the server transmits the corrected image to the user device.

33 to 37 are diagrams for describing a method of correcting an image including a photography-restricted object according to an exemplary embodiment.

Referring to FIG. 33, the smartphone 100-2 of the user A captures an image including the user B and the statue 20. The smart phone 100-2 of the user A transmits the image to the server 300 to request the server 300 to perform image processing. The server 300 determines whether the image includes a photography restriction element, and if it includes a photography restriction element, corrects the image and transmits the corrected image to the smart phone 100-2 of the user A. Hereinafter, a method of determining whether the photography restriction element is included in the image and correcting the image by the server 300 will be described with reference to FIGS. 34 to 37.

Referring to FIG. 34, the server 300 recognizes an object included in the original image 3400a. In other words, the statue 20 is identified by extracting features (such as shape and color) of the statue 20 included in the original image 3400a. The server 300 determines whether the statue is in a photographing restricted object database stored in the server 300. The photography restricted object database stores photos of photography restricted objects.

If the statue 20 is in the photography restriction object database, the server 300 determines that the statue 20 is a photography restriction element. The server 300 performs a mosaic process on the statue 20 in the original image 3400a to generate a corrected image 3400b, and transmits the corrected image 3400b to the smart phone 100-2 of the user A.

Meanwhile, in FIGS. 35 to 37, images are corrected by using different methods. A method for identifying objects in an image and determining whether a photography-restricted element is included in the image by using a pre-stored photography-restricted object database, and the method described with reference to FIG. 34 Same method. However, in FIG. 35, when the image is corrected, the corrected image 3500b is generated by performing a blurring process. In FIG. 36, the corrected image 3600b is generated by performing an orchestration process with a background image, as described above with reference to FIG. 8. In FIG. 37, the corrected image 3700b is generated by arranging the replacement image 3710.

FIG. 38 is a flowchart of a method of correcting an image including a photographing restricted object according to an exemplary embodiment.

Referring to FIG. 38, in operation 3801, a user device captures an image.

In operation 3802, the user device transmits an image to a server.

In operation 3803, the server recognizes an object included in the image. In detail, objects included in the image are detected and features (such as shape and color) of the detected objects are extracted.

In operation 3804, the server determines whether the identified object is included in a photographing restricted object database pre-stored in the server. The photography restricted object database stores a list of photography restricted objects.

If it is determined in operation 3804 that the identified object is in the photographing restricted object database, the server corrects the image by performing protection processing on the identified object in operation 3805, and corrects the corrected image in operation 3806. Transfer to user device.

If it is determined in operation 3804 that the identified object is not in the photographing restricted object database, then in operation 3806, the server transmits the image to the user device without correction.

39 to 41 are diagrams for describing a method of correcting an image captured at a photographing restriction position according to an exemplary embodiment.

Referring to FIG. 39, the smart phone 100-2 of the user A captures an image of a venue where photography is restricted, and transmits the image to the server 300 to request the server 300 to perform image processing. At this time, the smart phone 100-2 transmits the location information together with the image to the server 300. The location information is information about a location where an image is captured by the smart phone 100-2, wherein the smart phone 100-2 determines a location by using a GPS embedded therein and transmits the determined location to the server 300. Alternatively, the smartphone 100-2 may generate the location via any of a variety of methods (e.g., by using a base station or an access point (AP) to determine the location connected to the smartphone 100-2) to generate a location Information.

The server 300 determines whether the image includes a photographing restriction element by using the position information. A method for determining whether the image includes a photographing restriction element by the server 300 will be described in detail below with reference to FIGS. 40 and 41.

If it is determined that the image contains a photography restriction element, the server 300 corrects the image and transmits the corrected image to the smart phone 100-2.

Referring to FIG. 40, the server 300 determines the position of the captured image by analyzing the position information received from the smart phone 100-2, and determines whether the determined position is a photographing restricted position. In other words, the server 300 determines whether the determined position is in a photographing restricted position database pre-stored in the server 300. Here, the photography restricted location database can store information about locations that are restricted from being photographed by unauthorized individuals, such as concert venues, museums, and military security areas. When the server 300 determines that the determined position is a position stored in the database, the server 300 may determine whether the database stores a name of a position that exactly matches the name of the determined position. Alternatively, the server 300 may evaluate the similarity between the determined location and the location stored in the database And determine whether the determined position is in the database based on the degree of similarity.

If it is determined that the determined position is in the photographing restriction position database, the server 300 judges that the image includes a photographing restriction element and corrects the image. In FIG. 40, the server 300 performs a mosaic process on the original image 4000a to generate a corrected image 4000b.

Referring to FIG. 41, the server 300 determines a position by analyzing the position information received from the smart phone 100-2, and determines whether the position is a photographing restricted position. In other words, the server 300 determines whether the determined position is in a photographing restricted position database pre-stored in the server 300. Here, the photography restricted location database can store information about locations that are restricted from being photographed by unauthorized individuals, such as concert venues, museums, and military security areas.

If it is determined that the determined position is in the photographing restriction position database, the server 300 judges that the image includes a photographing restriction element and corrects the image. In FIG. 41, the server 300 generates a modified image 4100b by orchestrating the original image 4100a using a substitute image. Alternatively, the server 300 may correct the original image 4100a by performing a blurring process on the original image 4100a.

Meanwhile, the server 300 corrects the images in FIG. 39 to FIG. 41, but instead, the server 300 can only determine whether the determined position is a photographing restricted position by analyzing the position information received from the smart phone 100-2, and The result of the determination is transmitted to the smartphone 100-2. The smartphone 100-2 may then correct the image based on the received results.

FIG. 42 is a flowchart of a method of correcting an image captured at a photographing restricted position according to an exemplary embodiment.

Referring to FIG. 42, in operation 4201, a user device captures an image. Here, the user device may be a general photography device (such as a video camera or camcorder), a wearable device (such as smart glasses or a smart watch), or a mobile device (such as a smart phone or tablet PC), or Any other device capable of photographing.

In operation 4202, the user device determines the location of the captured image, and transmits the location information to the server. Here, the user device can determine the location by using the GPS embedded therein. Alternatively, the user device may determine the position by using the position of a base station or an AP connected to the user device.

In operation 4203, the server determines whether the determined position is a photographing restricted position. In detail, the server determines the position by analyzing the received position information, and the determined position is compared with a photographing restricted position database pre-stored in the server. Here, the photography restricted location database can store information about locations that are restricted from being photographed by unauthorized individuals, such as concert venues, museums, and military security areas.

If the determined position is in the photographing restriction position database, the server judges that the image contains photographing restriction elements and performs operation 4204. In operation 4204, the server corrects the image and transmits the corrected image to the user device. In other words, the server generates a corrected image by performing mosaic processing, blurring processing, or by orchestration processing of an alternative image, and transmits the corrected image to the user device.

43 to 45 are diagrams for describing a method of correcting an image captured at a photographing restriction position according to other exemplary embodiments. In detail, in FIG. 43 to FIG. 45, the user device determines that the position of the captured image is the photographing restriction position by recognizing the photographing prohibition flag.

Referring to FIG. 43, the smart phone 100-2 of the user A captures including a prohibition Photography of the logo 30 and the user B. Here, the photography prohibition sign 30 may include a symbol and phrase indicating photography prohibition as shown in FIG. 43, or may be in any other form. For example, the photography prohibition sign 30 may include a bar code, a quick response (QR) code, or a color code indicating that photography is prohibited. When the photo contains a barcode, QR code or color code that the photographer may not understand but indicates that photography is prohibited, the smartphone 100-2 prevents the photographer from accidentally or intentionally capturing the photo.

Referring to FIG. 44, the smartphone 100-2 of the user A recognizes a photographing prohibition mark 30 included in the original image 4400 a. In detail, the smartphone 100-2 recognizes at least one of the symbol 30-1 and the phrase 30-2 included in the photography prohibition sign 30. After recognizing the photography prohibition mark 30, the smartphone 100-2 determines that the image is captured at the photography restriction position. In other words, the smartphone 100-2 determines that the image contains a photographing restriction element. Therefore, the smart phone 100-2 performs a mosaic process on the original image 4400a to generate a corrected image 4400b.

Referring to FIG. 45, the smartphone 100-2 of the user A recognizes a photographing prohibition mark 30 included in the original image 4500a. In detail, the smartphone 100-2 recognizes at least one of the symbol 30-1 and the phrase 30-2 included in the photography prohibition sign 30. After recognizing the photography prohibition mark 30, the smartphone 100-2 determines that the image is captured at the photography restriction position. In other words, the smartphone 100-2 determines that the image contains a photographing restriction element. Therefore, the smartphone 100-2 generates a corrected image 4500b by using the replacement image to arrange the original image 4500a.

FIG. 46 is a flowchart of a method of correcting an image captured at a photographing restriction position according to another exemplary embodiment.

Referring to FIG. 46, in operation 4601, a user device captures an image.

In operation 4602, the user device recognizes an object included in the image.

In operation 4603, the user device determines whether the recognized object includes a photography prohibition flag. If it is determined that the recognized object includes a photography prohibition mark, the user device corrects the image in operation 4604. Here, the user device can correct the image by performing a mosaic process, a blur process on the image, or by an orchestration process instead of the image.

47 to 49 are diagrams for describing an example of restricting photography of an individual according to privacy protection information according to an exemplary embodiment.

Referring to FIG. 47, the smartphone 100-2 of the user A captures an image including the user B and the user C. Subsequently, the smart phone 100-2 transmits the image to the server 300 and requests the server 300 to perform image processing. At this time, the smart phone 100-2 may encode the image by using the encryption key received from the server 300, and transmit the encoded image to the server 300.

The server 300 determines whether the image contains a photographing restriction element by using the privacy protection information 4710 stored in the server 300 in advance. When the image is encoded, the server 300 decodes the encoded image by using an encryption key before determining whether the photography restriction element is included in the image. Here, the photography restriction time and location are mapped according to the user in the privacy protection information 4710.

A method for determining whether an image includes a photographing restriction element by using privacy protection information will be described in detail later with reference to FIGS. 48 and 49.

If it is determined that the image includes a photographing restriction element, the server 300 corrects the image and transmits the corrected image to the smart phone 100-2.

The use of privacy protection will be described in detail below with reference to FIGS. 48 and 49. Information 4710 A method for determining whether an image includes a photographing restriction element and correcting the image. The method is executed by the server 300.

Referring to FIG. 48, the smartphone 100-2 transmits the encoded image to the server 300. Here, the smart phone 100-2 also transmits position information about the position where the image was captured to the server 300 along with the encoded image. The smart phone 100-2 determines the position by using the GPS embedded therein or the position of the base station or AP connected thereto, and can transmit the position information to the server 300. In FIG. 48, the smartphone 100-2 transmits "school" as the location information to the server 300.

Referring to FIG. 49, the server 300 decodes the encoded image and performs facial recognition on the decoded image to recognize the faces of the user B and the user C. The server 300 determines the photographing restriction time and position of the recognized face based on the privacy protection information 4910. The privacy protection information 4910 stores at least one of the photography-restricted time and location according to the user.

Based on the privacy protection information 4910 of FIG. 49, the user A is not allowed to take pictures from 24:00 to 06:00 and is not allowed to take pictures at home. User B does not allow photography from 22:00 to 09:00 and has not set a photography restriction position. User C has not set the shooting position and time and is not allowed to take pictures at school.

The server 300 determines whether the image includes a photographing restriction element based on the location information (school) and the capture time from the smartphone 100-2 of the user A. Therefore, the server 300 can receive the capture time of the captured image and receive the image from the smart phone 100-2 at the same time.

In FIG. 49, the server 300 determines that the location time of the captured image is "school" based on the location information and the capture time received from the smartphone 100-2. "21:00".

The server 300 determines that the photography of the user B is restricted from 22:00 to 09:00 by referring to the privacy protection information 4910, but the location is not limited. Since the capture time “21:00” determined by the server 300 is not within the photography limitation time of the user B, it is determined that the photography of the user B is not limited. In other words, the server 300 determines that the user B is not an imaging restriction element.

At the same time, the server 300 judges that the photography of the user C is restricted at the "school" by referring to the privacy protection information 4910, but the time is not limited. Since the position information ("school") determined by the server 300 is within the photography restricted position of the user C, it is determined that the photography of the user C is restricted. In other words, the server 300 determines that the user C is a photographing restriction element.

Therefore, the server 300 corrects the image by performing mosaic processing only on the user C.

FIG. 50 is a flowchart of a method of limiting photography of an individual according to privacy protection information according to an exemplary embodiment.

Referring to FIG. 50, in operation 5001, a user device captures an image including a person. Here, the user device may be any of a general photography device (such as a video camera or camcorder), a wearable device (such as smart glasses or a smart watch), and a mobile device (such as a smart phone or tablet PC). One of them may be any of various devices capable of taking pictures.

In operation 5002, the user device encodes the image by using an encryption key received from the server. Here, the encryption key is managed by the server.

In operation 5003, the user device loads the encoded image and user equipment The configured location information is transmitted to the server.

In operation 5004, the server decodes the encoded image and performs facial recognition on the individuals included in the decoded image. In detail, facial recognition may be performed by detecting a facial area from the decoded image and extracting features from the detected facial area.

In operation 5005, the server determines whether the individual is allowed to be photographed by using the privacy protection information pre-stored in the server, the location information, and the capture time of the captured image received from the user device.

In operation 5006, the server performs a protection process on an individual who is not allowed to be photographed. In other words, the server can correct the decoded image by performing at least one of mosaic processing, blurring processing, layout processing with background images, and layout processing with replacement images, so that the person being photographed is not allowed to be unrecognizable Unrecognizable or unrecognizable.

In operation 5007, the server transmits the corrected image to the user device.

FIG. 51 is a block diagram of a user device 1000 according to an exemplary embodiment. Referring to FIG. 51, a user device 1000 according to an exemplary embodiment may include a user interface 1100, a camera 1200, a storage 1300, a communication interface 1400, and a controller 1500. In addition, the controller 1500 may include an image analyzer 1510 and an image corrector 1520.

Meanwhile, as described above, the user device 1000 according to the exemplary embodiment may be any of various devices capable of shooting, such as smart glasses, a smart watch, a smart phone, a tablet PC, a video camera, or a video camera. Movie machine.

User interface 1100 receives input from the user and displays the screen to the user Also, it can be configured differently according to the type of the user device 1000. For example, when the user device 1000 is a smart phone, the user interface 1100 may include a touch screen.

The camera 1200 may include a camera module, a smart watch, a smart phone, or a tablet PC embedded in the smart glasses. The camera 1200 may capture an image, such as a still image or a moving image, and transmit the image to the controller 1500.

The memory 1300 may include memory for storing various fragments of images or data. For example, the storage 1300 may store an address book, a photo, a moving image, or a usage record of the user device 1000, or may store an image modified by the controller 1500.

The communication interface 1400 can perform data communication with an external device or server via a wire or wirelessly.

The controller 1500 controls operations of components included in the user device 1000. In detail, the controller 1500 determines whether the image includes a photography restriction element by analyzing the image, and corrects the image if it is determined that the image includes a photography restriction element.

The image analyzer 1510 can identify an individual included in an image, an object contained in the image, or a position where the image is captured by analyzing the image, and based on the relationship between the user of the user device 1000 and the identified individual, the identified At least one of the object and the identified position determines whether the image needs to be corrected. In detail, the image analyzer 1510 can determine whether the user and the identified individual are related to each other by using the data stored in the storage 1300, and if it is determined that the user and the identified individual are not related to each other, the image needs to be determined Was fixed.

For example, the image analyzer 1510 may determine whether the identified individual is The address book stored in the storage 1300, and if the identified individual is not in the address book, it can be determined that the user and the identified individual are not related to each other. The image analyzer 1510 may perform content-based image capture based on the similarity of the image content to determine whether the identified individual is in the address book.

Alternatively, the image analyzer 1510 may determine whether the identified individual is included in at least one photo stored in the storage 1300, and if the identified individual is not included in at least one photo, the user and the identified person may be determined Of individuals are not related to each other.

Alternatively, the image analyzer 1510 may determine whether the user and the identified individual are related to each other by analyzing the user's SNS usage record using the data stored in the storage 1300.

Alternatively, the image 1510 may determine whether the identified object or the identified position is restricted from being photographed by accessing a database of an external server through the communication interface 1400, and if it is determined that the identified object or the identified position is restricted from being photographed When shooting, the image analyzer 1510 can determine that the image needs to be corrected.

Alternatively, if the photography prohibition mark is recognized in the image, the image analyzer 1510 may determine that the image needs to be corrected by determining that the recognized position is a photography restricted position.

At the same time, when the image analyzer 1510 determines that the image needs to be corrected, the image corrector 1520 corrects the image. The image corrector 1520 may perform at least one of a mosaic process, a blur process, a layout process by a background image, and a layout process by a substitute image on the identified individual, the recognized object, or the image. The method of correcting an image by the image corrector 1520 has been described above with reference to FIGS. 7 to 12.

Meanwhile, the components of the user device 1000 may perform operations for performing the method described above with reference to FIGS. 1 to 50, even if not described with reference to FIG. 51.

At the same time, the time point for correcting the captured image can be set differently. For example, the captured image can be corrected before being stored on the user device. Alternatively, the captured image may be stored in a user device and then corrected when transmitted to an external server or another user device. Alternatively, the captured image may not be stored in the user device that captured the captured image, but may be immediately modified and transmitted to an external server or another user device. Hereinafter, different points in time of correcting a captured image according to an exemplary embodiment will be described.

FIG. 52 is a diagram illustrating an example of a method for correcting a captured image before the captured image is stored in a user device according to an exemplary embodiment.

In FIG. 52, the smartphone 100-2 on the left displays multiple images on the screen when the gallery application is executed, and when the user touches and selects the image 5210 from the multiple images, the image 5210 is displayed as the right The enlarged image 5220 shown in the smartphone 100-2.

An image 5210 selected from a plurality of images includes a photographing-restricted person, and thus a mosaic process is performed on a region 5211 corresponding to the face of the photographing-restricted person. Mosaic processing is also performed on the area 5221 of the enlarged image 5220, and the area 5221 corresponds to the face of an individual whose photography is restricted.

The image is captured by the smart phone 100-2, and before the captured image is stored in the smart phone 100-2, it is determined whether the captured image includes a photographing restriction element. When the captured image contains a photography restriction element, the captured image Amended and then stored in smartphone 100-2. In other words, the original image of the captured image is not stored in the smartphone 100-2, and therefore only the corrected captured image can be viewed through the smartphone 100-2. Therefore, according to the current embodiment, the user of the smart phone 100-2 can only access only the corrected captured image.

The method of determining whether an image includes a photographing restriction element and correcting the image has been described in detail above.

FIG. 53 is a flowchart describing an example of modifying a captured image before storing the captured image in a user device according to a method for modifying the captured image according to an exemplary embodiment.

Referring to FIG. 53, in operation 5301, a user device captures an image. Here, the user device may be any device capable of photographing, such as a smart phone, a video camera, or a tablet PC.

In operation 5302, the user device recognizes at least one of a person included in the image, an object included in the image, and a location where the image is captured. The method of identifying a person, object, or location has been described above with reference to FIG. 2.

In operation 5303, the user device determines whether the image includes a photographing restriction element based on at least one of the relationship between the identified individual and the user of the user device, the identified object, and the identified position. The method of determining whether an image includes a photographing restriction element has been described above with reference to FIG. 2.

When it is determined in operation 5304 that the image contains a photographing restriction element, the user device corrects the image in operation 5305 and stores the image to be corrected. When it is determined in operation 5304 that the image does not include a photographing restriction element, the user loads in operation 5306 Save images without correction.

According to another exemplary embodiment, operations 5302 and 5303 in FIG. 53 may be ignored, and in operation 5304, the user device may determine that the image includes a photography restriction element based on a user input for selecting a specific object included in the image. . For example, when the user touches and keeps touching a part of the screen displaying the object, the user device may determine that the object is a photography restriction element.

54 and 55 are diagrams illustrating an example of a method for correcting a captured image according to an exemplary embodiment, and an example of correcting an image stored in a user device while transmitting the image externally.

In FIG. 54, when the gallery application is executed, multiple images are displayed on the screen of the smartphone 100-2 on the left, and when an image 5410 is touched and selected from the multiple images, the image 5410 is on the right. The enlarged image 5420 is displayed on the telephone 100-2.

Although the image 5410 contains a photographing-restricted person, the mosaic processing is not performed on the area 5411 corresponding to the face of the photographing-restricted person. In addition, the mosaic processing is not performed on the area 5421 of the enlarged image 5420, and the area 5221 corresponds to the face of a person whose photography is restricted.

In the current embodiment, since the smartphone 100-2 stores the image without correcting the image, the user can access the original image via the gallery application of the smartphone 100-2.

However, in the current embodiment, if the image stored in the smart phone 100-2 is to be transmitted to an external server or another user device, it is determined whether the image includes a photography restriction element; and when the image includes a photography restriction element When the image is corrected, The transmission is transmitted to an external server or another user device.

Referring to FIG. 55, when a user selects an image 5510 from a plurality of images displayed on the smartphone 100-2 via a gallery application and selects a "send" button 5530 to share the image 5510, the smartphone 100-2 determines the image Whether 5510 includes a photography restriction element, and when image 5510 includes a photography restriction element, correct image 5510 before sharing image 5510.

In FIG. 55, the smartphone 100-2 transmits an image 5520 corrected by performing a mosaic process on an area 5521 corresponding to the face of a photographing-restricted person to the server 300 designated as the target.

FIG. 56 is a flowchart describing an example of modifying a captured image while externally transmitting the image with respect to a method for modifying a captured image according to an exemplary embodiment.

Referring to FIG. 56, in operation 5601, a user device captures an image. Here, the user device may be any device capable of photographing, such as a smart phone, a video camera, or a tablet PC.

In operation 5602, the user device stores the image in a memory included in the user device.

In operation 5603, the user device receives a request to transmit an image from the user to an external server or another user device.

In operation 5604, the user device recognizes at least one of a person included in the image, an object included in the image, or a location where the image is captured. The method of identifying a person, object, or location has been described above with reference to FIG. 2.

In operation 5605, the user device is based on the identified individuals and users At least one of a relationship between a user of the device, an identified object, and an identified position is used to determine whether the image includes a photography restriction element. The method of determining whether an image includes a photographing restriction element has been described above with reference to FIG. 2.

When it is determined in operation 5606 that the image contains a photographing restriction element, the user device corrects the image and transmits the corrected image to an external server or another user device in operation 5607. When it is determined in operation 5606 that the image does not include a photography restriction element, the user device transmits the image without correction in operation 5608.

FIG. 57 is a description of a method for modifying a captured image according to an exemplary embodiment. The captured image is immediately corrected without storing the captured image in a user device and the modified captured image is transmitted externally. A diagram of an instance of the image.

Figure 57 illustrates a call between two smartphones 100-2a and 100-2b.

Here, the smart phones 100-2a and 1002-b do not store the video call screen unless requested. However, according to the current embodiment, since the photography restriction element needs to be corrected even during a video call, the smart phone 100-2a can immediately correct the image without storing the image and transmit the corrected image to the smart phone 100 -2b. Referring to the video call screen 5710 displayed on the smart phone 100-2b, the face area 5712 of the person talking on the phone is uncorrected, but the mosaic processing is performed on the face area 5711 of the photography-restricted person.

FIG. 58 is a description of a method for modifying a captured image according to an exemplary embodiment. The captured image is immediately corrected without storing the captured image in a user device and the modified captured image is transmitted externally. Flow chart of an example of images.

Referring to FIG. 58, in operation 5801, a user device captures an image. Here, the user device may be any device capable of photographing, such as a smart phone, a video camera, or a tablet PC.

In operation 5802, the user device recognizes at least one of a person included in the image, an object included in the image, and a location where the image is captured. The method of identifying a person, object, or location has been described above with reference to FIG. 2.

In operation 5803, the user device determines whether the image includes a photographing restriction element based on at least one of the relationship between the identified individual and the user of the user device, the identified object, and the identified position. The method of determining whether an image includes a photographing restriction element has been described above with reference to FIG. 2.

When it is determined in operation 5804 that the user device includes a photographing restriction element, the image is corrected in operation 5805 and then transmitted to an external server or another user device. When it is determined in operation 5804 that the user device does not contain a photographing restriction element, the image is transmitted to an external server or another user device without correction in operation 5806.

59 to 61 are diagrams for describing an example of correcting an image captured at a photographing restriction position according to an exemplary embodiment.

Referring to FIG. 59, the smart phone 100-2 of the user A can capture an image 5910a of a person 5911 at a photography-permitted location (such as a street or a restaurant), and at a photography-restricted location (such as a concert hall, a museum, or a military security area) Capture image 5920a of individual 5911. The photography restriction of the individual 5911 may be changed based on the photography position. For example, if the individual 5911 is a celebrity, photos located at the concert hall can be attributed to copyright restrictions, but photos located on the street can be unlimited. So wisdom The model phone 100-2 transmits an image to a server 300 for image processing. Here, the smart phone 100-2 transmits the image together with the location information to the server 300. The location information is information about the location where the image was captured. The smart phone 100-2 can determine the position by using the GPS included therein, and then transmit the position information to the server 300. Alternatively, the smart phone 100-2 generates location information via any of a variety of methods (e.g., by using a base station or an access point (AP) to determine a location that is connected to the smart phone 100-2).

Referring to FIG. 60, the server 300 may determine the position of the captured image by analyzing the position information received from the smart phone 100-2, and determine whether the position is a photographing restricted position. In other words, the server 300 determines whether the position is included in the pre-stored photography restricted position database. Here, the pre-stored photography-restricted-location database may store photography-restricted locations (such as concert halls, museums, or military security areas) that are restricted by the user's photography-restricted locations or by ordinary people.

When the position of the captured image 5910a is determined not to be a photographing restricted position, the server 300 may transmit the uncorrected image 5910b to the smart phone 100-2. Alternatively, the server 300 may transmit the image 5910a to the smart phone 100-2 which does not need to be corrected.

When the position of the captured image 5920a is determined to be a photography-restricted position, the server 300 determines that the image 5920a includes a photography-restricted element. Therefore, the server 300 can correct a part of the image 5920a corresponding to the individual whose photography is restricted. In detail, the server 300 may generate and execute a mosaic process on a face region of a person 5911 in the image 5920a of a person whose photography is restricted for photography, a substitute image composition process on a face region, or a blur process on a face region. The corrected image 5920b is output.

In FIGS. 59 and 60, the server 300 corrects the image, but instead, the server 300 can only determine whether the position where the captured image is a photographing restricted position by analyzing the position information received from the smartphone 100-2 and will determine The results are transmitted to the smart phone 100-2, and the smart phone 100-2 can correct the image.

Meanwhile, in FIG. 59 and FIG. 60, the photography restriction position differs depending on the subject, but alternatively, the photography restriction position can be applied to all users equally. In other words, the photography-restricted position can be determined regardless of whether an individual is included in the image, and when determining the position of the captured image corresponding to the photography-restricted position, a portion of the image corresponding to the person or object included in the image can be corrected .

Alternatively, as shown in FIG. 61, the smart phone 100-2 of the user A may determine the position of the captured image by analyzing the position information, and determine whether the position corresponds to the photographing restriction position. In other words, the smart phone 100-2 determines whether the location is included in the photographing restricted location database stored in the memory of the smart phone 100-2.

When it is determined that the position of the captured image 5910a is not the photography-restricted position of the individual 5911, the smartphone 100-2 may store the uncorrected image 5910b.

When it is determined that the position of the captured image 5920a is the photography-restricted position of the individual 5911, the smartphone 100-2 determines that the image 5920a includes a photography-restricted element. Therefore, the smartphone 100-2 can correct a part of the image 5920a corresponding to the individual 5911 who is a photography-restricted individual. In detail, the smartphone 100-2 may generate and output an image by performing mosaic processing on a facial area of the person 5911 in the image 5920a, performing alternative image composition processing on a facial area, or performing blur processing on a facial area. Corrected image 5920b.

Similarly, when photographing a person or object, Location, individuals or things can be restricted or unrestricted.

FIG. 62 is a flowchart illustrating a method of correcting an image captured at a photographing restriction position according to an exemplary embodiment.

Referring to FIG. 62, in operation 6201, a user device captures an image. Here, the user device may be any of a general photography device (such as a video camera or camcorder), a wearable device (such as smart glasses or a smart watch), and a mobile device (such as a smart phone or tablet PC). One, or alternatively, the user device may be any of a variety of devices capable of photographing.

In operation 6202, the user device identifies a position of the user device during photography. Here, the user device can determine the location by using the GPS included therein. Alternatively, the user device may determine the position by using the position of a base station or an AP connected to the user device.

In operation 6203, the server determines whether the position corresponds to a photographing restriction position. Alternatively, the user device may determine whether the position corresponds to a photographing restriction position. Specifically, it is determined whether the position corresponds to a photographing restriction position of an individual included in the video, or there is no photographing restriction position regarding the subject. Therefore, the server or the user device can analyze the position information to determine the position of the captured image, and compare the position with the photographing restriction position pre-stored in the photographing restriction position database. Here, the photography-restricted-location database may store photography-restricted locations (such as concert halls, museums, and military security areas) according to people's photography-restricted locations or by ordinary people.

When the position corresponds to the photographing restriction position, the server or the user device may determine whether the image includes a photographing restriction element in operation 6204. In other words, when the position corresponds to a photography-restricted position for a specific individual, the server or user device can determine the video Whether the image contains a specific individual; and when the location corresponds to the photography-restricted location of all people or objects, the server or user device can determine whether the image contains at least one individual or object. When the image includes a photography restriction element, the server may correct the image and transmit the corrected image to the user device in operation 6205. Alternatively, the user device may correct the image and store the corrected image. For example, the server or user device can generate a corrected image by performing mosaic processing, blurring, or alternative image composition processing on the image.

63 to 66 are diagrams for describing an example of correcting a captured image according to a user selection according to an exemplary embodiment.

Referring to FIG. 63, the smartphone 100-2 of the user A captures an image including the user B and the user C. Subsequently, the smartphone 100-2 displays the image on the screen.

In FIGS. 64 and 65, when the user A selects an individual on which protection processing is to be performed from an image through different methods, a part of the image corresponding to the selected individual is corrected. Here, before the image is stored in the memory of the smartphone 100-2 or shared with an external device, when (for example) a preview screen or a temporary screen is provided on the smartphone 100-2, the user A may Choose individuals.

Referring to FIG. 64A, the smartphone 100-2 recognizes the faces of the user B and the user C by performing facial recognition on the original image 6400a. Subsequently, the smart phone 100-2 may provide a result of facial recognition indicating that the faces of the user B and the user C are recognized. The results may be provided as visible feedback, for example, by highlighting (flickering or coloring) the faces or facial contours of User B and User C. Subsequently, when the user A selects one of the user B and the user C, for example, selects to use In the case of the user C, the smart phone 100-2 can perform protection processing on the user C. For example, the smart phone 100-2 may modify a part of the image corresponding to the user C so that the user C is indistinguishable. In detail, as shown in FIG. 64B, the smartphone 100-2 may display a modified image generated by performing a mosaic process, an alternative image composition process, or a blur process on the face area of the user C in the original image 6400a. Image 6400b.

As another example, referring to FIG. 65A, the smartphone 100-2 may recognize the faces of the user B and the user C by performing facial recognition on the original image. Subsequently, the smartphone 100-2 may determine whether the face corresponds to a photographing-restricted individual. When it is determined that the user B and the user C correspond to photography-restricted people, the smart phone 100-2 may perform protection processing on the user B and the user C. For example, the smart phone 100-2 may modify a portion of the original image corresponding to the user B and the user C so that the user B and the user C are indistinguishable. In detail, the smartphone 100-2 can display a corrected image 6500a obtained by performing mosaic processing, alternative image composition processing, or blur processing on the face areas of the user B and the user C who restrict people for photography. Subsequently, when the user A selects one of the user B and the user C, for example, when the user B is selected, the smart phone 100-2 may determine that the user B is not to be protected. Therefore, the smart phone 100-2 can correct a part of the corrected image 6500a corresponding to the user B so that the user B is discernable. In detail, as shown in Section 65B, the smartphone 100-2 may display a result obtained by releasing a mosaic process, an alternative image layout process, or a blur process performed on the face area of the user B in the corrected image 6500a. A corrected image of 6500b.

As another example, referring to FIG. 66A, the smart phone 100-2 Face recognition is performed on the original image 6600a to recognize the faces of the user B and the user C. Subsequently, the smart phone 100-2 may provide a pop-up screen 6610 asking whether the user A performs a protection process on the user C. When the user A selects a button requesting the smart phone 100-2 to perform the protection process via the pop-up screen 6610, the smart phone 100-2 can perform the protection process on the user C. Subsequently, as shown in FIG. 66B, the smartphone 100-2 may display a corrected image 6600b obtained by performing a mosaic process, an alternative image composition process, or a blur process on the face area of the user C in the original image 6600a. .

Before an image captured by the smart phone 100-2 is stored in the memory of the smart phone 100-2, a screen for selecting an individual on which protection processing is to be performed may be displayed. Alternatively, the screen may be displayed before the smartphone 100-2 provides an image to an external device such as an external server or a third-party smartphone.

Meanwhile, when an individual on which protection processing is to be performed is selected, information about the individual may be stored in a database of the smart phone 100-2 or the server 300. For example, individuals can be designated as photography-restricted elements in information. In this case, when an individual is photographed in a future image, the smart phone 100-2 may automatically determine that the individual is a photography-restricted individual. Subsequently, the smartphone 100-2 may perform a mosaic process, an alternative image layout process, or a blur process on an area corresponding to an individual's future image.

Also, when the smartphone 100-2 or the server 300 does not have information about a face included in the captured image, a method of restricting an individual by the user selecting photography is useful. In other words, when the smart phone 100-2 or the server 300 does not have information about a face included in the captured image, the smart phone 100-2 or the server 300 may inconsistently process the captured image (for example, protection Or not guaranteed Face protection), but it is possible to determine whether or not to process the face according to the user's choice, so the user's freedom for processing the captured image is increased.

67 and 68 are flowcharts illustrating a method of correcting a captured image according to a user selection according to an exemplary embodiment.

Referring to FIG. 67, in operation 6701, a user device captures an image including a person. Here, the user device may be any device capable of photographing, such as a smart phone, a video camera, or a tablet PC.

In operation 6702, the user device performs facial recognition on an individual included in the image.

In operation 6703, the user device may provide a result of facial recognition. For example, a user device may display a screen where visible feedback is applied to an individual.

In operation 6704, when the screen is displayed, the user of the user device may select an individual on which the protection process is to be performed.

When the user selects an individual on which the protection process is to be performed, the user device may perform the protection process on the selected individual in operation 6705. For example, the user device may modify the image so that the selected individual is indistinguishable.

As another example, referring to FIG. 68, the user device captures an image in operation 6801.

In operation 6802, the user device performs facial recognition on an individual included in the image.

In operation 6803, the user device may determine whether the individual is a photography restricted individual. When the individual is a photography restricted individual, the user device may perform protection processing on the individual in operation 6804. When an individual is not a photography restricted individual, the user device may No protection processing is performed.

In operation 6805, the user device may display an individual on which the protection process is performed. For example, the user device may perform a mosaic process, an alternative image layout process, or a blur process on an area corresponding to an image of an individual.

In operation 6806, when the individual on which the protection process is performed is displayed, the user may select the individual to be released from the protection process.

When the individual to be released from the self-protection process is selected, the user device may remove the protection process from the selected individual in operation 6807. For example, the user device may undo a mosaic process, an alternative image layout process, or a blur process performed on an area corresponding to a selected individual's image.

As described above, according to one or more exemplary embodiments, when an image including a photography-restricted person, object, or location is captured by a user device, a portion of the entire image may be modified such that the photography-restricted person, object, or location is corrected It is indistinguishable, and therefore protects the privacy of individuals whose photography is restricted, and prevents individuals from obtaining images containing objects or locations where photography is restricted due to security or copyright reasons.

When not limited thereto, the exemplary embodiments may be implemented as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium is any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer-readable recording medium include read-only memory (ROM), random-access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device. The computer-readable recording medium may also be distributed on a network-coupled computer system, so that the computer-readable code is stored and executed in a distributed manner. In addition, the exemplary embodiments may be implemented as computer programs transmitted on a computer-readable transmission medium such as a carrier wave. It can be written and can be received and implemented in a general-purpose or special-purpose digital computer running a program. In addition, it should be understood that in the exemplary embodiments, one or more units of the devices and apparatus described above may include circuits, processors, microprocessors, etc., and may execute a computer stored in a computer-readable medium Program.

The above exemplary embodiments are merely exemplary and should not be construed as limiting. The teachings of the present invention can be easily applied to other types of devices. In addition, the description of the exemplary embodiments is intended to be illustrative and not intended to limit the scope of patent application, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

Claims (13)

A user device includes: a memory configured to store data; and a processor configured to identify an image included in the user device capture or received by the user device from an external source The individual in the determination of the closeness of the relationship between the identified individual and a user registered as the owner of the user device based on whether at least one photo of the identified individual is stored in the storage. Degree of sexuality, determining that the degree of intimacy is lower than a predetermined level in response to any photo of the identified individual not being stored in the storage, and determining to capture the image in response to the image including a sign prohibiting photography The position of the image is a photography-restricted position, a portion of the image corresponding to the identified individual is changed to unrecognizable in response to the degree of intimacy being lower than the predetermined level, and in response to being judged The image is replaced with another image for the position of the photographing restriction position. The user device according to item 1 of the patent application scope further includes a camera configured to capture the image. The user device according to item 1 of the scope of patent application, wherein the processor is further configured to determine whether the position where the image is captured is a photography-restricted position by using the stored data. The user device according to item 1 of the scope of patent application, wherein the data stored in the storage is an address book, and the processor is further configured to respond that the identified individual does not include It is determined in the address book that the degree of intimacy is lower than the predetermined level. The user device according to item 1 of the scope of patent application, wherein the data stored in the storage is a record of social network service usage of the user, and the processor analyzes the Social network service usage records determine the degree of intimacy. The user device as described in claim 1, wherein the processor is further configured to determine whether the position capturing the image is a photography-restricted position by accessing a database of an external server. The user device according to item 1 of the scope of patent application, wherein the processor is further configured to identify a photography-restricted object from the image, and determine a capture location in response to identifying the photography-restricted object from the image. The position of the image is a photographing restriction position. The user device as described in claim 1, wherein the processor is further configured to perform mosaic processing, blur processing, background image composition processing, and alternative image composition processing on the portion of the image To change the portion of the image to be unrecognizable. A system for correcting an image, the system comprising: a server configured to perform image processing; and a user device configured to request the image processing when transmitting an image to the server; wherein The server is further configured to identify a person included in the image received from the user device, and determine the person based on whether at least one photo of the identified person is stored in the user device. The degree of intimacy of the relationship between the identified individual and the user registered as the owner of the user device, Determining that the degree of intimacy is lower than a predetermined level in response to any photo of the identified individual not being stored in the user device, and determining to capture the image in response to the image including an instruction to prohibit photography The position of is a photographing restricted position, a part of the image corresponding to the identified individual is changed to unrecognizable in response to the degree of intimacy being lower than the predetermined level, and in response to being judged as being The position of the photographing restricted position and replacing the image with another image. The system for correcting an image according to item 9 of the scope of patent application, wherein the server is further configured to determine whether the position capturing the image is a photographing restricted position by using the stored data. The system for correcting an image as described in claim 9 of the scope of patent application, wherein the server is further configured to receive the time and location from which the image was captured from the user device, and by using the stored The data determines whether photography of the identified individual is restricted at the received time and at the received location. The system for amending an image as described in claim 9 of the scope of patent application, wherein the user device is further configured to encode the image by using an encryption key received from the server and The encoded image is transmitted to the server, and the server is further configured to decode the encoded image and change the portion of the decoded image corresponding to the identified individual. A method for correcting an image by a user device, the method comprising: identifying an individual included in the image; and determining the identified based on whether at least one photo of the identified individual is stored in the user device. Of individuals and all registered as the user ’s device Degree of intimacy among users of the subject; determining that the degree of intimacy is lower than a predetermined level in response to any photo of the identified individual not being stored in the user device; Determining that the position where the image was captured is a photographing restricted position; in response to the degree of intimacy being lower than the predetermined level, changing a part of the image corresponding to the identified individual to Unrecognizable; and replacing the image with another image in response to the position determined to be the photographing restricted position.